JDS was engaged to performance test a National Sales Event at a very large scale in 2020. There are multiple events which happen throughout different periods of the year but this particular engagement was in preparation for the largest sales event of the year.
Test Scenario Approach and Modelling
Typical performance load tests would start off with a conservative ramp up followed by steady state. This can be seen in the image below.
This particular engagement called for a different kind of performance load test scenario which was very unique as it required two aggressive ramp ups across two different user groups (see image below).
The performance load test was also unlike any typical load tests as it required a large number of virtual users to be tested against (>100,000). Multiple approaches with available performance test tools were considered and the final call was to run the test via JMeter as it was cost effective and lightweight in terms of load generation resource utilisation.
Load Generation Infrastructure Approach
Load generator infrastructure was created via AWS EC2 as this allowed quick stand up and stand down of infrastructure which would allow easier management of load gen infrastructure cost.
One key approach to testing especially with the AWS load gen infrastructure was to test the load in increments. This allowed testing to be done in batches with different levels of load instead of trying to attempt the full load at the first or second attempt. By doing this, the cost of standing up and managing the load generation infrastructure will be much less.
During initial testing with a smaller load footprint, a performance bottleneck was identified. At about 1% of the expected load it was found that there was an issue when it came to user sign in. Users were then not allowed to log in and were thrown off the site. If this had happen during Go Live, the business would lose out on potential customers and the user confidence of the website would suffer and result in less customers in the future.
Once that was resolved it was also noted that response time performance across the client website improved once the application servers warmed up once load was introduced and further dropped when auto-scaling on the environment was triggered with the ramp up of load.
Server warm up can easily be executed prior to the site actually going live by introducing some artificial load on the live production site. This was something that the client had confirmed was already in place.