Tag: Performance Testing

Performance Testing of Large Scale National Sales Event Website

Summary

JDS was engaged to performance test a National Sales Event at a very large scale in 2020. There are multiple events which happen throughout different periods of the year but this particular engagement was in preparation for the largest sales event of the year.

Test Scenario Approach and Modelling

Typical performance load tests would start off with a conservative ramp up followed by steady state. This can be seen in the image below.

This particular engagement called for a different kind of performance load test scenario which was very unique as it required two aggressive ramp ups across two different user groups (see image below).

The performance load test was also unlike any typical load tests as it required a large number of virtual users to be tested against (>100,000). Multiple approaches with available performance test tools were considered and the final call was to run the test via JMeter as it was cost effective and lightweight in terms of load generation resource utilisation.

Load Generation Infrastructure Approach

Load generator infrastructure was created via AWS EC2 as this allowed quick stand up and stand down of infrastructure which would allow easier management of load gen infrastructure cost.

One key approach to testing especially with the AWS load gen infrastructure was to test the load in increments. This allowed testing to be done in batches with different levels of load instead of trying to attempt the full load at the first or second attempt. By doing this, the cost of standing up and managing the load generation infrastructure will be much less.

Findings

During initial testing with a smaller load footprint, a performance bottleneck was identified. At about 1% of the expected load it was found that there was an issue when it came to user sign in. Users were then not allowed to log in and were thrown off the site. If this had happen during Go Live, the business would lose out on potential customers and the user confidence of the website would suffer and result in less customers in the future.

Once that was resolved it was also noted that response time performance across the client website improved once the application servers warmed up once load was introduced and further dropped when auto-scaling on the environment was triggered with the ramp up of load.

Server warm up can easily be executed prior to the site actually going live by introducing some artificial load on the live production site. This was something that the client had confirmed was already in place.

To learn more, contact our team today on 1300 780 432, or email [email protected].

Is DevPerfOps a thing?

New technology terms are constantly being coined. One of our lead consultants answers the question: Is DevPerfOps a thing?


Hopefully it’s no surprise that traditional performance testing has struggled to adapt to the new paradigm of DevOps, or even Agile. The problems can be boiled down to a few key factors:

  1. Performance testing takes time—time to script, and time to execute. A typical performance test engagement is 4-6 weeks, during which time the rest of the project team has made significant progress. 
  2. Introducing performance testing to a CI/CD pipeline is really challenging. Tests typically run for an hour or more and can hold up the pipeline, and they are extremely sensitive to application changes as the scripts operate at the protocol level (typically HTTP requests). 
  3. Performance testing often requires a full-sized, production-like infrastructure environment. These aren’t cheap and are normally unavailable during early development, making “early performance testing” more of an aspirational idea, rather than something that is always practical.

All the software vendors will tell you that DevOps with performance testing is easy, but none of them have solved the above problems. For the most part, they have simply added a CI/CD hook without even attempting to challenge the concept of performance testing in DevOps at all.

A new world

So what would redefining the concept of performance testing look like? Allow me to introduce DevPerfOps. We would define the four main activities of a DevPerfOps engineer as:

  1. Tactical Load Tests
  2. Microbenchmarking
  3. APM and Code Reviews
  4. Container Workloads

Let’s start at the top.

Tactical Load Testing

This is can be broadly defined as running performance tests within the limitations of a CD/CI pipeline. Is it perfect? No. Is it better than not doing anything or struggling with a full end-to-end test? Absolutely, yes!

Tactical Load Testing has the following objectives:

  • Leverage CI toolkits like Jenkins for execution and results collection.
  • Use lightweight toolsets for execution (JMeter, Taurus, Gatling, LoadRunner, etc. are all perfectly good choices).
  • Configure performance tests to run for short durations. 15-20 minutes is the ideal balance. This can be achieved by minimising the run logic, reducing think times, and focusing on only testing individual business processes.
  • Expect to run your performance tests on a scaled-down environment, or a single server node. Think about what this will do to your workload profile.

Microbenchmarking

These are mini unit or integration tests designed to ensure that a component operates within a performance benchmark and that deviations are detected.

Microbenchmarking can target things like core application code, SQL queries, and Web Service integrations. There is an enormous amount of potential scope here; for example, you can write a microbenchmark test for a login, or a report execution, or a data transformation--anything goes if it makes sense.

Most importantly, microbenchmarking is a great opportunity to test early in the project lifecycle and provide early feedback.

APM and Code Reviews

In past years, having access to a good APM tool or profiler, or even the source code, has been a luxury. Not anymore--these tools are everywhere, and while a fully featured tool like AppDynamics or New Relic is beneficial for developers and operations, a lot can be achieved with low-cost tools like YourKit Profiler or VisualVM.

APM or profilers allow slow execution paths to be identified, and memory usage and CPU utilisation to be measured. Resource intensive workloads can be easily identified and performance baked into the application.

Container Workloads

Containers will one day rule the world. If you don’t know much about containers, you need to learn about them fast.

In terms of performance engineering, each container handles a small unit of workload and container orchestration engines like Kubernetes will auto-scale that container horizontally across the entire data centre. It’s not uncommon for applications to run on hundreds or thousands of containers.

What’s important here is that the smallest unit of workload is tiny, and scaling is something that a well-designed application will get for free. This allows your performance tests to scale accordingly, and it allows the whole notion of “what is application infrastructure” to be challenged. In the container world, an application is a service definition… and the rest is handled for you.

So, is DevPerfOps a thing? We think so, and we think it will only continue to grow and expand from here. It’s time that performance testing meets the needs of 2018 IT teams, and JDS can help. If you have any questions or want to discuss more, please get in touch with our performance engineering team by emailing [email protected]. If you’re looking for a quick, simple way of getting actionable insights into the performance health of your website or application, check out our current One Second Faster promotion.

Our team on the case

Read more Tech Tips

What if your application was one second faster?

Why one second faster?

Improving your website performance will increase your business. But don’t take our word for it—there is plenty of evidence.

According to Kissmetrics:

  • 25% of consumers will abandon a website that takes more than four seconds to load
  • 47% of consumers expect a webpage to load in two seconds or less
  • 79% of shoppers who are dissatisfied with website performance are less likely to buy from the same site again
  • A one-second delay in page response can result in a 7% reduction in conversions
  • A one-second delay (or three seconds of waiting) decreases customer satisfaction by about 16%

So, what would performing one second faster mean for your web application or website? JDS is now offering a limited time promotion that will allow you to realise the maximum performance of your website or application. Over the course of five days, our experts will work with your team to analyse your web application and accelerate its performance for your customers.

 

What’s included?

  • Your own dedicated performance expert for five days (either on-site or off-site)
  • A technical deep dive of your web application, turning over every rock to understand how it can work faster and harder for your business
  • Best practice tips and techniques straight from the guys in the know
  • Experts fluent in everything from Java and .NET through to SAP and Oracle
  • A presentation and roadmap of the findings and recommendations found

Why JDS?

We are Australia’s leading performance test consultancy with 15 years of experience partnering with organisations of every size, from startups to large enterprises and governments. We have a reputation for being a key player in making Australian web applications exceptional. Want to get started? Reach out to a JDS team member, send an email to [email protected], or call 1300 780 432 to confidentially discuss your web application and how we can help.

We partner with leading technologies

Case Study: CitiPower and Powercor take action to ensure quality

“Technology underpins the ability of CitiPower and Powercor to safely and effectively deliver power to more than a million customers. It’s critical that our applications are current, functioning and available,” says Fiona Hocking, CSA assurance team leader, Powercor.

Objective

To deliver top-quality, high-performing business applications that are current, functioning and available

Challenge

Develop an effective Quality Assurance (QA) discipline for application lifecycles across the enterprise

IT improvements

  • Provides HP Quality Centre (QC) to application groups with zero outages
  • Provides a holistic view of an application testing status
  • Allows application teams to manage up to 30 test cycles with 500 test cases, resulting in the management of over 6,500 defects

About the Client

CitiPower and Powercor are Australia’s leading electricity distributors owned by Cheung Kong Infrastructure Ltd (CKI) and Power Assets Holdings Ltd. Combined, the group operates the largest electricity distribution network in the state of Victoria, servicing over a million customers.

Operating in a highly controlled industry and tasked with providing critical services around the clock, the group relies heavily on its IT infrastructure to deliver energy to customers and to meet regulatory requirements.

This infrastructure is complex to say the least. With over 11 critical business applications, 2,000 users, and multifaceted interfaces to a central information hub for the business facing systems alone, delivery expectations on the IT department are high. Field staff, the group and the government regulator all need real-time information from the electricity network, such as usage patterns, meter data and consumption. Information accuracy and availability can mean the difference between someone getting power or not. Significant penalties apply for missing the mark, so it’s important that the IT department gets it right.

Industry

Power

Primary software

  • HP Quality Center

High availability

To deliver on its strategic objectives, the group mandates that its IT department provides high-performing business applications with a 99.8 percent availability rate. There is zero tolerance for level one defects.

As an internal service provider, the Quality Assurance Team had to define how to best facilitate enterprise testing capabilities with limited resources and tight timeframes. At any one time, an application could be managing 20 to 30 test cycles, with over 6,500 defects being handled across its project lifecycle. The Quality Assurance (QA) Team quickly identified that HP Quality Centre (QC) could be of immediate benefit to application owners. Providing a holistic view of an application’s testing status would allow an application owner to assess priorities, allocate resources and deliver quality applications on time and to specification.

In order to be effective and highly responsive when called upon by stakeholders, the QA Team identified three key areas for HP QC success. Firstly, they enlisted support services from HP Platinum Partner JDS Australia to ensure that HP QC continued to be operational with fast issue resolution.

“When it comes to HP Quality Center, JDS keeps the lights switched on. We simply don’t have to worry about HP QC not being available or operational. JDS ensures that we are aware of product developments, new features and functions, and that our system is kept healthy. I know that any time I call them, they can answer my questions. Our HP QC users have aggressive targets, and using JDS as support means that we can keep our users happy,” said Hocking.

Tester training

Secondly, the QA Team packaged training programs that could be easily rolled out when required. As each testing requirement is based on secondment, an application owner can nominate different testers as deemed appropriate. Given that testing groups have often never used testing tools, the QA Team has to equip new testers with the skills needed to get the job done.

“HP QC is very customisable to meet the needs of each specific project. This means that we have to be nimble enough to roll out project-specific training,” says Hocking. “The fact HP QC is intuitive makes the training process easier. In some cases, we can train large groups on the basics in less than thirty minutes. Thirdly, the QA Team assists project teams by helping define testing requirements, developing test scripts, and leveraging HP QC for optimal outcomes.

“Since we were tasked with rolling out HP QC capabilities across the enterprise, we have increased its use by 78 percent and improved user-uptake by 67 per cent.”
With a solid QA platform in place that is supported by JDS, CitiPower and Powercor are now better placed to maintain quality applications under pressure and amidst rapid change.

When the government of Victoria announced its smart meter program for improved accuracy of billing, faster connections and disconnections and real-time usage information, the group welcomed the challenge. The project includes a tight time-frame for the installation of 1.2 million smart meters that can transmit two-way data to a central repository.

Leveraging the capabilities provided by the QA Team, the Advanced Metering Infrastructure (AMI) Network Management System (NMS) support team used HP QC to drive the testing. HP QC is used to ensure that a third-party vendor meets Service Level Agreements (SLAs), that the AMI NMS application (Utility IQ) functions as it is supposed to, and that changes to the application do not present performance risks. HP QC allows the application owner to deliver quality solutions that meet company standards.

Field Mobile Applications

Further adhering to government regulations and business drivers, the group has deployed the Ventyx Service Suite to enable the accurate capture of real-time information from the field through over 500 PDAs. Adopting a thorough end-to-end testing process, the application group has aligned business expectations and security issues with the application itself, according to Alan King, manager, Field Mobile Applications. Incorporating HP QC allowed the team to effectively manage over 500 test cases and improve application functionality with the third-party vendor, based on discovered SLA deficiencies.

“HP QC allows us to manage our vendors and internal customers, as we can negotiate patches and articulate the priorities. It also improves our service levels to our internal customers as they have visibility into the defects and can contribute to the priority fix list,” says King.

Hocking adds: “selecting a high performing tool such as HP QC was the first step but having a skilled group of testing and monitoring consultants behind us makes all the difference. Put simply, JDS keeps the testing technology working so that other critical applications continue to be highly available with minimal defects. We would not have achieved the results without them,” says Hocking.

Powercor and CitiPower will continue to draw upon JDS’s expertise in the areas of technical testing, IT monitoring, and HP Software Support for the future.
“JDS Support has provided outstanding service levels. In the past three years, 60 percent of all support calls have been resolved instantly, and the rest within days,” concludes Hocking. “I can have skilled technicians on-site quickly if I need and can access a large pool of expertise for troubleshooting. With JDS, the Quality Assurance Team has been able to deliver on its service levels and we will continue to work together as we further enhance QA capabilities.”

Business Benefits

  • Rolled out HP Quality Centre across the enterprise to improve uptake of QA processes by 78 percent and increased application performance quality
  • Leveraged HP QC’s capabilities to support application testing for the Smart Meter project
  • Improved quality processes to enable IT application teams to deliver on service levels

Our team on the case

[amoteam max="3" categories="217" item-width="250" item-margin="20" full-width="yes" align="center" panel="right"]

Why choose JDS?

At JDS, our purpose is to ensure your IT systems work wherever, however, and whenever they are needed. Our expert consultants will help you identify current or potential business issues, and then develop customised solutions to suit you.

JDS is different from other providers in the market. We offer 24/7 monitoring capabilities and support throughout the entire application lifecycle. We give your IT Operations team visibility into the health of your IT systems, enabling them to identify and resolve issues quickly.

We are passionate about what we do, working seamlessly with you to ensure you are getting the best possible performance from your environment. All products sold by JDS are backed by our local Tier One support desk, ensuring a stress-free solution for the entire product lifecycle.