A Comparison of Open Source Load Testing Tools

This is the start of a multi-part series of articles that compares the available free/open source load testing tools.

I plan to break this up into bite-sized chunks roughly as follows:

1. Community (support resources, development community)
2. Tool architecture (what components, and how they fit together)
3. Scripting (recording, correlation, debugging)
4. Infrastructure Monitoring
5. Test Scenario Creation
6. Test Execution
7. Test Analysis and Reporting
8. Wrap up/Summary

Selection Criteria

The scope of this comparison has been limited to tools that a professional performance tester would recognise as a load testing tool.

• The tool must be intended to emulate multiple concurrent users on an application (i.e. be a load testing tool).
• Must be a tool for testing web-based applications. May support other protocols, but these will not be part of the evaluation.
• Must be a “protocol-level” tool. The replay engine should emulate multiple users by sending HTTP traffic, rather than by interacting with a GUI.
• Must be able to emulate a complex business process such as an e-commerce purchase transaction, where there are multiple steps and lots of dynamic data. It cannot be a “dumb” load testing tool that just makes requests for URLs (e.g. curl-loader, Siege, Deluge etc).
• Must be a free load testing tool (as in beer). May or may not be released under an Open Source license.

Which Tools?

Here are the tools that I could find that match the selection criteria:

• JMeter – An Apache Jakarta project. Used by performance testers at Google
• OpenSTA – Originally developed by CYRANO, a software company that went bankrupt.
• WebLOAD – Is commercial software, but a free version was released under an open source license by Radview in 2007 (I will not be evaluating the features of WebLOAD Professional, which is not free).
• The Grinder – Originally developed for the book “Professional Java 2 Enterprise Edition with BEA WebLogic Server” (published in October, 2000), it has had a couple of major rewrites since then.

Licensing

All of these tools are free to download. Source code is distributed under the following licenses:

• JMeter – Apache License 2.0. You are free to modify the source code, and distribute modified source or binaries. You are not required to distribute your modified source code, but source code must preserve the original copyright notice and disclaimer. It is very similar to the BSD license.
• OpenSTA – GNU General Public License 2.0. If you distribute the software as a binary, you must also give the source code to the people you distributed it to. This includes any changes you made to the source code.
• The Grinder – BSD License. Take the source code and do whatever you like with it (including turning it into a closed-source commercial software). Must retain the copyright notice on the sourcecode.
• WebLOAD – GNU General Public License (GPL). See license explanation for OpenSTA.

Someone who is just planning to use the tools is unlikely to care about the differences between the licenses.

A company that is planning on taking one of these tools and turning it into their own commercial product would want to base their product on The Grinder or JMeter, as the licenses that these are released under do not require that they release the source code to their customers.

A company that is planning on modifying these tools for their own internal use (or as a tool to be used by their consultants) could pick any of the tools, as they are free to modify them without releasing source code if they are using them internally or are using them to provide a service to their customers.

Development Community

How active is development?

• JMeter
  • Most recent version: 2.3.4, released on June 21st, 2009.
  • Hosted on Apache Jakarta.
• OpenSTA
  • Most recent version: 1.4.4, released on October 20th, 2007.
  • Hosted on SourceForge.
• The Grinder
  • Most recent version: 3.3, released on December 6th, 2009.
  • Hosted on SourceForge.
• WebLOAD
  • Most recent version: 8.1.0, released on October 23rd, 2007.
  • Hosted on SourceForge.

Purely based on the last release date for each tool, it appears that only JMeter and The Grinder are still under active development.

Note that the most recent version of WebLOAD Professional is 8.4, so it looks like Radview is still working on their commercial version, but are no longer releasing updates for the open source version.

User Community & Support

Personally, I like to use tools that have a large user community. This means that if I encounter a problem, there is probably someone else who has had the same problem and has written about their solution, or who will reply to a question posted to a forum or mailing list. Good documentation for the tool is also nice.

Here is a list of the community and support resources I could find for the different load testing tools:

• JMeter
  • Mailing Lists
  • Wiki
  • FAQ
  • User Manual
  • SQAForums
  • Yahoo Group
• OpenSTA
  • Mailing Lists
  • User Documentation
  • FAQ
  • SQAForums
  • Yahoo Group
• The Grinder
  • Mailing Lists
  • FAQ
  • User Guide
• WebLOAD
  • SQAForums
  • Yahoo Group, Yahoo Group

Please check back soon for part 2 of my open source load testing tools comparison, and feel free to leave a comment with your suggestions.