This is the start of a multi-part series of articles that compares the available free/open source load testing tools.
I plan to break this up into bite-sized chunks roughly as follows:
- Community (support resources, development community)
- Tool architecture (what components, and how they fit together)
- Scripting (recording, correlation, debugging)
- Infrastructure Monitoring
- Test Scenario Creation
- Test Execution
- Test Analysis and Reporting
- Wrap up/Summary
Selection Criteria
The scope of this comparison has been limited to tools that a professional performance tester would recognise as a load testing tool.
- The tool must be intended to emulate multiple concurrent users on an application (i.e. be a load testing tool).
- Must be a tool for testing web-based applications. May support other protocols, but these will not be part of the evaluation.
- Must be a "protocol-level" tool. The replay engine should emulate multiple users by sending HTTP traffic, rather than by interacting with a GUI.
- Must be able to emulate a complex business process such as an e-commerce purchase transaction, where there are multiple steps and lots of dynamic data. It cannot be a "dumb" load testing tool that just makes requests for URLs (e.g. curl-loader, Siege, Deluge etc).
- Must be a free load testing tool (as in beer). May or may not be released under an Open Source license.
Which Tools?
Here are the tools that I could find that match the selection criteria:
- JMeter - An Apache Jakarta project. Used by performance testers at Google
- OpenSTA - Originally developed by CYRANO, a software company that went bankrupt.
- WebLOAD - Is commercial software, but a free version was released under an open source license by Radview in 2007 (I will not be evaluating the features of WebLOAD Professional, which is not free).
- The Grinder - Originally developed for the book "Professional Java 2 Enterprise Edition with BEA WebLogic Server" (published in October, 2000), it has had a couple of major rewrites since then.
Licensing
All of these tools are free to download. Source code is distributed under the following licenses:
- JMeter - Apache License 2.0. You are free to modify the source code, and distribute modified source or binaries. You are not required to distribute your modified source code, but source code must preserve the original copyright notice and disclaimer. It is very similar to the BSD license.
- OpenSTA - GNU General Public License 2.0. If you distribute the software as a binary, you must also give the source code to the people you distributed it to. This includes any changes you made to the source code.
- The Grinder - BSD License. Take the source code and do whatever you like with it (including turning it into a closed-source commercial software). Must retain the copyright notice on the sourcecode.
- WebLOAD - GNU General Public License (GPL). See license explanation for OpenSTA.
Someone who is just planning to use the tools is unlikely to care about the differences between the licenses.
A company that is planning on taking one of these tools and turning it into their own commercial product would want to base their product on The Grinder or JMeter, as the licenses that these are released under do not require that they release the source code to their customers.
A company that is planning on modifying these tools for their own internal use (or as a tool to be used by their consultants) could pick any of the tools, as they are free to modify them without releasing source code if they are using them internally or are using them to provide a service to their customers.
Development Community
How active is development?
- JMeter
- Most recent version: 2.3.4, released on June 21st, 2009.
- Hosted on Apache Jakarta.
- OpenSTA
- Most recent version: 1.4.4, released on October 20th, 2007.
- Hosted on SourceForge.
- The Grinder
- Most recent version: 3.3, released on December 6th, 2009.
- Hosted on SourceForge.
- WebLOAD
- Most recent version: 8.1.0, released on October 23rd, 2007.
- Hosted on SourceForge.
Purely based on the last release date for each tool, it appears that only JMeter and The Grinder are still under active development.
Note that the most recent version of WebLOAD Professional is 8.4, so it looks like Radview is still working on their commercial version, but are no longer releasing updates for the open source version.
User Community & Support
Personally, I like to use tools that have a large user community. This means that if I encounter a problem, there is probably someone else who has had the same problem and has written about their solution, or who will reply to a question posted to a forum or mailing list. Good documentation for the tool is also nice.
Here is a list of the community and support resources I could find for the different load testing tools:
- JMeter
- OpenSTA
- The Grinder
- WebLOAD
Please check back soon for part 2 of my open source load testing tools comparison, and feel free to leave a comment with your suggestions.
Tech tips from JDS

Browser Console
Read More

Glide Variables
Read More

Understanding Database Indexes in ServiceNow
Read More

Fast-track ServiceNow upgrades with Automated Testing Framework (ATF)
Read More

Read More

Splunk .conf18
Read More

ServiceNow Catalog Client Scripts: G_Form Clear Values
Read More

Is DevPerfOps a thing?
Read More

The benefits of performance testing with LoadRunner
Read More

Monitoring Atlassian Suite with AppDynamics
Read More

5 quick tips for customising your SAP data in Splunk
Read More

How to maintain versatility throughout your SAP lifecycle
Read More

How to revitalise your performance testing in SAP
Read More

Reserve and import data through Micro Focus ALM
Read More

How to effectively manage your CMDB in ServiceNow
Read More

ServiceNow and single sign-on
Read More

How to customise the ServiceNow Service Portal
Read More

Integrating a hand-signed signature to an Incident Form in ServiceNow
Read More

Integrating OMi (Operations Manager i) with ServiceNow
Read More

Implementing an electronic signature in ALM
Read More

Service portal simplicity
Read More

Learning from real-world cloud security crises
Read More

Static Variables and Pointers in ServiceNow
Read More

Citrix and web client engagement on an Enterprise system
Read More

Understanding outbound web services in ServiceNow
Read More

How to solve SSL 3 recording issues in HPE VuGen
Read More

How to record Angular JS Single Page Applications (SPA)
Read More

Calculating Pacing for Performance Tests
Read More

Vugen and GitHub Integration
Read More

What’s new in LoadRunner 12.53
Read More

Filtered Reference Fields in ServiceNow
Read More

ServiceNow performance testing tips
Read More

Monitor Dell Foglight Topology Churn with Splunk
Read More

Straight-Through Processing with ServiceNow
Read More

Splunk: Using Regex to Simplify Your Data
Read More

ServiceNow Choice List Dependencies
Read More

Tips for replaying RDP VuGen scripts in BSM or LoadRunner
Read More

Incorporating iSPI metric reports into MyBSM dashboard pages
Read More

Using SV contexts to simulate stored data
Read More

What’s new in LoadRunner 12.02
Read More

Recycle Bin for Quality Center
Read More

LoadRunner Correlation with web_reg_save_param_regexp
Read More

LoadRunner 11.52
Read More

QC for Testers – Quiz
Read More

Agile Performance Tuning with HP Diagnostics
Read More

What’s new in HP Service Virtualization 2.30
Read More

Understanding LoadRunner Virtual User Days (VUDs)
Read More

Problems recording HTTPS with VuGen
Read More

Improving the management and efficiency of QTP execution
Read More

Performance testing Oracle WebCenter with LoadRunner
Read More

Generating custom reports with Quality Center OTA using Python
Read More

Asynchronous Communication: Scripting For Cognos
Read More

How to fix common VuGen recording problems
Read More

Monitoring Active Directory accounts with HP BAC
Read More

URL Attachments in Quality Center
Read More

What’s new in LoadRunner 11.00?
Read More

Restore old License Usage stats after upgrading Quality Center
Read More

Changing LoadRunner/VuGen log options at runtime
Read More

Restricting large attachments in Quality Center
Read More

Retrieving Quality Center user login statistics
Read More

A comparison of open source load testing tools
...
Read More

Worst practices in performance testing
Read More

LoadRunner Sales Questions
Read More

LoadRunner Analysis: Hints and tips
Read More

LoadRunner in Windows 7
HP Loadrunner 11 is now available. This new version now natively supports Windows 7 and Windows Server 2008. I ...
Read More

Using the QuickTest Professional “commuter” license
Read More

Installing HP Diagnostics
Read More

Understanding LoadRunner licensing
Read More

VuGen scripting for YouTube video
Read More

Creating a Web + MMS vuser
Read More

Why you should use backwards dates
Read More

How to get the host’s IP address from within VuGen
Read More

VuGen scripting for BMC Remedy Action Request System 7.1
Read More

Unique usernames for BPM scripts
Read More

Mapping drives for LoadRunner Windows monitoring
Read More

VuGen feature requests
Read More

LoadRunner script completion checklist
Read More

Querying Quality Center user roles
Read More

Querying the Quality Center Database
Read More

HPSU 2009 Presentation – Performance Testing Web 2.0
Read More

Scaling HP Diagnostics
Read More

Global variables aren’t really global in LoadRunner
Read More

Client-side certificates for VuGen
Read More

Detect malicious HTML/JavaScript payloads with WebInspect (e.g. ASPROX, Gumblar, Income Iframe)
Read More

VuGen code snippets
Read More

Integrating QTP with Terminal Emulators
Read More

Why you must add try/catch blocks to Java-based BPM scripts
Read More

Querying a MySQL database with LoadRunner
Read More

ANZTB 2009 Presentation: Performance Testing Web 2.0
Read More

How to make QTP “analog mode” steps more reliable
Read More

Testing multiple browsers in a Standardized Operating Environment (SOE)
Read More

DNS-based load balancing for virtual users
Read More

What’s new in LoadRunner 9.50?
Read More

Calculating the difference between two dates or timestamps
Read More

The “is it done yet” loop
Read More

Think time that cannot be ignored
Read More

Understanding aggregate variance within LoadRunner analysis
Read More

Load balancing vusers without a load balancer
Read More

Harvesting file names with VuGen
Read More

Parameterising Unix/Posix timestamps in VuGen
Read More

HP Software trial license periods
Read More

How to handle HTTP POSTs with a changing number of name-value pairs
Read More

VuGen string comparison behaviour
Read More

Persistent data in VuGen with MySQL
Read More

How to write a Performance Test Plan
Read More

Unable to add virtual machine
To get ...
Read More

LoadRunner scripting languages
Read More

WDiff replacement for VuGen
Read More

Testing web services with a standard Web Vuser
Read More

Why your BPM scripts should use Download Filters
Read More

Querying your web server logs
Read More

Importing IIS Logs into SQL Server
Read More

QTP “Uninstall was not completed” problem
Read More

VuGen correlation for SAP Web Dynpro
Read More

How to save $500 on your HP software license
Read More

Testing and monitoring acronyms
Read More

Solving VuGen script generation errors
Read More

An introduction to SiteScope EMS Topology
Read More

Using the BAC JMX Console
Read More
Also I’d like to add one more resource, related to JMeter – http://blazemeter.com/blog
Hi there,
Great job.
Just wondering part 2 was hiding somewhere as I can;t seem to find it!
Stuart you’ve done great job. I clearly understand every topic in your open source code comparison. Some tools are better while compare to other tools. Every testing tool has its own criteria. We have to choose one which is suitable for our requirements. Thanks for your comparison and information.
Kiran
To me it is the great efforts by Stuart who has devoted time to figure out all such facts. I understand that every topic in itself could have open ended points to discuss but on the same side it is important to appreciate the one who actually has initiated it with most of the relevant information.
I appreciate the good work by Stuart.
Wish you all the best!
great work. but i think you should compare these at same page. there should be separate page for each comparison. referring is not good job to understand it better
Hi,
I think, that you are very incorrect, referring to curl-loader as to a “dumb” tool.
People are loading by very very complex scenario, including dynamic URLs, URLs based on responses, various authentications, etc
You can get more info from the tool README and also (less updated) FAQ page:
http://curl-loader.sourceforge.net/doc/faq.html
Antony Marcano, who runs Testing Reflections (which aggregates various testing blogs), wrote OpenSTA, the free performance testing tool, versus the big-guns… in 2004, which is still an interesting read.
Also, check out this large list of Open Source Performance Testing Tools (most of which are useless).
This Linux.com article on Using free software for HTTP load testing covers some of the “dumb” load testing tools – curl-loader, httperf (released free by HP labs), Siege, and Tsung (formerly known as IDX-Tsunami).