Understanding Aggregate Variance within LoadRunner Analysis

From time to time, you may notice variances within the Loadrunner Analysis tool. This is most apparent when reviewing the Transaction Response Time (Percentile) graph.
 
If you look at the following Transaction Summary Report:

Understanding Variances In Analysis - Summary Report

And look at the following Transaction Response Time (Percentile) graph:
UnderstandingVariancesInAnalysisPercentileGraph

 
As you can see there are some conflicting values:
 

Summary Report Graph Legend Graph Mouseover (Tooltip)
Average 50th Percentile Average Median 50th Percentile
7.718 5.728 7.942 5.830 5.728

 
Firstly, I’ll state that all these values are correct. The LoadRunner Analysis engine calculates all values correctly, and all variances can be easily explained by examining the data.
 

Median vs Mean vs Average vs Percentile

The first concept to understand and permanently implant into your brain, is the difference between these four aggregate types. An “Aggregate” is a single value that represents an underlying set of multiple values, commonly known aggregates are Average, Minimum and Maximum.
Most people understand what an “Average” is…that is the sum of a value set divided by the count of the value set. “Average” is also known as “Mean”. “Median” however is a different concept, a median value is calculated by sorting a set of data values from smallest to largest value then dividing this sorted set by two. The “Median” is the middle value at the place you divided the set. Median is identical to the 50th Percentile. Percentiles are an extension on the median concept, but instead of dividing the sorted value set by two…we instead divide the value set by a percentage (starting with the smallest value). So the 15th Percentile, refers to the value 15% along the sorted set…the 75th Percentile refers to the value 75% along the sorted set. Also the 0th Percentile is undefinable value.

Based on this information, we can safely separate the values into the following two groups.

Average Median/ 95th Percentile
Summary Report 7.718 5.728
Graph Legend 7.942 5.830
Graph Mouseover 5.728

 
Clearly the median value for the Graph Mouseover and the Summary Report are identical, so for the rest of the article I’ll will no longer refer to the Graph Mouseover.
 

Graph Values vs Raw Values

When the Percentile graph is generated (or any other graph for that matter), the analysis tool generates a small sample set for the graph. The sample set size is configurable in some graphs by setting the graph’s granularity, however the Percentile graph is fixed at a sample set of 100 data values…with each corresponding to a separate percentile value.
Therefore the values for Average and Median for the Graph Legend are calculated from the Graph data, not the Raw data.
Keeping this in mind, the Graph Average value is simply the average of the 1th, 2nd, 3rd…100th Percentile. Whilst this is graph average is close, it is not as accurate as the Summary Report average (which is calculated from the complete raw data set).
 
Okay, so up to this point we’ve explained all the value variations, except for the difference with the Median value. This problem is interesting, remember I mentioned that the median value is simply the sorted value set divided into two…and that the 0th Percentile is undefined. Well, the analysis tool equally divides the graph data result set into a set of values representing the 1th Percentile through to the 50th Percentile, and the 51st Percentile through to the 100th Percentile. As there is no clearly defined midpoint in this data set the analysis engine uses the next value…in this case the 51st Percentile.
So in summary. the Graph Median value actually the value of the 51st Percentile…not the 50th Percentile. This may be considered a minor bug.
 

In Summary

In summary, the graph value data is adequate for producing a graph…however for aggregate values I recommend only using the Summary Report data as the Summary report is the only report in the analysis engine that is guaranteed to display accurate data based on the complete data set.

2 comments

Which response time we should consider?

Average response time or 90 percentile response time

Thanks,
Rudra

Narasimha Rao

Hi,

What you have explained in above section was so good .. here I have a doubt , Like percentile graph average transaction response time also take a set of values to generate graph ?

Thanks
N V Narasimha Rao

Leave a Reply