By default it is set to fixed which causes JMeter report statistics for last {{ backend_metrics_large_window }} (default 5000) number of samples, even though {{ count }} is reset every 1s (default for graphite backend listener used in this case) This is a sequence of JMeter requests (4) gathered over 5 minutes ( (elapsed [ms], timestamp): 3249, 5/8/2020 14:27:44 2761, 5/8/2020 14:28:57 2920, 5/8/2020 14:31:25 2720, 5/8/2020 14:31:33 This gives mean elapsed [ms]= (3249+2761+2920+2720) / 4 = 2912.5 Now, what is sent via Graphite Backend listener is: (mean, timestamp, count) 3249, 5/8/2020 14:27, 1 3005, 5/8/2020 14:29, 1 2976.7, 5/8/2020 14:31, 1 2912.5, 5/8/2020 14:31, 1 You can easily check, that each subsequent mean sent by graphite listener is: sum(elaped so far) / sum(counts so far), i.e 3249 = 3249 / 1 3005 = (3249+2761) / 2 2976.7 ~= (3249+2761+2920) / 3 and yes, we've been already there: (3249+2761+2920+2720) / 4 = 2912.5 I have 1) backend_metrics_window_mode=fixed (default) and 2) backend_metrics_large_window = 5000 (default) This configuration results in samples mean calculated using backend_metrics_large_window # of last samples, while counter of samples will be reset each graphite listener interval (1s) – links below: Source code: https://github.com/apache/jmeter/blob/master/src/components/src/main/java/org/apache/jmeter/visualizers/backend/SamplerMetric.java#L156 – do not clear descriptive statistics which holds sample times https://github.com/apache/jmeter/blob/master/src/components/src/main/java/org/apache/jmeter/visualizers/backend/SamplerMetric.java#L168 – but clears sample count ! This is why we see count=1 at each graphite aggregate. This way, having only graphite listener I can't calculate mean response time over the whole test period by summing 'mean' and dividing by sum of 'count. I know, that using 'timed' mode will fix the problem for me, yet still I think consequence of using default settings are surprising. I think mentioning this caveat in documentation might safe some folks investigation time :)
Hi Team, Do you think this behaviour should be changed or documented ? Thanks
As the current behaviour is surprising to (at least one) user, we should enhance the documentation for it. What change of behaviour do you propose?
Hi Felix, I would allow only the behaviour of 'TIMED' mode and removed support for 'FIXED'. Except for the issue described originally in the ticket which was found during test, I have I suspect other problems by looking at the code of SamplerMetric class. I believe FIXED mode will result in inconsistency between results provided by getKoPercentile, getOkPercentile on one hand and getAllPercentile on the other because * get{Ko,Ok}Percentile method use {ok,ko}ResponseStats which store up to LARGE_SLIDING_WINDOW_SIZE samples (=5000 by default) and * getAllPercentile method uses pctResponseStats member which stores up to SLIDING_WINDOW_SIZE samples (=100 by default) E.g. we sent 5000 samples in the test, which had response times 1000ms for all but last 101 samples, which had elapsed time 500ms. For simplicity, let's assume there were only successful samples. getOkPercentile(90) in FIXED mode will return 1000ms for the last period of the test while getAllPercentile(90) will return 500ms although all requests were Ok in the test. This is because getAllPercentile will catch only last 100 samples, while getOkPercentile will catch 5000. Allowing for non successful request doesn't invalidate this reasoning, there will be only more special cases, when it could work. Ok, what if we enforced that pctResponseStats have always the same window as {ko,ok}ResponseStats. Now things improve if we have only successful or only failed samples, but again it doesn't work in general. E.g. Let's assume the unified window size is 100. We run during the test 200 requests. First request lasted 1ms, second 2ms, third 3ms,..., 200th one lasted 200ms. Each odd request was successful, each even was failing. So successful samples are 1, 3, 5, ..., 199 ms long and failing ones 2, 4, 6, ... 200 ms long. getKoPercentile(90) = 180ms getOkPercentile(90) = 179ms getAllPercentile(90) = 190ms > getKoPercentile(90) > getOkPercentile(90) which seems impossible unless we recall, that getAllPercentile covers only last 100 samples with times 101, 102 ..., 200. How about test plans with multiple samplers reported separately? E.g. with the window size 100, if the sampler A has throughput 100/min and sampler B has throughput 10/min, the percentile for sampler A will describe what happened in last minute, while for sampler B it will describe what happened in last 10 minutes (10 req/min x10min = 100 req = window size). A bit complicated to interpret. I think leaving 'TIMED' mode as the only option helps us avoid all this issues. On the other hand, if for some reason someone really needs to aggregate over fixed number of samples, custom listener can be coded or .jtl file used.
By leaving 'TIMED' mode only I understand cleaning all the DescriptiveStatistics members of SampleMetric class including pctResponseStats every refresh interval, or rather removing pctResponseStats and replace it with with allResponseStats.
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/5327