Bug 57321 - BackendListener reports partial results in master-slave configuration (nightly build r1642603)
Summary: BackendListener reports partial results in master-slave configuration (nightl...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 2.13
Hardware: PC All
: P2 major (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on:
Blocks: 55932
  Show dependency tree
 
Reported: 2014-12-06 21:27 UTC by Chaitanya Bhatt
Modified: 2016-09-27 06:36 UTC (History)
2 users (show)



Attachments
Grafana output (274.26 KB, image/jpeg)
2014-12-11 22:48 UTC, Chaitanya Bhatt
Details
Final graphana result screenshot confirming proportional increase in throughput after adding more slave instances (129.07 KB, image/jpeg)
2014-12-15 00:55 UTC, Chaitanya Bhatt
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Chaitanya Bhatt 2014-12-06 21:27:03 UTC
BackendListener reports metrics from only 1 slave, it fails to aggregate results across multiple slave instance.

I had 4 slave instance which generates about 1000 TPS (aggregate), with each instance contributing 250TPS (responses/sec). However, with InfluxDB as the backend client and Jmeter BackendListener as the data source, I see only 250 TPS as the aggregate TPS instead of 1000 TPS.

Here is the influxdb query i used: "select value from  jmeter.cumulated.total where time > now() -5m"

InfluxDB output:
time	sequence_number	value
1417900538000	1	250
1417900537000	1	250
....


ThreatGroup
|
|
TransactionController
---|
   |
   HTTPTestSample
|
|
BackendListener






BackendListener Config is as below:



     <BackendListener guiclass="BackendListenerGui" testclass="BackendListener" testname="Backend Listener" enabled="true">
          <elementProp name="arguments" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="graphiteMetricsSender" elementType="Argument">
                <stringProp name="Argument.name">graphiteMetricsSender</stringProp>
                <stringProp name="Argument.value">org.apache.jmeter.visualizers.backend.graphite.TextGraphiteMetricsSender</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="graphiteHost" elementType="Argument">
                <stringProp name="Argument.name">graphiteHost</stringProp>
                <stringProp name="Argument.value">jmeterInfluxDBHost.ie.bigcorp.net</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="graphitePort" elementType="Argument">
                <stringProp name="Argument.name">graphitePort</stringProp>
                <stringProp name="Argument.value">2003</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="rootMetricsPrefix" elementType="Argument">
                <stringProp name="Argument.name">rootMetricsPrefix</stringProp>
                <stringProp name="Argument.value">jmeter.</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="summaryOnly" elementType="Argument">
                <stringProp name="Argument.name">summaryOnly</stringProp>
                <stringProp name="Argument.value">false</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="samplersList" elementType="Argument">
                <stringProp name="Argument.name">samplersList</stringProp>
                <stringProp>HTTPTestSample</stringProp>
              </elementProp>
              <elementProp name="percentiles" elementType="Argument">
                <stringProp name="Argument.name">percentiles</stringProp>
                <stringProp name="Argument.value">90;95;99</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="classname">org.apache.jmeter.visualizers.backend.graphite.GraphiteBackendListenerClient</stringProp>
        </BackendListener>
Comment 1 Philippe Mouawad 2014-12-06 22:15:31 UTC
Hi,
Are you using distributed mode or running 4 independant jmeter instances ?

Thanks
Comment 2 Chaitanya Bhatt 2014-12-06 23:10:56 UTC
(In reply to Philippe Mouawad from comment #1)
> Hi,
> Are you using distributed mode or running 4 independant jmeter instances ?
> 
> Thanks

I am running the test in distributed mode. 

Please note that, I confirmed the actual response/second(Throughput) is equal to 1000 using server side monitoring tool(new relic). 

Feel free to let me know any more tests/artifacts you may require to resolve this issue.

Regards,
Chaitanya  M Bhatt
Comment 3 Philippe Mouawad 2014-12-07 20:38:59 UTC
Date: Sun Dec  7 20:38:33 2014
New Revision: 1643716

URL: http://svn.apache.org/r1643716
Log:
Bug 57321 - BackendListener reports wrong number of Active Users in master-slave configuration (nightly build r1642603)
Bugzilla Id: 57321

Modified:
    jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/UserMetric.java
    jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/graphite/GraphiteBackendListenerClient.java
    jmeter/trunk/xdocs/changes.xml
Comment 4 Philippe Mouawad 2014-12-09 20:56:13 UTC
@Chaitanya Bhatt, 
could you give us feedback on the new nightly build ?
Thanks
Regards
Comment 5 Chaitanya Bhatt 2014-12-09 22:52:05 UTC
(In reply to Philippe Mouawad from comment #4)
> @Chaitanya Bhatt, 
> could you give us feedback on the new nightly build ?
> Thanks
> Regards


Philippe,

You have done a mind blowing job! I verified that the cumulated  metrics for response time and users are working fine. However, I am still not sure about the "success" and "total" columns. I see a huge discrepancy in the "total success / second" reported by BackendListener and the perf wrapper which I use to aggregate results form slave instances.  

I am doing a lot of R&D on this, I will promptly let you know the results.

Once more, this feature is such a big leap for Jmeter; I just cannot enumerate the number of benefits this feature offers the user community.

Brilliant work!!!

Chaitanya M Bhatt
Comment 6 Chaitanya Bhatt 2014-12-11 22:48:50 UTC
Created attachment 32285 [details]
Grafana output

Notice that there is no effect on the Total transactions reported to InfluxDB by the BackendListener even after bumping up the thread.
Comment 7 Chaitanya Bhatt 2014-12-11 22:57:30 UTC
(In reply to Chaitanya Bhatt from comment #6)
> Created attachment 32285 [details]
> Grafana output
> 
> Notice that there is no effect on the Total transactions reported to
> InfluxDB by the BackendListener even after bumping up the thread.

Philippe,

To add more info on this issue: 
The cumulated total,success an failure value does not seem to show values from more than 1 jmeter agent when executed in distributed mode. 
I confirmed this by running a query against InfluxDb:
select sum(value) from jmeter.cumulated.total where time > now() - 10m
immediately after finishing the test and saw that the query returned only half as much many of transaction executed.

For comparison purpose I had the summary report listener turned ON, which reported accurate metrics and this helped me prove that there is indeed a discrepancy in BackendListener. 

However, the fix in r1643716 seems to have fixed the cumulated thread count issue. It does reflect total thread accurately when run in distributed mode.

I also suspect individual success and failure metrics are also not accurate. However, I am
Comment 8 UbikLoadPack support 2014-12-12 22:30:53 UTC
Hi Chaitanya,
I found issue, I will be implementing the fix in upcoming days.

Thanks for your tests
Comment 9 Chaitanya Bhatt 2014-12-13 07:23:55 UTC
Philippe,

From a feature design point of view, do you think it would make sense for the slave instances in distributed mode to directly write "individual sampler results" to graphite and for cumulated metrics each child instance should cumulate its results and directly send it graphite as well, leaving Jmeter master server instance completely free from doing any of these tasks. This design would be great from a scalability point of view. 

If it is too late to make such changes, then we should at least consider to make this as an option in the BackendListener, where user could choose to select Jmeter master to do all the sampler result collection,aggregation and sending of data to graphite OR could choose an option where all the slave instances could directly send data to graphite. 

In my opinion, the key to successful adoption of BackendListener is if it enables Jmeter to overcome its biggest limitation, which is, the ability to provide a way for the user to collect result metrics without taking a hit on the performance of Jmeter server itself.

Kindly let me know your opinion. If this is not the right place to carry on such discussion, please let me know the right forum where I can bring up this discussion.


Thanks
Chaitanya M Bhatt
Comment 10 Philippe Mouawad 2014-12-13 08:01:30 UTC
Hi Chaitanya,
First thanks for your early tests, they really help a lot.
Regarding your proposal, can you raise a discussion on dev mailing list mentionning this Bugzilla.
Having each backendlistener its data is possible I think but it is an important change.
Maybe the next step once this feature is fully operational (not a lot work as is seems)

Regards
Philippe
Comment 11 Philippe Mouawad 2014-12-14 21:58:33 UTC
Date: Sun Dec 14 21:57:17 2014
New Revision: 1645532

URL: http://svn.apache.org/r1645532
Log:
Bug 57321 - BackendListener reports partial results in master-slave configuration (nightly build r1642603)
Bugzilla Id: 57321

Modified:
    jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/BackendListener.java
    jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/SamplerMetric.java
    jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/graphite/GraphiteBackendListenerClient.java
    jmeter/trunk/xdocs/changes.xml
Comment 12 Chaitanya Bhatt 2014-12-15 00:52:58 UTC
Woohooo!!! Issue Fixed! Result looks beautiful! Awesome! Awesome! Awesome!

Great work!

Regards,
Chaitanya M Bhatt
Comment 13 Chaitanya Bhatt 2014-12-15 00:55:43 UTC
Created attachment 32289 [details]
Final graphana result screenshot confirming proportional increase in throughput after adding more slave instances
Comment 14 Nimish V 2015-07-14 20:29:16 UTC
I am using JMeter in client-server mode and using generic Backend Listener component for Graphite. When script is executed in standard mode, data is being sent back to Graphite DB. But, when script executed in client-server mode, Backend Listener does not send data to Graphite. Test is running fine as we are generating report and all data from all the server machines are successfully being aggregated by client machine after test.

Can you guys help me if I need to setup anything additional to have Backend Listener work in client-server mode for JMeter version 2.13?
Comment 15 sombat.wongsrisupphakul 2016-09-27 06:16:29 UTC
Hi Nimish V,

It seems to be an issue that we cannot use any variables or properties in Backend listener context. It works when I used static values.

Example,
We cannot use: graphiteHost = ${graphiteHost}
It works when: graphiteHost = 192.168.99.100
Comment 16 sombat.wongsrisupphakul 2016-09-27 06:19:02 UTC
(In reply to sombat.wongsrisupphakul from comment #15)
> Hi Nimish V,
> 
> It seems to be an issue that we cannot use any variables or properties in
> Backend listener context. It works when I used static values.
> 
> Example,
> We cannot use: graphiteHost = ${graphiteHost}
> It works when: graphiteHost = 192.168.99.100

This issue happen on distributed mode (Master-Slave) only.
Comment 17 Philippe Mouawad 2016-09-27 06:36:47 UTC
(In reply to sombat.wongsrisupphakul from comment #16)
> (In reply to sombat.wongsrisupphakul from comment #15)
> > Hi Nimish V,
> > 
> > It seems to be an issue that we cannot use any variables or properties in
> > Backend listener context. It works when I used static values.
> > 
> > Example,
> > We cannot use: graphiteHost = ${graphiteHost}
> > It works when: graphiteHost = 192.168.99.100
> 
> This issue happen on distributed mode (Master-Slave) only.

Please use user mailing list for such questions.
This bug is already fixed and has no relation with your problem.
Thank you