Bug 62251 - TextGraphiteMetricsSender does not invalidate lost connections in case of network errors
Summary: TextGraphiteMetricsSender does not invalidate lost connections in case of net...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 2.13
Hardware: All All
: P2 normal (vote)
Target Milestone: JMETER_5.0
Assignee: JMeter issues mailing list
URL:
Keywords: FixedInTrunk
Depends on:
Blocks:
 
Reported: 2018-04-03 14:02 UTC by Zsolt Kecskemeti
Modified: 2018-09-05 15:35 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zsolt Kecskemeti 2018-04-03 14:02:14 UTC
We are running load tests continuously from multiple instances against multiple services and the results are sent to hostedgraphite.com using Backend Listener component with TextGraphiteMetricsSender.

From time to time we experience an issue where metrics stop being sent to hostedgraphite.com. Sometimes it is just a few instances but occasionally all of them appear to lose connection to HG.

With enabled debug logging we still see 'Wrote x metrics' in the logs from TextGraphiteMetricsSender.writeAndSendMetrics() and we don't see any errors or warns that would be logged in case of exceptions.

We have other applications sending data to HG, and those did not experience a similar outage at the same time, so we are assuming the issue is with JMeter.

After experimenting with PickleGraphiteMetricsSender we think we found the reason for this issue.

For the test using PickleGraphiteMetricsSender we see the following logs when such a loss of connection occurs:

2018-04-03 11:36:17,880 ERROR o.a.j.v.b.g.PickleGraphiteMetricsSender: Error writing to Graphite: Broken pipe (Write failed)
2018-04-03 11:36:17,880 DEBUG o.a.j.v.b.g.PickleGraphiteMetricsSender: Wrote 444 metrics

The connection is then reestablished and data appears correctly in HG.

In the same JMeter process we have another BackendListener that is using TextGraphiteMetricsSender that does not log any errors:

2018-04-03 11:36:17,884 DEBUG o.a.j.v.b.g.TextGraphiteMetricsSender: Wrote 66 metrics

This seems to come down to the use of PrintWriter in TextGraphiteMetricsSender. PrintWriter does not throw exceptions: https://docs.oracle.com/javase/8/docs/api/java/io/PrintWriter.html.

So when the connection fails in TextGraphiteMetricsSender it is never invalidated.

PickleGraphiteMetricsSender does not use PrintWriter, it uses OutputStreamWriter directly.
Comment 1 Philippe Mouawad 2018-04-05 11:03:33 UTC
Author: pmouawad
Date: Thu Apr  5 11:03:13 2018
New Revision: 1828416

URL: http://svn.apache.org/viewvc?rev=1828416&view=rev
Log:
Bug 62251 - TextGraphiteMetricsSender does not invalidate lost connections in case of network errors
Bugzilla Id: 62251

Modified:
    jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/graphite/TextGraphiteMetricsSender.java
    jmeter/trunk/xdocs/changes.xml
Comment 2 Philippe Mouawad 2018-04-05 11:04:24 UTC
Thanks for detailed report, can you test next build which will be available within 30 minutes from now and confirm wether issue is fixed.

Thanks
Comment 3 Philippe Mouawad 2018-04-11 19:26:18 UTC
Did you have any opportunity to test nightly build ?

Thanks for your feedback
Comment 4 Zsolt Kecskemeti 2018-04-11 21:09:59 UTC
Not yet, sorry, hopefully I will get a chance to test it tomorrow.
Comment 5 Zsolt Kecskemeti 2018-06-01 11:06:08 UTC
Sorry for the delay.

I had to setup Graphite in docker and break connection using iptables, but just didn't have the time to do it properly for ages.  

This is fixed now, the bad connection is invalidated correctly in TextGraphiteMetricsSender as well.