|Summary:||TextGraphiteMetricsSender does not invalidate lost connections in case of network errors|
|Product:||JMeter||Reporter:||Zsolt Kecskemeti <kecskemetizsolt>|
|Component:||Main||Assignee:||JMeter issues mailing list <issues>|
Description Zsolt Kecskemeti 2018-04-03 14:02:14 UTC
We are running load tests continuously from multiple instances against multiple services and the results are sent to hostedgraphite.com using Backend Listener component with TextGraphiteMetricsSender. From time to time we experience an issue where metrics stop being sent to hostedgraphite.com. Sometimes it is just a few instances but occasionally all of them appear to lose connection to HG. With enabled debug logging we still see 'Wrote x metrics' in the logs from TextGraphiteMetricsSender.writeAndSendMetrics() and we don't see any errors or warns that would be logged in case of exceptions. We have other applications sending data to HG, and those did not experience a similar outage at the same time, so we are assuming the issue is with JMeter. After experimenting with PickleGraphiteMetricsSender we think we found the reason for this issue. For the test using PickleGraphiteMetricsSender we see the following logs when such a loss of connection occurs: 2018-04-03 11:36:17,880 ERROR o.a.j.v.b.g.PickleGraphiteMetricsSender: Error writing to Graphite: Broken pipe (Write failed) 2018-04-03 11:36:17,880 DEBUG o.a.j.v.b.g.PickleGraphiteMetricsSender: Wrote 444 metrics The connection is then reestablished and data appears correctly in HG. In the same JMeter process we have another BackendListener that is using TextGraphiteMetricsSender that does not log any errors: 2018-04-03 11:36:17,884 DEBUG o.a.j.v.b.g.TextGraphiteMetricsSender: Wrote 66 metrics This seems to come down to the use of PrintWriter in TextGraphiteMetricsSender. PrintWriter does not throw exceptions: https://docs.oracle.com/javase/8/docs/api/java/io/PrintWriter.html. So when the connection fails in TextGraphiteMetricsSender it is never invalidated. PickleGraphiteMetricsSender does not use PrintWriter, it uses OutputStreamWriter directly.
Comment 1 Philippe Mouawad 2018-04-05 11:03:33 UTC
Author: pmouawad Date: Thu Apr 5 11:03:13 2018 New Revision: 1828416 URL: http://svn.apache.org/viewvc?rev=1828416&view=rev Log: Bug 62251 - TextGraphiteMetricsSender does not invalidate lost connections in case of network errors Bugzilla Id: 62251 Modified: jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/graphite/TextGraphiteMetricsSender.java jmeter/trunk/xdocs/changes.xml
Comment 2 Philippe Mouawad 2018-04-05 11:04:24 UTC
Thanks for detailed report, can you test next build which will be available within 30 minutes from now and confirm wether issue is fixed. Thanks
Comment 3 Philippe Mouawad 2018-04-11 19:26:18 UTC
Did you have any opportunity to test nightly build ? Thanks for your feedback
Comment 4 Zsolt Kecskemeti 2018-04-11 21:09:59 UTC
Not yet, sorry, hopefully I will get a chance to test it tomorrow.
Comment 5 Zsolt Kecskemeti 2018-06-01 11:06:08 UTC
Sorry for the delay. I had to setup Graphite in docker and break connection using iptables, but just didn't have the time to do it properly for ages. This is fixed now, the bad connection is invalidated correctly in TextGraphiteMetricsSender as well.