Bug 62251

We are running load tests continuously from multiple instances against multiple services and the results are sent to hostedgraphite.com using Backend Listener component with TextGraphiteMetricsSender.

From time to time we experience an issue where metrics stop being sent to hostedgraphite.com. Sometimes it is just a few instances but occasionally all of them appear to lose connection to HG.

With enabled debug logging we still see 'Wrote x metrics' in the logs from TextGraphiteMetricsSender.writeAndSendMetrics() and we don't see any errors or warns that would be logged in case of exceptions.

We have other applications sending data to HG, and those did not experience a similar outage at the same time, so we are assuming the issue is with JMeter.

After experimenting with PickleGraphiteMetricsSender we think we found the reason for this issue.

For the test using PickleGraphiteMetricsSender we see the following logs when such a loss of connection occurs:

2018-04-03 11:36:17,880 ERROR o.a.j.v.b.g.PickleGraphiteMetricsSender: Error writing to Graphite: Broken pipe (Write failed)
2018-04-03 11:36:17,880 DEBUG o.a.j.v.b.g.PickleGraphiteMetricsSender: Wrote 444 metrics

The connection is then reestablished and data appears correctly in HG.

In the same JMeter process we have another BackendListener that is using TextGraphiteMetricsSender that does not log any errors:

2018-04-03 11:36:17,884 DEBUG o.a.j.v.b.g.TextGraphiteMetricsSender: Wrote 66 metrics

This seems to come down to the use of PrintWriter in TextGraphiteMetricsSender. PrintWriter does not throw exceptions: https://docs.oracle.com/javase/8/docs/api/java/io/PrintWriter.html.

So when the connection fails in TextGraphiteMetricsSender it is never invalidated.

PickleGraphiteMetricsSender does not use PrintWriter, it uses OutputStreamWriter directly.