Bug 58583 - HTTP client fails to close connection if server misbehaves by not sending "connection: close", violating HTTP RFC 2616 / RFC 7230
Summary: HTTP client fails to close connection if server misbehaves by not sending "co...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 2.13
Hardware: All All
: P2 major (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on: 57319
Blocks:
  Show dependency tree
 
Reported: 2015-11-04 17:14 UTC by Paul Millar
Modified: 2016-02-26 22:51 UTC (History)
1 user (show)



Attachments
The JMeter configuration + wireshark capture of network traffic (22.28 KB, application/x-compressed-tar)
2015-11-04 17:14 UTC, Paul Millar
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Millar 2015-11-04 17:14:00 UTC
Created attachment 33256 [details]
The JMeter configuration + wireshark capture of network traffic

Hi,

I have an HTTP server I am testing.  The particular test pattern has the following behaviour (where some details are excluded for brevity)

    client>  GET /path/to/file [sent to server.example.org]
    server> 302 Found [Location: http://other.example.org:port/path/to/file]
    client>  GET /path/to/file [sent to other.example.org]
    server> 200 OK [with the entity as the response body]

That operation is repeated several times (using a Loop Controller), but the Thread Group has a single thread, so the requests are serialised.  In addition, I placed a one second delay in the loop (using a Constant Timer), so there is at least a one second delay between successive GET requests to other.example.org.

I've configured JMeter NOT to reuse connections (the "Use KeepAlive" checkbox is cleared).  This means that the client includes the `Connection: close` header in the requests.  Both server.example.org and other.example.org close the connection after sending the response.

The other point to note is that the port number of other.example.org is always the same.

Using wireshark, I have observed that JMeter never reacts to other.example.org sending the FIN packet: there is a corresponding ACK from the client (generated by the OS), but no FIN.  This leaves the TCP connection in the half-closed state: CLOSE_WAIT.

This is in contrast with JMeter's behaviour with server.example.org, where it does react to FIN and sends its own FIN, fully closing the TCP connection.

I have also observed JMeter attempting to reuse these half-closed connections and send a GET request on a CLOSE_WAIT connection.  Since the server has closed both ends of the socket, the server OS replies with a RST.  This forcefully closes the connection.

JMeter appears to have two modes of operation: one where it does not reuse connections and one where it does.

When JMeter is not attempting to reuse connections with other.example.org, fresh connections are made to the server for each request.  This is despite the existing connection not being closed on the client.  When operating like this, all GET requests to other.example.org succeed.

When attempting to reuse connections, the client sends a GET request on an existing connection (which is in CLOSE_WAIT state).  The server's OS responds with a RST.  This RST results in the next iteration's GET request to other.example.org creating a fresh TCP connection.  The subsequent GET request succeeds; however, it includes the "Connection: close" header, with the server closing the connection.  In the next iteration, the GET request to other.example.org will then fail (RST).  Thus, when attempting to reuse connections, the GET requests to other.example.org will alternate between succeeding and failing.

I have observed JMeter when running the Test Plan several times.  In some cases JMeter initially does not attempt to reuse connections and, after several iterations, it starts attempting to reuse connections.  For some Test Plan runs, JMeter always attempts to reuse connections.  However, once JMeter has started attempting to reuse connections, I have not seen it revert to not attempting to reuse connections.  I haven't been able to determine what factors control this behaviour.

Apart from being undesirable behaviour, this is clearly a violation of RFC 2616: where section 8.1.2 "Once a close has been signaled, the client MUST NOT send any more requests on that connection."

I'd be grateful if someone could investigate.  I have attached the JMeter configuration and a copy of the network traffic (where both client, server.example.org and other.example.org is 'localhost').

Cheers,

Paul.
Comment 1 Philippe Mouawad 2015-11-04 21:26:08 UTC
Hello,
Could you test with nightly build and check what's happening now ?
See Bug 57921

Thanks
Comment 2 Paul Millar 2015-11-05 14:56:12 UTC
Tested with latest nightly build (r1710693) and the problem persists.

From its description, bug 57921 is unrelated: that bug refers to a problem when "Keep Alive" enabled.  This bug is about a problem with "Keep Alive" switched off.
Comment 3 Rainer Jung 2015-11-08 12:20:31 UTC
Supporting the OPs request: RFC 7230 also says:

6.6.  Tear-down

...

   A client that sends a "close" connection option MUST NOT send further
   requests on that connection (after the one containing "close") and
   MUST close the connection after reading the final response message
   corresponding to this request.


In this case here, the the requests to the first and the second server contains the client side "Connection: close", but only the first server responds with "Connection: close". The second does not, but both close the connection. Although it is not nice from the second server to not send "Connection: close", it is allowed (same RFC section: "The server SHOULD send a "close" connection option in its final response on that connection.").

Looking at HTTPHC3Impl and HTTPHC4Impl it really seems we only set the "Connection: close" header but don't tell the HTTP clent to actually close the connection. I didn't check for the Java impl.

So there really seems to be a bug here.
Comment 4 Paul Millar 2015-11-10 08:08:25 UTC
You're quite correct in noting that the second server failed to send the "Connection: close" header in the response.  Independent of this problem, that should be fixed.
Comment 5 Philippe Mouawad 2015-12-20 21:21:48 UTC
Hi,
Isn't this issue:
https://issues.apache.org/jira/browse/HTTPCLIENT-1655

@Paul Millar, we upgraded today to HC4.5.1. 
Could you try nightly build tomorrow and give us some feedback ?
Thanks
Comment 6 Philippe Mouawad 2015-12-21 17:01:28 UTC
(In reply to Philippe Mouawad from comment #5)
> Hi,
> Isn't this issue:
> https://issues.apache.org/jira/browse/HTTPCLIENT-1655
> 
> @Paul Millar, we upgraded today to HC4.5.1. 
> Could you try nightly build tomorrow and give us some feedback ?
> Thanks

Forget my last comment.
I misunderstood the problem.

Now let me recap what I understand:
The issue is due to server not setting header "Connection: close", and JMeter as a side effect not closing the connection while it should .

Is my understanding correct ?
Thanks
Comment 7 Rainer Jung 2015-12-21 19:23:38 UTC
Yes, that was my impression when I looked at the ticket. I wrote

"Looking at HTTPHC3Impl and HTTPHC4Impl it really seems we only set the "Connection: close" header but don't tell the HTTP clent to actually close the connection. I didn't check for the Java impl."

If the *server* sends "Connection: close", then the client will automatically close the ocnnection, but only sending a "Connection: close" from the client side does not lead to automaticaly closing it in the client. IMHO we have to tell the client not only to send "Connection: close" but also to close the connection. Most servers do send "Connection: close" so often the problem doesn't occur.
Comment 8 Philippe Mouawad 2015-12-30 13:59:03 UTC
Hi,
It appears it was an issue in HTTPCORE:
https://issues.apache.org/jira/browse/HTTPCORE-397

Could you please test nightly build and report wether issue is fixed for you ?
Thanks
Comment 9 Philippe Mouawad 2015-12-30 14:02:27 UTC
Hi,
New update , fix is not in HTTPcore-4.4.4 so no need to test.

Regards
Philippe
Comment 10 Philippe Mouawad 2016-01-02 23:46:56 UTC
Fix should be available in HTTPClient 4.5.2 as per commit:

- https://github.com/ok2c/httpclient/commit/8f38a042494482e18f0129baf4f5118067acedde


To have it in place, we will need to update HTTPHC4Impl to use the DefaultClientConnectionReuseStrategy.INSTANCE:
((AbstractHttpClient) httpClient).setReuseStrategy(DefaultClientConnectionReuseStrategy.INSTANCE);
Comment 11 Philippe Mouawad 2016-02-26 22:48:23 UTC
Author: pmouawad
Date: Fri Feb 26 22:35:18 2016
New Revision: 1732569

URL: http://svn.apache.org/viewvc?rev=1732569&view=rev
Log:
Upgrade to HttpClient/httpmime 4.5.2 which fixes at the same time:
- Bug 58881 - HTTP Request : HTTPHC4Impl shows exception when server uses "deflate" compression
- Bug 58583 - HTTP client fails to close connection if server misbehaves by not sending "connection: close", violating HTTP RFC 2616 / RFC 7230
- Bug 56358 - Cookie manager supports cross port cookies and RFC6265
- Bug 57319 - Upgrade to HttpClient 4.5.2
Bugzilla Id: 57319

Modified:
    jmeter/trunk/build.properties
    jmeter/trunk/eclipse.classpath
    jmeter/trunk/lib/   (props changed)
    jmeter/trunk/lib/aareadme.txt
    jmeter/trunk/licenses/bin/README.txt
    jmeter/trunk/res/maven/ApacheJMeter_parent.pom
    jmeter/trunk/test/src/org/apache/jmeter/protocol/http/control/TestCookieManager.java
    jmeter/trunk/test/src/org/apache/jmeter/protocol/http/sampler/TestHTTPSamplersAgainstHttpMirrorServer.java
    jmeter/trunk/xdocs/changes.xml
Comment 12 Philippe Mouawad 2016-02-26 22:51:58 UTC
Author: pmouawad
Date: Fri Feb 26 22:50:37 2016
New Revision: 1732574

URL: http://svn.apache.org/viewvc?rev=1732574&view=rev
Log:
Bug 58950 - NoHttpResponseException when Pause between samplers exceeds keepalive sent by server
Bugzilla Id: 58950

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
    jmeter/trunk/xdocs/changes.xml