Bug 64240

Summary: http 0.9 request return nothing
Product: Tomcat 8 Reporter: dingli <382188587>
Component: ConnectorsAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal    
Priority: P2    
Version: 8.5.53   
Target Milestone: ----   
Hardware: All   
OS: All   

Description dingli 2020-03-19 03:18:43 UTC
when send http 0.9 request to tomcat, the server will return nothing and close the TCP connection.
the request contains only one line:
"GET /CRLF"
there is no space between / and CRLF

if there is one space between / and CRLF, tomcat will return content in response

but this bug can't be reproduced every time
after restart tomcat, it maybe disappear
and may next time restart, it maybe appear
Comment 1 Mark Thomas 2020-03-19 07:56:51 UTC
Tomcat tightened up the HTTP 0.9 parsing. It looks like there is an issue with requests of the form:

GET / LF

Prior to the parsing changes, this would have been accepted as a (malformed) HTTP 0.9 request. It is now rejected as an invalid HTTP 1.1 request. The HTTP 0.9 spec allows either way of handling the request.

I'll take a look to see if the parsing can be relaxed to accept requests like this without creating problems elsewhere.

I'm curious. What clients are you using that sent malformed HTTP 0.9 requests?
Comment 2 dingli 2020-03-19 08:09:21 UTC
(In reply to Mark Thomas from comment #1)
> Tomcat tightened up the HTTP 0.9 parsing. It looks like there is an issue
> with requests of the form:
> 
> GET / LF
> 
> Prior to the parsing changes, this would have been accepted as a (malformed)
> HTTP 0.9 request. It is now rejected as an invalid HTTP 1.1 request. The
> HTTP 0.9 spec allows either way of handling the request.
> 
> I'll take a look to see if the parsing can be relaxed to accept requests
> like this without creating problems elsewhere.
> 
> I'm curious. What clients are you using that sent malformed HTTP 0.9
> requests?


my tomcat is behinde one F5 load balancer, F5 have http monitor to check the tomcat's health. The default send string of F5 http monitor is "GET /CRLF", total 7 bytes.  when tomcat close the socket without return anything, F5 think tomcat is out of service.
below is the tcpdump of F5 monitor connection:

22:45:04.215888 IP 172.16.97.5.15379 > 172.16.28.103.ircu-4: Flags [S], seq 3311525713, win 5840, options [mss 1460,sackOK,TS val 3987447303 ecr 0,nop,wscale 7], length 0
	0x0000:  4500 003c 20ba 4000 3f06 4575 ac10 6105  E..<..@.?.Eu..a.
	0x0010:  ac10 1c67 3c13 1a0c c561 df51 0000 0000  ...g<....a.Q....
	0x0020:  a002 16d0 d51c 0000 0204 05b4 0402 080a  ................
	0x0030:  edab 9e07 0000 0000 0103 0307            ............
22:45:04.215910 IP 172.16.28.103.ircu-4 > 172.16.97.5.15379: Flags [S.], seq 3991552491, ack 3311525714, win 14480, options [mss 1460,sackOK,TS val 3320472856 ecr 3987447303,nop,wscale 7], length 0
	0x0000:  4500 003c 0000 4000 4006 652f ac10 1c67  E..<..@.@.e/...g
	0x0010:  ac10 6105 1a0c 3c13 edea 41eb c561 df52  ..a...<...A..a.R
	0x0020:  a012 3890 5872 0000 0204 05b4 0402 080a  ..8.Xr..........
	0x0030:  c5ea 6518 edab 9e07 0103 0307            ..e.........
22:45:04.217823 IP 172.16.97.5.15379 > 172.16.28.103.ircu-4: Flags [.], ack 1, win 46, options [nop,nop,TS val 3987447305 ecr 3320472856], length 0
	0x0000:  4500 0034 20bb 4000 3f06 457c ac10 6105  E..4..@.?.E|..a.
	0x0010:  ac10 1c67 3c13 1a0c c561 df52 edea 41ec  ...g<....a.R..A.
	0x0020:  8010 002e bf9e 0000 0101 080a edab 9e09  ................
	0x0030:  c5ea 6518                                ..e.
22:45:04.217849 IP 172.16.97.5.15379 > 172.16.28.103.ircu-4: Flags [P.], seq 1:8, ack 1, win 46, options [nop,nop,TS val 3987447305 ecr 3320472856], length 7
	0x0000:  4500 003b 20bc 4000 3f06 4574 ac10 6105  E..;..@.?.Et..a.
	0x0010:  ac10 1c67 3c13 1a0c c561 df52 edea 41ec  ...g<....a.R..A.
	0x0020:  8018 002e eb1c 0000 0101 080a edab 9e09  ................
	0x0030:  c5ea 6518 4745 5420 2f0d 0a              ..e.GET./..
22:45:04.217858 IP 172.16.28.103.ircu-4 > 172.16.97.5.15379: Flags [.], ack 8, win 114, options [nop,nop,TS val 3320472858 ecr 3987447305], length 0
	0x0000:  4500 0034 24f6 4000 4006 4041 ac10 1c67  E..4$.@.@.@A...g
	0x0010:  ac10 6105 1a0c 3c13 edea 41ec c561 df59  ..a...<...A..a.Y
	0x0020:  8010 0072 bf51 0000 0101 080a c5ea 651a  ...r.Q........e.
	0x0030:  edab 9e09                                ....
22:45:04.219749 IP 172.16.28.103.ircu-4 > 172.16.97.5.15379: Flags [F.], seq 1, ack 8, win 114, options [nop,nop,TS val 3320472860 ecr 3987447305], length 0
	0x0000:  4500 0034 24f7 4000 4006 4040 ac10 1c67  E..4$.@.@.@@...g
	0x0010:  ac10 6105 1a0c 3c13 edea 41ec c561 df59  ..a...<...A..a.Y
	0x0020:  8011 0072 bf4e 0000 0101 080a c5ea 651c  ...r.N........e.
	0x0030:  edab 9e09                                ....
22:45:04.220836 IP 172.16.97.5.15379 > 172.16.28.103.ircu-4: Flags [F.], seq 8, ack 2, win 46, options [nop,nop,TS val 3987447308 ecr 3320472860], length 0
	0x0000:  4500 0034 20bd 4000 3f06 457a ac10 6105  E..4..@.?.Ez..a.
	0x0010:  ac10 1c67 3c13 1a0c c561 df59 edea 41ed  ...g<....a.Y..A.
	0x0020:  8011 002e bf8e 0000 0101 080a edab 9e0c  ................
	0x0030:  c5ea 651c                                ..e.
22:45:04.220850 IP 172.16.28.103.ircu-4 > 172.16.97.5.15379: Flags [.], ack 9, win 114, options [nop,nop,TS val 3320472861 ecr 3987447308], length 0
	0x0000:  4500 0034 24f8 4000 4006 403f ac10 1c67  E..4$.@.@.@?...g
	0x0010:  ac10 6105 1a0c 3c13 edea 41ed c561 df5a  ..a...<...A..a.Z
	0x0020:  8010 0072 bf49 0000 0101 080a c5ea 651d  ...r.I........e.
	0x0030:  edab 9e0c                                ....

you can see the real payload data is "47 45 54 20 2f 0d 0a" (7 bytes in Hex)
Comment 3 dingli 2020-03-19 08:14:40 UTC
(In reply to Mark Thomas from comment #1)
> Tomcat tightened up the HTTP 0.9 parsing. It looks like there is an issue
> with requests of the form:
> 
> GET / LF
> 
> Prior to the parsing changes, this would have been accepted as a (malformed)
> HTTP 0.9 request. It is now rejected as an invalid HTTP 1.1 request. The
> HTTP 0.9 spec allows either way of handling the request.
> 
> I'll take a look to see if the parsing can be relaxed to accept requests
> like this without creating problems elsewhere.
> 
> I'm curious. What clients are you using that sent malformed HTTP 0.9
> requests?


I think the "GET /CRLF" is a valid HTTP 0.9 requet
in RFC1945:

Simple-Request  = "GET" SP Request-URI CRLF

and one strange thing is sometime tomcat will response content for this kind of request and sometime won't.
Comment 4 Remy Maucherat 2020-03-19 08:24:34 UTC
Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to still have it (clients which insist would still get something back).
Comment 5 Michael Osipov 2020-03-19 08:36:02 UTC
(In reply to Remy Maucherat from comment #4)
> Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> still have it (clients which insist would still get something back).

+1 for this in Tomcat 10.
Comment 6 dingli 2020-03-19 08:41:07 UTC
(In reply to Remy Maucherat from comment #4)
> Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> still have it (clients which insist would still get something back).

It is F5's default setting :(
and we can't change F5's setting by ourselves :(
Comment 7 Michael Osipov 2020-03-19 08:44:16 UTC
(In reply to dingli from comment #6)
> (In reply to Remy Maucherat from comment #4)
> > Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> > still have it (clients which insist would still get something back).
> 
> It is F5's default setting :(
> and we can't change F5's setting by ourselves :(

You have paid for a commercial product you should have support for that. mod_proxy does a simple HEAD request against / with HTTP/1.1. Works flawlessly.
Comment 8 dingli 2020-03-19 08:50:17 UTC
(In reply to Michael Osipov from comment #7)
> (In reply to dingli from comment #6)
> > (In reply to Remy Maucherat from comment #4)
> > > Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> > > still have it (clients which insist would still get something back).
> > 
> > It is F5's default setting :(
> > and we can't change F5's setting by ourselves :(
> 
> You have paid for a commercial product you should have support for that.
> mod_proxy does a simple HEAD request against / with HTTP/1.1. Works
> flawlessly.

yes, we can make a ticket to network maintenance department and wait.
it is a long time process :(

anyway, the upgrade from 8.5.50 to 8.5.51/8.5.53 will break some tomcat instance behind F5 load balancer.
Comment 9 Remy Maucherat 2020-03-19 09:02:28 UTC
Anyway, about the actual "issue", I don't see how it happens right now.
Comment 10 Remy Maucherat 2020-03-19 13:52:24 UTC
I fail to see the problem so I added a test case to test HTTP/0.9 support (using "GET /CRLF"), and it works for me.
Comment 11 Christopher Schultz 2020-03-19 19:37:03 UTC
(In reply to Michael Osipov from comment #5)
> (In reply to Remy Maucherat from comment #4)
> > Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> > still have it (clients which insist would still get something back).
> 
> +1 for this in Tomcat 10.

-1 for this in Tomcat 10.

There are a huge number of stupid devices in the world that nobody can change. Sure, you can tell everyone using NoName-brand WiFi light bulbs that HTTP/0.9 is dead, but you can't tell F5 that HTTP/0.9 is dead and they had better upgrade.
Comment 12 Michael Osipov 2020-03-19 22:19:42 UTC
(In reply to Christopher Schultz from comment #11)
> (In reply to Michael Osipov from comment #5)
> > (In reply to Remy Maucherat from comment #4)
> > > Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> > > still have it (clients which insist would still get something back).
> > 
> > +1 for this in Tomcat 10.
> 
> -1 for this in Tomcat 10.
> 
> There are a huge number of stupid devices in the world that nobody can
> change. Sure, you can tell everyone using NoName-brand WiFi light bulbs that
> HTTP/0.9 is dead, but you can't tell F5 that HTTP/0.9 is dead and they had
> better upgrade.

This is the same discussion as with the dropped reason phrase. HTTP/1.1. has been introduced almost 21 years ago. If F5 Networks did not manage to update their code in 21 years, they won't do it ever.
Comment 13 dingli 2020-03-20 05:10:38 UTC
(In reply to Michael Osipov from comment #12)
> (In reply to Christopher Schultz from comment #11)
> > (In reply to Michael Osipov from comment #5)
> > > (In reply to Remy Maucherat from comment #4)
> > > > Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> > > > still have it (clients which insist would still get something back).
> > > 
> > > +1 for this in Tomcat 10.
> > 
> > -1 for this in Tomcat 10.
> > 
> > There are a huge number of stupid devices in the world that nobody can
> > change. Sure, you can tell everyone using NoName-brand WiFi light bulbs that
> > HTTP/0.9 is dead, but you can't tell F5 that HTTP/0.9 is dead and they had
> > better upgrade.
> 
> This is the same discussion as with the dropped reason phrase. HTTP/1.1. has
> been introduced almost 21 years ago. If F5 Networks did not manage to update
> their code in 21 years, they won't do it ever.

F5 can send HTTP/1.0 or HTTP/1.1 request in http monitor. but ths stupid thing is  the default setting is HTTP/0.9 

we have change the F5 setting last night,now it works for tomcat 8.5.53
Comment 14 dingli 2020-03-20 05:27:11 UTC
(In reply to Remy Maucherat from comment #10)
> I fail to see the problem so I added a test case to test HTTP/0.9 support
> (using "GET /CRLF"), and it works for me.

yesterday I can reproduce the bug in my local windows machine and local Ubuntu VM sometime(NOT everytime)
but I can't reproduce it today :(
maybe it is related with some corner case ? such as uninitialized variable or memory?

But for tomcat 8.5.51, I can reproduce it everytime
tomcat 8.5.51 won't send content for "GET /CRLF" same as 8.5.53
the difference is 8.5.53 will close the socket immediately  
8.5.51 will keep the socket open and close the socket after 20 seconds

below is the tomcat 8.5.51 catalina log:

20-Mar-2020 13:13:37.718 FINE [http-nio-8080-exec-3] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Processing socket [org.apache.tomcat.util.net.NioChannel@6fc5100f:java.nio.channels.SocketChannel[connected local=/192.168.31.50:8080 remote=/192.168.31.6:51783]] with status [OPEN_READ]
20-Mar-2020 13:13:37.719 FINE [http-nio-8080-exec-3] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Found processor [null] for socket [org.apache.tomcat.util.net.NioChannel@6fc5100f:java.nio.channels.SocketChannel[connected local=/192.168.31.50:8080 remote=/192.168.31.6:51783]]
20-Mar-2020 13:13:37.719 FINE [http-nio-8080-exec-3] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Popped processor [org.apache.coyote.http11.Http11Processor@2d8c7e6c] from cache
20-Mar-2020 13:13:37.719 FINE [http-nio-8080-exec-3] org.apache.coyote.http11.Http11InputBuffer.parseRequestLine Received [GET /^M
]
20-Mar-2020 13:13:37.725 FINE [http-nio-8080-exec-3] org.apache.coyote.AbstractProcessorLight.process Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@3eced8ba:org.apache.tomcat.util.net.NioChannel@6fc5100f:java.nio.channels.SocketChannel[connected local=/192.168.31.50:8080 remote=/192.168.31.6:51783]], Status in: [OPEN_READ], State out: [LONG]
20-Mar-2020 13:13:57.751 FINE [http-nio-8080-exec-4] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Processing socket [org.apache.tomcat.util.net.NioChannel@6fc5100f:java.nio.channels.SocketChannel[connected local=/192.168.31.50:8080 remote=/192.168.31.6:51783]] with status [ERROR]
20-Mar-2020 13:13:57.751 FINE [http-nio-8080-exec-4] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Found processor [org.apache.coyote.http11.Http11Processor@2d8c7e6c] for socket [org.apache.tomcat.util.net.NioChannel@6fc5100f:java.nio.channels.SocketChannel[connected local=/192.168.31.50:8080 remote=/192.168.31.6:51783]]
20-Mar-2020 13:13:57.751 FINE [http-nio-8080-exec-4] org.apache.coyote.AbstractProtocol.removeWaitingProcessor Removed processor [org.apache.coyote.http11.Http11Processor@2d8c7e6c] from waiting processors
20-Mar-2020 13:13:57.752 FINE [http-nio-8080-exec-4] org.apache.coyote.AbstractProcessorLight.process Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@3eced8ba:org.apache.tomcat.util.net.NioChannel@6fc5100f:java.nio.channels.SocketChannel[connected local=/192.168.31.50:8080 remote=/192.168.31.6:51783]], Status in: [ERROR], State out: [CLOSED]
20-Mar-2020 13:13:57.753 FINE [http-nio-8080-exec-4] org.apache.coyote.AbstractProtocol$ConnectionHandler.release Pushed Processor [org.apache.coyote.http11.Http11Processor@2d8c7e6c]
Comment 15 mgrigorov 2020-03-20 07:48:30 UTC
(In reply to Remy Maucherat from comment #4)
> Why is 0.9 support not removed by now ? It sounds *quite* ridiculous to
> still have it (clients which insist would still get something back).

+1 from me too.
Comment 16 Christopher Schultz 2020-03-20 18:47:20 UTC
(In reply to dingli from comment #13)
> F5 can send HTTP/1.0 or HTTP/1.1 request in http monitor. but ths stupid
> thing is  the default setting is HTTP/0.9 
> 
> we have change the F5 setting last night,now it works for tomcat 8.5.53

In that case, I can probably withdraw my -1. It would be nice to know if there are any major players out there who still *require* anything like HTTP/0.9.
Comment 17 Mark Thomas 2020-03-22 21:37:30 UTC
There is a regression in 8.5.51 (and probably 9.0.31, 7.0.100 and 10.0.0-M1) that means HTTP 0.9 requests of the form "GET /CRLF" are not processed correctly. The root cause is the improved CRLF parsing for headers. The CRLF on the request line is not fully parsed so the LF gets passed to the header parsing that then waits for more data. This is why the 20s timeout is observed with 8.5.51.

With 8.5.53 things are better but there is still an edge case. "GET / LF" which is either a malformed but allowable HTTP 0.9 request or an illegal HTTP /1.1 request (Tomcat opts for HTTP 0.9) may fail if the request line is split across multiple packets (which you may see when testing depending on the testing tools you use).

I'm working on a fix for this edge case.
Comment 18 Mark Thomas 2020-03-24 16:07:23 UTC
The final edge case has been fixed in:
- master for 10.0.0-M4 onwards
- 9.0.x for 9.0.34 onwards
- 8.5.x for 8.5.54 onwards
- 7.0.x for 7.0.104 onwards