Bug 49413 - Apache Mod_jk 1.2.30 is shutting down communication with tomcat and getting socket error read.
Apache Mod_jk 1.2.30 is shutting down communication with tomcat and getting s...
Status: RESOLVED FIXED
Product: Tomcat Connectors
Classification: Unclassified
Component: mod_jk
1.2.30
PC All
: P2 regression (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2010-06-09 07:12 UTC by sammy
Modified: 2011-10-25 19:20 UTC (History)
0 users



Attachments
mod_jk.log in the debug mod (31.38 KB, text/plain)
2010-06-09 07:12 UTC, sammy
Details

Note You need to log in before you can comment on or make changes to this bug.
Description sammy 2010-06-09 07:12:05 UTC
Created attachment 25565 [details]
mod_jk.log in the debug mod

Hi All
My application has been working just fine with of Jboss 4.3.0 and tomcat engine.
We decided to upgrade the connector from version 1.2.28 and 1.2.30 we are using apache 2.2.15
We are experiencing errors in the mod_jk log and as the result some of the pages in my app are not displaying right, the same pages display fine with the an1.2.28.
I start to think that it could be the size of the chunked data between the connector and the tomcat because I see this error.
Unexpected AJP13_SEND_BODY_CHUNK
(myapp) sending request to tomcat failed (recoverable), because of server error (attempt=1)
[Wed Jun 09 06:47:14.545 2010] [5900:2248] [debug] jk_connect.c (722): About to shutdown socket 784

Problem is reproducible with the new connector 1.2.30 but not the previous version.



I have attached the log in the debug mod, have you seen these types of errors?
Comment 1 Rainer Jung 2010-06-09 07:53:41 UTC
I think you hit the following JBoss bug. If you have a support contract with them, please raise an issue there. You should mention, which AJP connector in JBoss you are using:

Since I will refer to those types, as a reminder here's the list of AJP
packet types used by the backend:

3  	Send Body Chunk
4 	Send Headers
5 	End Response
6 	Get Body Chunk
9 	CPong Reply

What happens is, that after mod_jk sends the request, JBoss responds, sending
type "3" packets containing the response data, and after the final response packet it send a type "5" packet with the reuse bit set, which tells mod_jk that the response is finished.

So far everything is fine.

Now very shortly after that, the backend sends another type "3" packet with data of size "0" over the same connection. That's the packet that is not allowed and breaks the protocol. mod_jk doesn't consume that packet, because packet type "5" ends response processing. The packet stays in the connection buffer. I think that happens only for some JBos AJP connectors, but I don't know which of them.

When the next request arrives, mod_jk sends the request headers and expects a type "6" packet from the backend. Instead it reads the old type "3" packet still in the buffer and throws an error because of a protocol corruption.

To sum up: the problem is JBoss sending an empty type "3" packet after
the response is finished. It doesn't happen for all requests, but it
happens quite often. It is not unlikely that the error is triggered by
strange webapp behaviour, but the webapp should not be able to corrupt
the AJP 13 protocol.

Other users exhibit the same probem when enabling cping/cpong. Then the erroneous type "3" packet is detected when waiting for the cping answer of type "9" and triggers again a failure. That failure also leads to closing the connection, but might be handled a bit faster and more transparent from the point of view of the browser and application.

I'll mark this as invalid, because we made mod_jk stricter on protocol checks for security reasons. Feel free to reopen if you find any information that contradicts my explanation. Please let us know how JBoss responds.

Rainer
Comment 2 Rainer Jung 2010-06-09 08:41:23 UTC
Adding to self:

Problem seems to be spurious flush packets coming from the backend. Could be a webapp problem but still the app server should not send them after finishing the request. A similar problem, sendng flushes before the headers existed a while ago.

On the other hand we already fixed the cping/cpong problem by silently consuming flushes (nothing else) even when not allowed by protocol.

So could you please test the following patch:

http://people.apache.org/~rjung/patches/mod_jk-trunk-ignore-flush-while-sending-request.patch

It drops those flush packets. 

Thanks!

Nevertheless also talking to JBoss makes sense.
Comment 3 sammy 2010-06-10 10:01:27 UTC
Thanks for the detailed explanation
Would anyone knows of the patch will be integrated with future release of the Mod_jk.?
Also is there a nightly build of the mod_jk?, I have been using the binary from the Apache foundation and not compiling the module. We only compile the module for Solaris and AIX, but for windows we go with the binaries, but I will compile if there is no binaries available.
Comment 4 Rainer Jung 2010-06-10 10:08:26 UTC
(In reply to comment #3)
> Would anyone knows of the patch will be integrated with future release of the
> Mod_jk.?

Yes, that's the plan at the moment, subject to further testing. The patch is already part of the source tree used to cut the next release.

> Also is there a nightly build of the mod_jk?, I have been using the binary from
> the Apache foundation and not compiling the module. We only compile the module
> for Solaris and AIX, but for windows we go with the binaries, but I will
> compile if there is no binaries available.

Unfortunately no nightly builds. It would be nice, if you could do the build on windows yourself.

Regards,

Rainer
Comment 5 Rainer Jung 2011-10-25 19:20:44 UTC
The patch was applied as r952980 and is contained in version 1.2.31 and beyond.
If you think the bug is still in the latest version 1.2.32 please reopen.