| Summary: | Apache Mod_jk 1.2.30 is shutting down communication with tomcat and getting socket error read. | ||
|---|---|---|---|
| Product: | Tomcat Connectors | Reporter: | sammy <osaloum> |
| Component: | mod_jk | Assignee: | Tomcat Developers Mailing List <dev> |
| Status: | RESOLVED FIXED | ||
| Severity: | regression | ||
| Priority: | P2 | ||
| Version: | 1.2.30 | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | All | ||
| Attachments: | mod_jk.log in the debug mod | ||
I think you hit the following JBoss bug. If you have a support contract with them, please raise an issue there. You should mention, which AJP connector in JBoss you are using: Since I will refer to those types, as a reminder here's the list of AJP packet types used by the backend: 3 Send Body Chunk 4 Send Headers 5 End Response 6 Get Body Chunk 9 CPong Reply What happens is, that after mod_jk sends the request, JBoss responds, sending type "3" packets containing the response data, and after the final response packet it send a type "5" packet with the reuse bit set, which tells mod_jk that the response is finished. So far everything is fine. Now very shortly after that, the backend sends another type "3" packet with data of size "0" over the same connection. That's the packet that is not allowed and breaks the protocol. mod_jk doesn't consume that packet, because packet type "5" ends response processing. The packet stays in the connection buffer. I think that happens only for some JBos AJP connectors, but I don't know which of them. When the next request arrives, mod_jk sends the request headers and expects a type "6" packet from the backend. Instead it reads the old type "3" packet still in the buffer and throws an error because of a protocol corruption. To sum up: the problem is JBoss sending an empty type "3" packet after the response is finished. It doesn't happen for all requests, but it happens quite often. It is not unlikely that the error is triggered by strange webapp behaviour, but the webapp should not be able to corrupt the AJP 13 protocol. Other users exhibit the same probem when enabling cping/cpong. Then the erroneous type "3" packet is detected when waiting for the cping answer of type "9" and triggers again a failure. That failure also leads to closing the connection, but might be handled a bit faster and more transparent from the point of view of the browser and application. I'll mark this as invalid, because we made mod_jk stricter on protocol checks for security reasons. Feel free to reopen if you find any information that contradicts my explanation. Please let us know how JBoss responds. Rainer Adding to self: Problem seems to be spurious flush packets coming from the backend. Could be a webapp problem but still the app server should not send them after finishing the request. A similar problem, sendng flushes before the headers existed a while ago. On the other hand we already fixed the cping/cpong problem by silently consuming flushes (nothing else) even when not allowed by protocol. So could you please test the following patch: http://people.apache.org/~rjung/patches/mod_jk-trunk-ignore-flush-while-sending-request.patch It drops those flush packets. Thanks! Nevertheless also talking to JBoss makes sense. Thanks for the detailed explanation Would anyone knows of the patch will be integrated with future release of the Mod_jk.? Also is there a nightly build of the mod_jk?, I have been using the binary from the Apache foundation and not compiling the module. We only compile the module for Solaris and AIX, but for windows we go with the binaries, but I will compile if there is no binaries available. (In reply to comment #3) > Would anyone knows of the patch will be integrated with future release of the > Mod_jk.? Yes, that's the plan at the moment, subject to further testing. The patch is already part of the source tree used to cut the next release. > Also is there a nightly build of the mod_jk?, I have been using the binary from > the Apache foundation and not compiling the module. We only compile the module > for Solaris and AIX, but for windows we go with the binaries, but I will > compile if there is no binaries available. Unfortunately no nightly builds. It would be nice, if you could do the build on windows yourself. Regards, Rainer |
Created attachment 25565 [details] mod_jk.log in the debug mod Hi All My application has been working just fine with of Jboss 4.3.0 and tomcat engine. We decided to upgrade the connector from version 1.2.28 and 1.2.30 we are using apache 2.2.15 We are experiencing errors in the mod_jk log and as the result some of the pages in my app are not displaying right, the same pages display fine with the an1.2.28. I start to think that it could be the size of the chunked data between the connector and the tomcat because I see this error. Unexpected AJP13_SEND_BODY_CHUNK (myapp) sending request to tomcat failed (recoverable), because of server error (attempt=1) [Wed Jun 09 06:47:14.545 2010] [5900:2248] [debug] jk_connect.c (722): About to shutdown socket 784 Problem is reproducible with the new connector 1.2.30 but not the previous version. I have attached the log in the debug mod, have you seen these types of errors?