As reported on the users list [2] (note that I used Tomcat 7.0.27, but Bugzilla seems to have only 7.0.26): Hi all, some time ago, I wrote about a BufferOverflowException which I got sporadically on my Tomcat 7.0.21 [1], when using the AJP-APR-Connector. However, at that time I couldn't reproduce the exception. When I looked at this again today, I was able to create a client that sends an AJP message and aborts the connection, resulting in the BufferOverflowException in Tomcat. I was using Tomcat 7.0.27 with Tomcat Native 1.1.23, on Oracle JDK 1.7.0_03 and Windows 7 32-bit. To reproduce, 1. On the server side, create a web application containing a servlet which will generate some random data: @WebServlet("/index.html") public class MyTestServlet extends HttpServlet { private static final long serialVersionUID = 1L; protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { try { response.setContentType("text/plain"); response.setCharacterEncoding("utf-8"); byte[] bytesToWrite = {48, 49, 50, 51, 52, 53, 54, 55, 56, 57}; // Try to write 1,000,000 bytes to the client. try (OutputStream out = response.getOutputStream()) { for (int i = 0; i < 100000; i++) { out.write(bytesToWrite); } } } catch (IOException ex) { // Ignore } } } Deploy the application as ROOT, so that the servlet will be reachable at "/". Configure Tomcat to have an AJP-APR connector listening on port 8009. 2. On the client side, create a program like this: public class AjpTester { public static void main(String[] args) throws IOException { byte[] ajpPacketBytes = { // JK_AJP13_FORWARD_REQUEST for "/" 0x12, 0x34, 0x00, 0x54, 0x02, 0x02, 0x00, 0x08, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x00, 0x00, 0x01, 0x2f, 0x00, 0x00, 0x07, 0x31, 0x2e, 0x32, 0x2e, 0x33, 0x2e, 0x34, 0x00, 0x00, 0x07, 0x31, 0x2e, 0x32, 0x2e, 0x33, 0x2e, 0x34, 0x00, 0x00, 0x09, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x68, 0x6f, 0x73, 0x74, 0x00, 0x00, 0x50, 0x00, 0x00, 0x02, (byte)0xa0, 0x06, 0x00, 0x0a, 0x4b, 0x65, 0x65, 0x70, 0x2d, 0x41, 0x6c, 0x69, 0x76, 0x65, 0x00, (byte)0xa0, 0x0b, 0x00, 0x09, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x68, 0x6f, 0x73, 0x74, 0x00, (byte)0xff }; Socket s = new Socket("localhost", 8009); // connect to AJP port OutputStream sout = s.getOutputStream(); InputStream sin = s.getInputStream(); sout.write(ajpPacketBytes); // request Url "/" // Read between 10,000 and 11,023 bytes, then exit the JVM byte[] buf = new byte[1024]; int read; int bytesRead = 0; while ((read = sin.read(buf)) > 0) { bytesRead += read; if (bytesRead >= 10000) { System.exit(1); } } } } The client will send an JK_AJP13_FORWARD_REQUEST message which requests URL "/" and sends "Connection: keep-alive" and "Host: localhost" headers. 3. Tomcat will display this exception: Apr 20, 2012 10:57:08 PM org.apache.coyote.ajp.AjpAprProcessor process Schwerwiegend: Error processing request java.nio.BufferOverflowException at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) at org.apache.coyote.ajp.AjpAprProcessor.output(AjpAprProcessor.java:285) at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1119) at org.apache.coyote.Response.doWrite(Response.java:504) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283) at org.apache.catalina.connector.Response.finishResponse(Response.java:511) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:197) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1812) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) I guess the exception itself is harmless, but it may fill up log files, if clients disconnect frequently while receiving data. Note that before that exception occurs, a regular "ClientAbortException: java.io.IOException: Failed to send AJP message" is thrown by org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388). The exception occurs neither with AJP-BIO nor with AJP-NIO. [1] http://markmail.org/message/zogi3tfbz2kyw3jg [2] http://markmail.org/message/ntejdw36pfhlqodr
I can't reproduce this with the provided test case but I believe I have tracked down the root cause and fixed it. The fix has been applied to trunk and 7.0.x and will be included in 7.0.28 onwards.
Thanks, Mark. I tested with Tomcat 7 Trunk and can confirm that r1344253 fixes the issue.
Created attachment 28890 [details] 2012-06-05_tc6_53119_AjpAprProcessor.patch Backport of r1344253 to be proposed for 6.0. The patched method name is different, but code is the same.
Regarding review of r1344253 and porting it to other processor implementations http://tomcat.markmail.org/thread/d4mzx52gj3omchrr The question of whether repeated call to *processor#output(..) could fail because of insufficient cleanup from previous failed call. - AjpProcessor: There is nothing to clean up. It just uses passed byte[] array as is. - AjpNioProcessor: Improved in r1346365 in trunk, proposed for 7.0.
I use Tomcat with Nginx wia ajp connector. I found the followings: - In my tomcat side: Feb 27, 2012 4:53:02 PM com.ys.cache.StaticFileCache getFile INFO: /usr/local/tomcat/webapps/ROOT/i/a.swf : 1 Feb 27, 2012 4:53:02 PM org.apache.coyote.ajp.AjpAprProcessor process SEVERE: Error processing request java.nio.BufferOverflowException at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) at org.apache.coyote.ajp.AjpAprProcessor.output(AjpAprProcessor.java:285) at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1082) at org.apache.coyote.Response.doWrite(Response.java:533) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283) at org.apache.catalina.connector.Response.finishResponse(Response.java:507) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:197) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) nginx log (~1 minute before tomcat log): 78.131.23.132 - - [27/Feb/2012:16:52:01 +0100] "GET /i/a.swf HTTP/1.1" 200 369 "http://dictzone.com/osszes-szotar/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11" I think a timer ended of Nginx side, and close the ajp connection. In this case the problem on the Nginx side or on the Tomcat side?
Patch applied to 6.0.x and will be in 6.0.36 onwards.
Hi, as reported on the users list [1], shortly after I switched to the AJP-NIO connector on Tomcat 7.0.32 (Java 1.7.0_09), I got another java.nio.BufferOverflowException: Okt 20, 2012 2:58:51 PM org.apache.coyote.ajp.AjpNioProcessor process SEVERE: Error processing request java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183) at org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281) at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1122) at org.apache.coyote.Response.doWrite(Response.java:504) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283) at org.apache.catalina.connector.Response.finishResponse(Response.java:514) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434) at org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:184) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) The stacktrace seems to be very similar to the one that I got when using the AJP-APR connector. However, unfortunately I'm currently unable to reproduce the issue that causes this BufferOverflowException. Thanks! [1] http://markmail.org/message/uiife6gx7zwmyfku
Is it absolutely sure that this bug is fixed? I tried to use ajp-nio with the latest 7.0.52 version and get the same problem right on the first page: Mar 12, 2014 12:18:09 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8091"] Mar 12, 2014 12:18:09 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-nio-8016"] Mar 12, 2014 12:18:09 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 32142 ms Mar 12, 2014 12:18:46 AM org.apache.coyote.ajp.AjpNioProcessor process SEVERE: Error processing request java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183) at org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:296) at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1195) at org.apache.coyote.Response.doWrite(Response.java:520) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:491) at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:464) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:293) at org.apache.catalina.connector.Response.finishResponse(Response.java:510) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:435) at org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:177) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
Update! We're using shibboleth for SAML authentication and they recommend that you increase the package size. We've been running with the default AJP-connector for over a year with packetSize="65536" and ProxyIOBufferSize 65536 in Apache, but when we turned that off the AjpNioProcessor works again. What is the best way to debug this problem? Should I create a new ticket?
It is not immediately obvious that this is the same issue so yes, please create a new bugzilla issue and provide the necessary information to enable this to be reproduced.
*** Bug 56296 has been marked as a duplicate of this bug. ***