Bug 53119 - java.nio.BufferOverflowException in AjpAprProcessor.output() when AJP client disconnects
java.nio.BufferOverflowException in AjpAprProcessor.output() when AJP client ...
Status: RESOLVED FIXED
Product: Tomcat 7
Classification: Unclassified
Component: Connectors
7.0.27
PC All
: P2 minor (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
: 56296 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2012-04-21 11:51 UTC by Konstantin Preißer
Modified: 2014-03-23 22:58 UTC (History)
1 user (show)



Attachments
2012-06-05_tc6_53119_AjpAprProcessor.patch (774 bytes, patch)
2012-06-05 11:03 UTC, Konstantin Kolinko
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Konstantin Preißer 2012-04-21 11:51:17 UTC
As reported on the users list [2] (note that I used Tomcat 7.0.27, but Bugzilla seems to have only 7.0.26):

Hi all,

some time ago, I wrote about a BufferOverflowException which I got sporadically on my Tomcat 7.0.21 [1], when using the AJP-APR-Connector.
However, at that time I couldn't reproduce the exception.

When I looked at this again today, I was able to create a client that sends an AJP message and aborts the connection, resulting in the BufferOverflowException in Tomcat. I was using Tomcat 7.0.27 with Tomcat Native 1.1.23, on Oracle JDK 1.7.0_03 and Windows 7 32-bit.


To reproduce,

1. On the server side, create a web application containing a servlet which will generate some random data:

@WebServlet("/index.html")
public class MyTestServlet extends HttpServlet {
    private static final long serialVersionUID = 1L;
       
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        try {
            response.setContentType("text/plain");
            response.setCharacterEncoding("utf-8");
            
            byte[] bytesToWrite = {48, 49, 50, 51, 52, 53, 54, 55, 56, 57};
            
            // Try to write 1,000,000 bytes to the client.
            try (OutputStream out = response.getOutputStream()) {
                for (int i = 0; i < 100000; i++) {
                    out.write(bytesToWrite);
                }
            }
            
        } catch (IOException ex) {
            // Ignore
        }
    }
}

Deploy the application as ROOT, so that the servlet will be reachable at "/".
Configure Tomcat to have an AJP-APR connector listening on port 8009.


2. On the client side, create a program like this:

public class AjpTester {
    public static void main(String[] args) throws IOException {

        byte[] ajpPacketBytes = { // JK_AJP13_FORWARD_REQUEST for "/"
            0x12, 0x34, 0x00, 0x54, 0x02, 0x02, 0x00, 0x08,
            0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31,
            0x00, 0x00, 0x01, 0x2f, 0x00, 0x00, 0x07, 0x31,
            0x2e, 0x32, 0x2e, 0x33, 0x2e, 0x34, 0x00, 0x00,
            0x07, 0x31, 0x2e, 0x32, 0x2e, 0x33, 0x2e, 0x34,
            0x00, 0x00, 0x09, 0x6c, 0x6f, 0x63, 0x61, 0x6c,
            0x68, 0x6f, 0x73, 0x74, 0x00, 0x00, 0x50, 0x00,
            0x00, 0x02, (byte)0xa0, 0x06, 0x00, 0x0a, 0x4b, 0x65,
            0x65, 0x70, 0x2d, 0x41, 0x6c, 0x69, 0x76, 0x65,
            0x00, (byte)0xa0, 0x0b, 0x00, 0x09, 0x6c, 0x6f, 0x63,
            0x61, 0x6c, 0x68, 0x6f, 0x73, 0x74, 0x00, (byte)0xff
        };

        Socket s = new Socket("localhost", 8009); // connect to AJP port
        OutputStream sout = s.getOutputStream();
        InputStream sin = s.getInputStream();

        sout.write(ajpPacketBytes); // request Url "/"

        // Read between 10,000 and 11,023 bytes, then exit the JVM
        byte[] buf = new byte[1024];
        int read;
        int bytesRead = 0;
        while ((read = sin.read(buf)) > 0) {
            bytesRead += read;
            if (bytesRead >= 10000) {
                System.exit(1);
            }
        }
    }
}

The client will send an JK_AJP13_FORWARD_REQUEST message which requests URL "/" and sends "Connection: keep-alive" and "Host: localhost" headers.


3. Tomcat will display this exception:

Apr 20, 2012 10:57:08 PM org.apache.coyote.ajp.AjpAprProcessor process
Schwerwiegend: Error processing request
java.nio.BufferOverflowException
	at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357)
	at org.apache.coyote.ajp.AjpAprProcessor.output(AjpAprProcessor.java:285)
	at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1119)
	at org.apache.coyote.Response.doWrite(Response.java:504)
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
	at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
	at org.apache.catalina.connector.Response.finishResponse(Response.java:511)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
	at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:197)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1812)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


I guess the exception itself is harmless, but it may fill up log files, if clients disconnect frequently while receiving data.
Note that before that exception occurs, a regular "ClientAbortException:  java.io.IOException: Failed to send AJP message" is thrown by org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388).
The exception occurs neither with AJP-BIO nor with AJP-NIO.


[1] http://markmail.org/message/zogi3tfbz2kyw3jg
[2] http://markmail.org/message/ntejdw36pfhlqodr
Comment 1 Mark Thomas 2012-05-30 13:35:58 UTC
I can't reproduce this with the provided test case but I believe I have tracked down the root cause and fixed it. The fix has been applied to trunk and 7.0.x and will be included in 7.0.28 onwards.
Comment 2 Konstantin Preißer 2012-05-30 14:23:27 UTC
Thanks, Mark.

I tested with Tomcat 7 Trunk and can confirm that r1344253 fixes the issue.
Comment 3 Konstantin Kolinko 2012-06-05 11:03:49 UTC
Created attachment 28890 [details]
2012-06-05_tc6_53119_AjpAprProcessor.patch

Backport of r1344253 to be proposed for 6.0. The patched method name is different, but code is the same.
Comment 4 Konstantin Kolinko 2012-06-05 12:27:57 UTC
Regarding review of r1344253 and porting it to other processor implementations
http://tomcat.markmail.org/thread/d4mzx52gj3omchrr

The question of whether repeated call to *processor#output(..) could fail because of insufficient cleanup from previous failed call.

- AjpProcessor:
There is nothing to clean up. It just uses passed byte[] array as is.

- AjpNioProcessor:
Improved in r1346365 in trunk, proposed for 7.0.
Comment 5 Ferenc Lutischan 2012-06-08 10:47:13 UTC
I use Tomcat with Nginx wia ajp connector.
I found the followings:
- In my tomcat side:
Feb 27, 2012 4:53:02 PM com.ys.cache.StaticFileCache getFile
INFO: /usr/local/tomcat/webapps/ROOT/i/a.swf : 1
Feb 27, 2012 4:53:02 PM org.apache.coyote.ajp.AjpAprProcessor process
SEVERE: Error processing request
java.nio.BufferOverflowException
    at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357)
    at org.apache.coyote.ajp.AjpAprProcessor.output(AjpAprProcessor.java:285)
    at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1082)
    at org.apache.coyote.Response.doWrite(Response.java:533)
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
    at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
    at org.apache.catalina.connector.Response.finishResponse(Response.java:507)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
    at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:197)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
    at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1805)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722) 

nginx log (~1 minute before tomcat log):
78.131.23.132 - - [27/Feb/2012:16:52:01 +0100] "GET /i/a.swf HTTP/1.1" 200 369 "http://dictzone.com/osszes-szotar/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"

I think a timer ended of Nginx side, and close the ajp connection.
In this case the problem on the Nginx side or on the Tomcat side?
Comment 6 Mark Thomas 2012-06-21 11:17:37 UTC
Patch applied to 6.0.x and will be in 6.0.36 onwards.
Comment 7 Konstantin Preißer 2012-10-27 12:54:21 UTC
Hi,

as reported on the users list [1], shortly after I switched to the AJP-NIO connector on Tomcat 7.0.32 (Java 1.7.0_09), I got another java.nio.BufferOverflowException:

Okt 20, 2012 2:58:51 PM org.apache.coyote.ajp.AjpNioProcessor process
SEVERE: Error processing request
java.nio.BufferOverflowException
	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
	at org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
	at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1122)
	at org.apache.coyote.Response.doWrite(Response.java:504)
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
	at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
	at org.apache.catalina.connector.Response.finishResponse(Response.java:514)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
	at org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:184)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


The stacktrace seems to be very similar to the one that I got when using the AJP-APR connector. However, unfortunately I'm currently unable to reproduce the issue that causes this BufferOverflowException.


Thanks!

[1] http://markmail.org/message/uiife6gx7zwmyfku
Comment 8 Claes Mogren 2014-03-11 23:32:25 UTC
Is it absolutely sure that this bug is fixed? I tried to use ajp-nio with the latest 7.0.52 version and get the same problem right on the first page:

Mar 12, 2014 12:18:09 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8091"]
Mar 12, 2014 12:18:09 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-nio-8016"]
Mar 12, 2014 12:18:09 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 32142 ms
Mar 12, 2014 12:18:46 AM org.apache.coyote.ajp.AjpNioProcessor process
SEVERE: Error processing request
java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
    at org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:296)
    at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1195)
    at org.apache.coyote.Response.doWrite(Response.java:520)
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
    at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:491)
    at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:464)
    at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:293)
    at org.apache.catalina.connector.Response.finishResponse(Response.java:510)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:435)
    at org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:177)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Comment 9 Claes Mogren 2014-03-12 09:12:39 UTC
Update! 

We're using shibboleth for SAML authentication and they recommend that you increase the package size. We've been running with the default AJP-connector for over a year with packetSize="65536" and ProxyIOBufferSize 65536 in Apache, but when we turned that off the AjpNioProcessor works again. What is the best way to debug this problem? Should I create a new ticket?
Comment 10 Mark Thomas 2014-03-12 13:04:29 UTC
It is not immediately obvious that this is the same issue so yes, please create a new bugzilla issue and provide the necessary information to enable this to be reproduced.
Comment 11 Mark Thomas 2014-03-23 22:58:55 UTC
*** Bug 56296 has been marked as a duplicate of this bug. ***