Bug 57638 - IllegalArgumentException in AjpNioProcessor when packetSize > 8192
IllegalArgumentException in AjpNioProcessor when packetSize > 8192
Status: RESOLVED FIXED
Product: Tomcat 8
Classification: Unclassified
Component: Catalina
8.0.20
PC Mac OS X 10.1
: P2 normal (vote)
: ----
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2015-02-26 14:03 UTC by Christopher Schultz
Modified: 2015-03-07 19:13 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Schultz 2015-02-26 14:03:32 UTC
For reference: http://markmail.org/thread/vu7kgdpqxdw7tlkz

This stack trace is from 8.0.17 but I have been able to reproduce it in 8.0.20 and 7.0.56 - 7.0.59:

java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at
org.apache.coyote.ajp.AjpNioProcessor.readSocket(AjpNioProcessor.java:179)
        at
org.apache.coyote.ajp.AjpNioProcessor.read(AjpNioProcessor.java:159)
        at
org.apache.coyote.ajp.AbstractAjpProcessor.readMessage(AbstractAjpProcessor.java:1067)
        at
org.apache.coyote.ajp.AbstractAjpProcessor.receive(AbstractAjpProcessor.java:1005)
        at
org.apache.coyote.ajp.AbstractAjpProcessor.refillReadBuffer(AbstractAjpProcessor.java:1131)

Connector configuration:

    <Connector port="@connector-port@"
       redirectPort="443"
           protocol="org.apache.coyote.ajp.AjpNioProtocol"
        URIEncoding="UTF-8"
         packetSize="65536"
           executor="tomcatThreadPool" />

Here's what I know so far:

1. AjpNioProtocol fails under some conditions (my post-login page)
   with packetSize larger than the default (I have tried 65536, 32768,
   16184, and 8200). The failing page is ~30k in size, enough to
   exceed the default packet size but not e.g. maximum 64k

2. Using AjpProtocol (BIO) connector solves the problem

3. Using the default AJP packet size (8192) solves the problem

4. Specifying socket.appReadBufSize and socket.appWriteBufSize to be
the same as the packet size solves the problem

I'm still working on a small test case to help motivate debugging.

Wild-guessing that the root cause is mismatched buffer sizes, or lack of checking when using a smaller protocol buffer than whatever buffer is pumping into the protocol's buffer.
Comment 1 Konstantin Kolinko 2015-03-06 17:49:36 UTC
What version the stacktrace is from?

> at java.nio.Buffer.limit(Buffer.java:275)

The line matches Java 8u31 sources.
The IAE is triggered by the following check:

[[[
public final Buffer limit(int newLimit) {
   if ((newLimit > capacity) || (newLimit < 0))
            throw new IllegalArgumentException();
]]]

> org.apache.coyote.ajp.AjpNioProcessor.readSocket(AjpNioProcessor.java:179)

The above does not match Tomcat 7 sources. By method name I guess that is line 356.

[[[
    private int readSocket(byte[] buf, int pos, int n, boolean block)
            throws IOException {
        int nRead = 0;
        ByteBuffer readBuffer =
                socketWrapper.getSocket().getBufHandler().getReadBuffer();
        readBuffer.clear();
        readBuffer.limit(n);
]]]

Apparently it tries to read n bytes from the Socket read buffer at once. The buffer is smaller that that thus the IAE.
Comment 2 Christopher Schultz 2015-03-06 18:08:49 UTC
(In reply to Konstantin Kolinko from comment #1)
> What version the stacktrace is from?
> 
> > at java.nio.Buffer.limit(Buffer.java:275)
> 
> The line matches Java 8u31 sources.
> The IAE is triggered by the following check:
> 
> [[[
> public final Buffer limit(int newLimit) {
>    if ((newLimit > capacity) || (newLimit < 0))
>             throw new IllegalArgumentException();
> ]]]
> 
> > org.apache.coyote.ajp.AjpNioProcessor.readSocket(AjpNioProcessor.java:179)
> 
> The above does not match Tomcat 7 sources. By method name I guess that is
> line 356.

The reference in comment #1 says Tomcat 8.0.17. I'm sorry I didn't set the correct version. I was able to reproduce this in 7.0.56, 7.0.57, 7.0.50, 8.0.17 and 8.0.20 with varying stack traces. The one posted here appears to be from Tomcat 8.0.17.

> [[[
>     private int readSocket(byte[] buf, int pos, int n, boolean block)
>             throws IOException {
>         int nRead = 0;
>         ByteBuffer readBuffer =
>                 socketWrapper.getSocket().getBufHandler().getReadBuffer();
>         readBuffer.clear();
>         readBuffer.limit(n);
> ]]]
> 
> Apparently it tries to read n bytes from the Socket read buffer at once. The
> buffer is smaller that that thus the IAE.

That seems like a reasonable diagnosis, but I'm not familiar enough with how these components interact to understand the root cause and most appropriate fix.

I suspect this is a rare problem since most people probably stick to the default packetSize (8192) with the AJP connector.
Comment 3 Mark Thomas 2015-03-07 15:00:04 UTC
I have a test case for this. Sending any AJP request body chunk message with size > socket read buffer is sufficient to trigger this with NIO or NIO2.

BIO, APR and 9.0.x are all unaffected.

The fix looks to be simple. I need to clean everything up and should be able to commit a fix shortly.
Comment 4 Mark Thomas 2015-03-07 15:33:54 UTC
Fixed in 8.0.x and 8.0.21 onwards and in 7.0.x for 7.0.60 onwards.

Neither trunk nor 6.0x. were affected.
Comment 5 Christopher Schultz 2015-03-07 18:31:49 UTC
(In reply to Mark Thomas from comment #4)
> Fixed in 8.0.x and 8.0.21 onwards and in 7.0.x for 7.0.60 onwards.


Excellent, I'll reproduce, then update and re-test in my environment.

> Neither trunk nor 6.0x. were affected.

That was my expectation based upon Rémy'y comments about the refactoring in trunk having unified lots of things; this bug would have been much more obvious with other connectors.
Comment 6 Christopher Schultz 2015-03-07 18:56:40 UTC
I just re-built with Tomcat 7.0.x trunk and I'm now getting a different exception:

SEVERE: Servlet.service() for servlet velocity threw exception
java.nio.BufferOverflowException
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
        at org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:305)
        at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1234)
        at org.apache.coyote.Response.doWrite(Response.java:499)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:402)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:485)
        at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:464)
        at org.apache.tomcat.util.buf.CharChunk.append(CharChunk.java:302)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:527)
        at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
        at org.apache.velocity.io.VelocityWriter.flushBuffer(VelocityWriter.java:129)
        at org.apache.velocity.io.VelocityWriter.write(VelocityWriter.java:306)
        at org.apache.velocity.io.VelocityWriter.write(VelocityWriter.java:322)
        at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:491)
        at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at org.apache.velocity.Template.merge(Template.java:356)
        at org.apache.velocity.Template.merge(Template.java:260)
        at org.apache.velocity.tools.view.VelocityView.performMerge(VelocityView.java:942)
        at org.apache.velocity.tools.view.VelocityView.merge(VelocityView.java:902)
        at org.apache.velocity.tools.view.VelocityViewServlet.mergeTemplate(VelocityViewServlet.java:318)
        at org.apache.velocity.tools.view.VelocityLayoutServlet.mergeTemplate(VelocityLayoutServlet.java:247)
        at org.apache.velocity.tools.view.VelocityViewServlet.doRequest(VelocityViewServlet.java:220)
        at org.apache.velocity.tools.view.VelocityViewServlet.doGet(VelocityViewServlet.java:182)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
  [...]

Might this be the other side of the buffer coin?
Comment 7 Christopher Schultz 2015-03-07 19:04:27 UTC
I added some debug logging to the line immediately before the writeBuffer.put(), and I got these logs before the exception:

Mar 07, 2015 2:02:09 PM org.apache.coyote.ajp.AjpNioProcessor output
WARNING: Writing to output buffer of capacity=8192, position=0, limit=8192, remaining=8192 from source buffer of size=65536, starting at offset=0, len=79

Mar 07, 2015 2:02:09 PM org.apache.coyote.ajp.AjpNioProcessor output
WARNING: Writing to output buffer of capacity=8192, position=0, limit=8192, remaining=8192 from source buffer of size=6, starting at offset=0, len=6

Mar 07, 2015 2:02:09 PM org.apache.coyote.ajp.AjpNioProcessor output
WARNING: Writing to output buffer of capacity=8192, position=0, limit=8192, remaining=8192 from source buffer of size=65536, starting at offset=0, len=113

Mar 07, 2015 2:02:09 PM org.apache.coyote.ajp.AjpNioProcessor output
WARNING: Writing to output buffer of capacity=8192, position=0, limit=8192, remaining=8192 from source buffer of size=6, starting at offset=0, len=6

Mar 07, 2015 2:02:10 PM org.apache.coyote.ajp.AjpNioProcessor output
WARNING: Writing to output buffer of capacity=8192, position=0, limit=8192, remaining=8192 from source buffer of size=65536, starting at offset=0, len=42

Mar 07, 2015 2:02:10 PM org.apache.coyote.ajp.AjpNioProcessor output
WARNING: Writing to output buffer of capacity=8192, position=0, limit=8192, remaining=8192 from source buffer of size=65536, starting at offset=0, len=8200
Comment 8 Mark Thomas 2015-03-07 19:07:01 UTC
That is a different bug. It is in output rather than input. Please open a new issue.
Comment 9 Christopher Schultz 2015-03-07 19:13:41 UTC
Filed the output-oriented bug under bug #57674.