Bug 56663 - Can not get all data from InputStream in onDataAvailable
Can not get all data from InputStream in onDataAvailable
Status: RESOLVED FIXED
Product: Tomcat 8
Classification: Unclassified
Component: Catalina
8.0.x-trunk
PC Mac OS X 10.4
: P2 major (vote)
: ----
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2014-06-24 09:45 UTC by Long Zou
Modified: 2014-06-27 16:02 UTC (History)
0 users



Attachments
TestAsyncServlet is the full code by this case. (2.81 KB, text/plain)
2014-06-24 14:14 UTC, Long Zou
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Long Zou 2014-06-24 09:45:02 UTC
By from ReadListener, I implemented the method onDataAvailable same as below.
        
public void onDataAvailable() throws IOException {
            byte[] buf = new byte[1024];
            int len = 0;
            try{
                while ( _input.isReady() && (len = _input.read(buf)) != -1) {
                    _bufferStream.write(buf, 0, len);
                }
            }catch(Exception ex){
                logger.debug(ex.getMessage());
            }
}

When I send more than 1024 bytes from client, I can not get all data from method. It will exit the loop by _input.isReady() is false. 
But when I change the buf size to 2048, I can get all data by this code.
Comment 1 Long Zou 2014-06-24 10:01:41 UTC
public void onDataAvailable() throws IOException {
            byte[] buf = new byte[1024];
            int len = 0;
            try{
                while ( _input.isReady() && (len = _input.read(buf)) != -1) {
                    _bufferStream.write(buf, 0, len);
                }
            }catch(Exception ex){
                logger.debug(ex.getMessage());
            }
}

The client send 1406 bytes. But I can not get all.

If I changed the buf size to 2048, I can get all of 1406 bytes. But if I send more than 2048, I can not get all again.
Comment 2 Konstantin Kolinko 2014-06-24 10:36:13 UTC
1. Exact version of Tomcat 8.0.x = ?

2. What connector implementation is being used (NIO, NIO2, APR, BIO) ?
(See startup logs of your Tomcat, or ask on the Users mailing list)

3. The onDataAvailable() does not guarantee that _all_ data can be read. It just says that _some_ data can be read.

If "isReady()" returns false then you are expected to exit from this method, and wait until "onDataAvailable()" is being called the second time. See [1].



[1] http://docs.oracle.com/javaee/7/api/javax/servlet/ReadListener.html#onDataAvailable%28%29
Comment 3 Mark Thomas 2014-06-24 10:46:23 UTC
This works - Tomcat's WebSocket implementation is (currently) built on top of Servlet 3.1 async.

There are also a number of unit tests that cover this functionality.

There is insufficient information provided in this report to enable the problem to be reproduced.

Please seek help on the Tomcat users mailing list and only re-open this issue if a) that discussion concludes that there is a bug here and b) you have a reproducible test case (which should be as simple as possible) to demonstrate the issue.
Comment 4 Long Zou 2014-06-24 13:42:24 UTC
1. I test tomcat 8.0.5 and 8.0.8.
2. Connector is using NIO.
3. I understood the routines. 

Enclosing my code as below:
 
    final class SyncServiceReadListener implements ReadListener {
        private final AsyncContext _asyncCtx;
        private final ServletInputStream _input;
        private final HttpServletResponse _response;
        private final Locale _locale;
        private final ByteArrayOutputStream _bufferStream = new ByteArrayOutputStream();
        SyncServiceReadListener(ServletInputStream input, HttpServletResponse res,  AsyncContext ctx, Locale locale){
            _input = input;
            _asyncCtx = ctx;
            _response = res;
            _locale = locale;
            _serviceResponse = serviceResponse;
        }
        
        public void onDataAvailable() throws IOException {
            byte[] buf = new byte[1024];
            int len = 0;
            try{
                while ( _input.isReady() && (len = _input.read(buf)) != -1) {
                    _bufferStream.write(buf, 0, len);
                }
            }catch(Exception ex){
                logger.debug(ex.getMessage());
            }
        }

        public void onAllDataRead() throws IOException {
            try{
                _bufferStream.flush();
                _bufferStream.close();
            }catch(Exception ex){
                logger.debug(ex.getMessage());
            }

            ///By here, the _bufferStream.toByteArray() just returned 1024 bytes. 
            ...
        }

        public void onError(Throwable thrwbl) {
            if( thrwbl != null )
                logger.error(thrwbl.getMessage(), thrwbl);
            _asyncCtx.complete();
        }
    }
Comment 5 Konstantin Preißer 2014-06-24 14:04:28 UTC
Hi,

I can reproduce this problem with current trunk with the provided non-blocking ByteCounter example.

1) Apply the following patch to your working copy:

Index: ByteCounter.java
===================================================================
--- ByteCounter.java	(revision 1604977)
+++ ByteCounter.java	(working copy)
@@ -81,7 +81,7 @@
 
         private volatile boolean readFinished = false;
         private volatile long totalBytesRead = 0;
-        private byte[] buffer = new byte[8192];
+        private byte[] buffer = new byte[1024];
 
         private CounterListener(AsyncContext ac, ServletInputStream sis,
                 ServletOutputStream sos) {


2. Build Tomcat and start it with default settings (NIO connector).

3. Open a TCP connection and send the following request (it contains a Request Body with 1406 bytes):

POST /examples/servlets/nonblocking/bytecounter HTTP/1.1
Host: localhost
Connection: keep-alive
Content-Type: text/plain
Content-Length: 1406

0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcd

(remove the line breaks so that the body actually contains 1406 characters).



Actual Result:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/plain;charset=UTF-8
Content-Length: 28
Date: Tue, 24 Jun 2014 13:59:37 GMT

Total bytes written = [1024]



Expected Result:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/plain;charset=UTF-8
Content-Length: 28
Date: Tue, 24 Jun 2014 14:03:42 GMT

Total bytes written = [1406]
Comment 6 Long Zou 2014-06-24 14:14:47 UTC
Created attachment 31748 [details]
TestAsyncServlet is the full code by this case.

Client will send below code by text/json content-type.

{
  "key-0" : "AAAAAAAA",
  "key-4" : "AAAAAAAA",
  "deviceId" : "22C6A9AFE19D4577808FD1589ADF2AA8",
  "key-8" : "AAAAAAAA",
  "completed" : false,
  "key-11" : "AAAAAAAA",
  "key-20" : "AAAAAAAA",
  "key-13" : "AAAAAAAA",
  "key-22" : "AAAAAAAA",
  "key-15" : "AAAAAAAA",
  "key-24" : "AAAAAAAA",
  "key-17" : "AAAAAAAA",
  "key-1" : "AAAAAAAA",
  "key-19" : "AAAAAAAA",
  "key-33" : "AAAAAAAA",
  "key-31" : "AAAAAAAA",
  "key-35" : "AAAAAAAA",
  "key-5" : "AAAAAAAA",
  "sessionToken" : "O968K64KIL1KHCF3A62CSTAF00",
  "key-37" : "AAAAAAAA",
  "key-44" : "AAAAAAAA",
  "key-28" : "AAAAAAAA",
  "key-9" : "AAAAAAAA",
  "key-46" : "AAAAAAAA",
  "key-42" : "AAAAAAAA",
  "key-48" : "AAAAAAAA",
  "key-26" : "AAAAAAAA",
  "key-39" : "AAAAAAAA",
  "key-40" : "AAAAAAAA",
  "count" : 50,
  "key-2" : "AAAAAAAA",
  "key-6" : "AAAAAAAA",
  "key-10" : "AAAAAAAA",
  "key-12" : "AAAAAAAA",
  "key-21" : "AAAAAAAA",
  "key-14" : "AAAAAAAA",
  "key-23" : "AAAAAAAA",
  "key-16" : "AAAAAAAA",
  "key-30" : "AAAAAAAA",
  "key-18" : "AAAAAAAA",
  "key-32" : "AAAAAAAA",
  "key-25" : "AAAAAAAA",
  "key-3" : "AAAAAAAA",
  "key-34" : "AAAAAAAA",
  "key-27" : "AAAAAAAA",
  "key-36" : "AAAAAAAA",
  "key-29" : "AAAAAAAA",
  "key-7" : "AAAAAAAA",
  "key-45" : "AAAAAAAA",
  "key-43" : "AAAAAAAA",
  "key-41" : "AAAAAAAA",
  "key-47" : "AAAAAAAA",
  "key-38" : "AAAAAAAA",
  "key-49" : "AAAAAAAA"
}

There are 1381 bytes.
Comment 7 Remy Maucherat 2014-06-27 16:02:39 UTC
I could reproduce a number of bad behaviors using ByteCounter, not limited exclusively to onDataAvailable. r1606136, will be included in 8.0.10.