This problem is seen to happen in Tomcat 5.5.20 and 6.0.16 as well as SJAS81/82PE. In some cases when the issue occurs one will see: SEVERE: Servlet.service() for servlet default threw exception java.io.IOException: Bad file number at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:194) at java.io.BufferedInputStream.read1(BufferedInputStream.java:254) at java.io.BufferedInputStream.read(BufferedInputStream.java:313) at java.io.FilterInputStream.read(FilterInputStream.java:90) at org.apache.catalina.servlets.DefaultServlet.copyRange(DefaultServlet.java:1993) at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:1739) at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:824) It has currently only been reproduced on a Sparc T1 machine. Root cause: An optimization in org.apache.catalina.servlets.DefaultServlet copy (CacheEntry, InputStream, ServletOutputStream) was introduced in tomcat 5.5. This optimization is getting the binary content into a byte buffer from the cache and writes directly to ServletOutputStream using ServletOutputStream's write method. The ServletOutputStream implementation also uses its its own buffering for writing the byte buffer, while writing bytes using write(b, off, len) is not guaranteed to be in order. In a highly threaded environment ServletOutputStream's write method gets accessed by multiple threads, so ServletOutputStream's write method may sometimes get called while the previous thread's writing of the byte buffer is not yet finished or its own buffer is not yet flushed, which results in corrupted output. Fix: The fix is to remove this optimization. Said removal is expected to result in a negligible performance impact. We would still be using cache entries and the copying of cache contents to ServletOutputStream would be done using BufferedInputStream which remains efficient. Here is the diff for the fix as applied to Sun's Appserver: Index: DefaultServlet.java =================================================================== RCS file: /m/jws/appserv-webtier/src_imported/jakarta-tomcat-catalina/catalina/s rc/share/org/apache/catalina/servlets/DefaultServlet.java,v retrieving revision 1.6 diff -u -r1.6 DefaultServlet.java --- DefaultServlet.java 1 Nov 2004 19:40:13 -0000 1.6 +++ DefaultServlet.java 18 Jul 2008 12:02:47 -0000 @@ -1758,14 +1758,7 @@ IOException exception = null; InputStream resourceInputStream = null; - // Optimization: If the binary content has already been loaded, send - // it directly if (cacheEntry.resource != null) { - byte buffer[] = cacheEntry.resource.getContent(); - if (buffer != null) { - ostream.write(buffer, 0, buffer.length); - return; - } resourceInputStream = cacheEntry.resource.streamContent(); } else { resourceInputStream = is;
(In reply to comment #0) > This problem is seen to happen in Tomcat 5.5.20 and 6.0.16 The line numbers quoted in the stack trace do not match either of the Tomcat versions above, nor could I find a Tomcat 5.5.x or 6.0.x version that they did match. I tried using the patch to identify an offset but that didn't work either. In short, it is very difficult to work out what exactly this stack trace represents. > In a highly threaded environment > ServletOutputStream's write method gets accessed by multiple threads, so > ServletOutputStream's write method may sometimes get called while the previous > thread's writing of the byte buffer is not yet finished or its own buffer is > not yet flushed, which results in corrupted output. Every response has its own ServletOutputStream. Since a response is handled by a single thread, I don't see how a threading issue can exist here unless response objects are being re-used across requests by the application. That would be an application bug. > The fix is to remove this optimization. The stack trace shows reading from a File. The optimisation the patch removes copies the data directly from the cache without any file reading. I can't see how the proposed fix relates to the stack trace. Further, removal of the optimisation makes it more likely that the content will be read from the file. Given that the stack trace is related to reading data from a file I would expect the proposed patch to make any error more likely not less likely. A Google search suggests that a lack of OS resources could also be a cause of this error. Given that the environment is highly threaded, and taking this to also mean highly loaded, this looks more like a JVM/OS issue to me. Therefore I am closing this as invalid as I can't see anything in the code that Tomcat is doing incorrectly. That said, I do have a nagging doubt I am missing the obvious so if I am, feel free to re-open and point it out.
*** Bug 48760 has been marked as a duplicate of this bug. ***
If you see this error then the users list is the best place to figure out what is going wrong. I'm quite happy to apply a patch to fix a problem I can't reproduce providing that: - there is a logical explanation for a) why the problem is occurring and b) how the patch addresses it - the patch is confirmed to fix the issue in an environment where the issue can be produced - the patch isn't going to cause a regression for other use cases If the discussion on the users list can provide satisfactory answers to all of the above points feel free to re-open this issue.