|Summary:||ArrayIndexOutOfBoundsException after setting org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true|
|Product:||Tomcat 6||Reporter:||Juergen <juergen.schmidt>|
|Component:||Jasper||Assignee:||Tomcat Developers Mailing List <dev>|
|Attachments:||patch for BodyContentImpl class of trunk|
Description Juergen 2011-04-27 06:31:47 UTC
First of all, I know that there has been a bug regarding a similar problem before and that it was fixed. But currently this bugs occurs even with tomcat version 6.0.32. We have a problem with the tomcat runnign with the parameter org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true. After the tomcat has just started, everything is fine but after one or two days a lot of ArrayIndexOutOfBoundsException occur on random pages of the application. Tomcats started without this parameter don't seem to have that problem. Unfortunately we weren't able reproduce the bug locally, It appears only on our production system. Could there be a concurrency problem, that only shows if a lot it going on on the platform? I can provide two small stackstraces that can show the classes where the problem occurs: 1. Caused by: java.lang.ArrayIndexOutOfBoundsException at org.apache.jasper.runtime.BodyContentImpl.write(Unknown Source) at java.io.PrintWriter.write(PrintWriter.java:382) at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(Unknown Source) at org.apache.jasper.runtime.PageContextImpl.release(Unknown Source) at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(Unknown Source) at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(Unknown Source) at org.apache.jsp.members.lists.short_.incomingShortList_jsp._jspService(incomingShortList_jsp.java:796) at org.apache.jasper.runtime.HttpJspBase.service(Unknown Source) at javax.servlet.http.HttpServlet.service(Unknown Source) 2. Caused by: java.lang.ArrayIndexOutOfBoundsException at java.lang.String.getChars(String.java:854) at org.apache.jasper.runtime.BodyContentImpl.write(Unknown Source) at org.apache.jasper.runtime.BodyContentImpl.write(Unknown Source) at org.apache.jasper.runtime.BodyContentImpl.print(Unknown Source) at org.apache.taglibs.standard.tag.common.fmt.MessageSupport.doEndTag(MessageSupport.java:203) at org.apache.jsp.members.profiles.profile_jsp._jspx_meth_fmt_005fmessage_005f2(profile_jsp.java:4897) at org.apache.jsp.members.profiles.profile_jsp._jspService(profile_jsp.java:481) at org.apache.jasper.runtime.HttpJspBase.service(Unknown Source) at javax.servlet.http.HttpServlet.service(Unknown Source)
Comment 1 Ramiro 2011-05-04 09:39:28 UTC
Created attachment 26956 [details] patch for BodyContentImpl class of trunk We are reproducing this problem after getting an OutOfMemoryError on BodyContentImpl.reAllocBuff. java.lang.OutOfMemoryError: Java heap space at org.apache.jasper.runtime.BodyContentImpl.reAllocBuff(BodyContentImpl.java:595) at org.apache.jasper.runtime.BodyContentImpl.write(BodyContentImpl.java:109) at java.io.PrintWriter.write(PrintWriter.java:382) We are on version 6.0.29, but the cause is still in trunk. When the buffer needs to increase the size and an OOM error is thrown the bufferSize member variable is increased before the buffer is reallocated. But the BodyContentImpl is accessible for later requests (from the PageContext pool) with a bufferSize greater than the real buffer size, causing the ArrayIndexOutOfBoundsException to be thrown when trying to use the buffer space above the real size. To solve the problem the bufferSize member variable should be updated only after the buffer array is properly created. The patch fixes this issue.
Comment 2 Rainer Jung 2011-05-04 13:04:15 UTC
Once the JVM issues an OutOfMemoryError (especially for heap), there are no more guarantees that anything works correctly. After such an OOME you must restart the JVM. Relying on things to work just because it seems to is dangerous.
Comment 3 Juergen 2011-05-04 13:15:29 UTC
In our case there was no Out-Of-Memory that forced this bug to show. The application still ran fine but the Array-Index out of bounds exception were thrown anyway.
Comment 4 Mark Thomas 2011-05-04 19:53:39 UTC
Hmm. Looking at case 2 above, I don't see any way for that to happen unless bufferSize > cb.length. As far as I can see, these are always changed in sync. That leads me to think that there is an OOME that is being missed for some reason. While I agree 100% with Rainer that once an OOME occurs, all bets are off, I think that having this patch in 7.0.x will at least allow us to determine if the root cause is an OOME or something else. I have applied a variation of the patch suggested in comment 2 to Tomcat 7.0.x which will be included in 7.0.13 onwards. The patch won't get back-ported to 6.0.x until there is some evidence that it useful. Please test your application with 7.0.13 once released. and let us know how you get on. If an upgrade to 7.0.13 is not possible, we could provide a binary patch for 6.0.32 *purely for testing purposes*. Let us know if you want to go that route.
Comment 5 Ramiro 2011-05-05 07:42:04 UTC
No plans for us to upgrade to 7.0 in the near future. We are already testing 6.0.32 with the patch in #2, so no binary patch needed for us, thanks. We will apply Mark patch in trunk to our build to test with the same code. Although with an OOME we almost always need to restart a JVM in some applications I think this is not always the case with Tomcat (or other server applications) if the lack of memory is due to a server traffic peak and servlets in the applications use many tags with big buffered content. Of course it depends on which point the OOME is thrown. When this happens the affected request is aborted and all local resources are freed, including huge buffers allocated by tags (which are the cause of the problem for us), so it is not strange that the server can recover in some cases without needing to restart the JVM. If this happens (that JVM can free memory after the first OOME to handle following requests) Tomcat is still unusable because of BodyContentImpl instances with an invalid buffer are pooled in PageContext instances at JspFactoryImpl. Although the invalid BodyContentImpl instances should be recycled by PageContextImpl.release, in our experience the instances with the invalid buffer are used after the OOME. Last time we saw this problem, our server seemed to recover from the error after some time (several minutes) throwing the ArrayIndexOutOfBoundsException (AIOOBE) in certain requests very often. After some time working fine (more than 30 minutes) AIOOBE started again in some requests (without any other OOME), we supose because pooled PageContextImpl instances with invalid BodyContextImpl instances were used again needing a big buffer. So, and always with our experience, we think that in certain circumstances Tomcat can recover from an OOME in BodyContentImp.reAllocBuff, but these pooled invalid BodyContentImpl instances will force to restart JVM to clean them. Of course, we would like the patch in trunk applied also to 6.0.x ;)
Comment 6 Juergen 2011-05-06 06:44:44 UTC
We also used the code in the tomcat repository to patch our current tomcat 6.0.32 and will get back to you after it has been in production for some time. Thanks again for you help!
Comment 7 Juergen 2011-05-23 05:09:09 UTC
We have the modified code in production now for about one and half weeks, and the ArrayIndexOutOfBounds exceptions didn't occur anymore. It seems like the fix works accordingly. We would be glad if this fix could be backported to tomcat 6.0.x Please leave a comment here, to indicate whether this will be done or whether it won't. Thank you again for your great help!
Comment 8 Mark Thomas 2011-05-24 17:18:55 UTC
Thanks for the feedback. I have proposed the patch for 6.0.x.
Comment 9 Mark Thomas 2011-06-20 13:27:07 UTC
Fixed in 6.0.x and will be included in 6.0.33 onwards.
Comment 10 Vijay Ghadage 2015-08-10 09:12:37 UTC