Bug 53752

Summary: Tomcat restarts with error:java.lang.ArrayIndexOutOfBoundsException: -60
Product: Tomcat 6 Reporter: thyag <thyag_020>
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Severity: normal    
Priority: P2    
Version: 6.0.24   
Target Milestone: default   
Hardware: Other   
OS: Linux   
Attachments: tomcat.log

Description thyag 2012-08-21 09:52:46 UTC
Created attachment 29257 [details]

During the load test Tomcat restarts, with error :java.lang.ArrayIndexOutOfBoundsException: -60
INFO: Initialization processed in 501 ms
Aug 17, 2012 1:58:08 PM org.apache.catalina.core.StandardService start
INFO: Starting service TspWebGui
Aug 17, 2012 1:58:08 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.24
Aug 17, 2012 1:58:08 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive tsp.war
Aug 17, 2012 1:58:08 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-9099
Aug 17, 2012 1:58:08 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 354 ms
Aug 17, 2012 2:15:42 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing request
java.lang.ArrayIndexOutOfBoundsException: -60
	at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:284)
	at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:132)
	at org.apache.coyote.http11.filters.ChunkedInputFilter.end(ChunkedInputFilter.java:189)
	at org.apache.coyote.http11.InternalInputBuffer.endRequest(InternalInputBuffer.java:336)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:886)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:662)

Attached the tomcat.log file.
Comment 1 Christopher Schultz 2012-08-27 22:12:53 UTC
Can this really be CRITICAL if it hasn't been identified in the 10 releases since 6.0.24 what you are using?

Your log file is useless: it contains nothing other than your CLASSPATH and the stack trace you already provided. At least it confirms you are using a very old version of Tomcat 6: 6.0.24 (2.5 years old).

Please re-test with 6.0.latest. If you can still get it to crash, attach a minimal webapp (or better yet, use the examples webapp that ships with Tomcat) and a JMeter configuration file (or similar) so we can reproduce the crash. Then you can mark it as CRITICAL.
Comment 2 Konstantin Kolinko 2012-08-27 22:54:41 UTC
> During the load test Tomcat restarts

What do you mean by "restarts"? Tomcat cannot restart itself.

The restart process is not reflected in your log file, as Christopher correctly noted.

> java.lang.ArrayIndexOutOfBoundsException: -60

The exception is caused by invalid request data. Tomcat should have rejected it silently. Does your testing framework notice that Tomcat has been restarted?

AIOOBE also causes the request to be rejected, so besides the nasty log message the behaviour does not differ much from the intended one.

I'll fix the AIOOBE. Thank you for the report.

I am closing this report as DUPLICATE, as there has been an earlier report for this issue.

*** This bug has been marked as a duplicate of bug 42181 ***
Comment 3 thyag 2012-08-29 04:41:26 UTC
>>Does your testing framework notice that Tomcat has been restarted?
yes our framework sees that Tomcat is not stable and i will restart tomcat.

Below are the logs from our framework, where it check for status of tomcat at regular intervals and if the service is not consistent it will restart.
27-08-2012 18:24:52  119-70    information  "TSP:atcacscfn16.RtpSnm01: RtpTomcatCheck -p 99 exits with ExitCode = 102"
27-08-2012 18:24:52  119-49    warning      "TSP:atcacscfn16.RtpSnm01: Subsystem TOMCAT (Id=7) detector failed - ExitCode = 102"
27-08-2012 18:24:52  119-8     minor        "TSP:atcacscfn16.RtpSnm01: Restart required for subsystem TOMCAT - Exitcode = 102"

27-08-2012 18:25:39  119-6     clear        "TSP:atcacscfn16.RtpSnm01: Subsystem TOMCAT successfully stopped"
27-08-2012 18:25:45  119-3     clear        "TSP:atcacscfn16.RtpSnm01: Subsystem TOMCAT successfully started"

more information about the testcase.
Codenomicon is a well known Commercial robustness testing tool available for various protocols. While running Codenomicom Diameter Server suite (which sends both valid and invalid malicious Diameter messages to test the robustness of the Diameter interface), the tomcat subsystem which hosts Platforms TSP GUI goes for restart. During this time TSP GUI access fails. During a long run of the codenomicon suite it is observed that the tomcat process restarts.

The testcase or the test case group where the issue is noticed is as below:

← 260820 →
Repeat of 8 octets: '\'' 0x00
Overflow CWE-633 CWE-150 CWE-138 CWE-122 CWE-120 CWE-119 CWE-118 CWE-20 CWE-19 CWE-18 CWE-17
Attack Modifier = +50 CVSS/BS = 10.0 (components)

 HTTP Request [with anomaly]
Comment 4 Mark Thomas 2012-09-17 07:25:05 UTC
 You are confusing multiple issues.

The first issue is the  invalid request triggers an exception. This was logged and is now silently swallowed. That is a minor cosmetic issue.

The second issue is that *your* framework detects that Tomcat is "not stable" and *you* then restart Tomcat. Based on the limited information provided here, there is no Tomcat issue.

You have provided no information that defines "not stable" nor have you provided any information on how to trigger this condition. The processing of the invalid request (with or without the message in the logs) certainly does not meet any definition of "not stable" that the Tomcat developers accept.
Comment 5 thyag 2012-09-18 11:38:57 UTC

There are no two issues reported.
its all the same issue: tomcat is inconsistent and it is restarted.
RtpTomcatCheck -p 99 is a script which runs perodically for tomcat when the exitcode is not 0, it will restart.

below are the log when the check is done using RtpTomcatCheck.
I felt that this have no connection with this issue, so i did not provide earlier.

2012/07/27-20:47:19 RtpTomcatCheck started
2012/07/27-20:47:19 readFromPipe entered
2012/07/27-20:47:19 readFromPipe: totalBytesRead 5
2012/07/27-20:47:19 pkgparam: bytes read from pipe: 5
2012/07/27-20:47:19 pkgparam, return: 9000
2012/07/27-20:47:19 Webserver port: 9099
2012/07/27-20:47:19 Check timeout : 10
2012/07/27-20:47:19 Retry count   : 1
2012/07/27-20:47:19 Retry period  : 10
2012/07/27-20:47:19 ---
2012/07/27-20:47:19 Webserver check #1
2012/07/27-20:47:19 Connect call failed: 111 [Connection refused]
2012/07/27-20:47:19 Could not establish connection to webserver
2012/07/27-20:47:19 Webserver check failed, trying SSL Webserver check
2012/07/27-20:47:19 before SSL_library_init()
2012/07/27-20:47:19 before BIO_do_connect() to localhost:9099
2012/07/27-20:47:19 Error connecting to SSL server
2012/07/27-20:47:19 Reason:
[pid]:error:[error code]:[library name]:[function name]:[reason string]:[file name]:[line]:[optional text message]
4151228152:error:0200206F:system library:connect:Connection refused:bss_conn.c:269:host=localhost:9099
4151228152:error:20073067:BIO routines:CONN_STATE:connect error:bss_conn.c:273:
2012/07/27-20:47:19 before BIO_free_all()
2012/07/27-20:47:19 Giving up: All webserver checks failed
2012/07/27-20:47:19 exit status = 102