Bug 63836 - TestWebdavServletOptions test fails with OutOfMemoryError for NIO on 32-bit Java
Summary: TestWebdavServletOptions test fails with OutOfMemoryError for NIO on 32-bit Java
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 7.0.96
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-11 12:09 UTC by Konstantin Kolinko
Modified: 2019-10-26 20:31 UTC (History)
0 users



Attachments
TEST-org.apache.catalina.servlets.TestWebdavServletOptions.NIO.txt (466 bytes, text/plain)
2019-10-11 12:12 UTC, Konstantin Kolinko
Details
TEST-org.apache.catalina.servlets.TestWebdavServletOptions.APR.txt (675.79 KB, text/plain)
2019-10-11 12:13 UTC, Konstantin Kolinko
Details
ant_test.log (zipped) (44.88 KB, application/x-zip-compressed)
2019-10-11 12:19 UTC, Konstantin Kolinko
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Konstantin Kolinko 2019-10-11 12:09:41 UTC
I encountered this issue when testing a 7.0.97 release candidate.

The test "org.apache.catalina.servlets.TestWebdavServletOptions" fails for NIO connector when I run the tests with a 32-bit JVM.

Essentially, it is a parameterized test (@RunWith(Parameterized.class)) that has 312 iterations. Each iteration creates a Servlet, starts Tomcat and does two requests (OPTIONS, and one of other HTTP methods). Processing a request may involve reading a resource.

I am testing with different versions of Java on Windows 10. I start Apache Ant from a command line, without an IDE.

Thus far I know the following:

1. The test fails reliably when I run the test alone. (Running a single test is done by adding the following line to build.properties:

test.entry=org.apache.catalina.servlets.TestWebdavServletOptions

2. The test fails for NIO connector only and a 32-bit JDK.

- It completes successfully for BIO and APR.

- It fails for 32-bit Java 6u45, 7u80, 8u202. It completes successfully for 64-bit 8u222, 13u0.

3. The same test exists in Tomcat 8.5.47 and it completes successfully there. I did run it with 32-bit Java 7u80.

4. When the test fails:

- TEST-org.apache.catalina.servlets.TestWebdavServletOptions.NIO.txt
This file is essentially empty, with a message "Forked Java VM exited abnormally." and no other details.

- The error can be seen in stdout/stderr output of Apache Ant.

a. The test runs successfully up to iteration 235
(iterations are numbered starting with 0)

> [junit] INFO: Starting test case [testOptions[234]]
> ... (followed by log messages of starting and stopping Apache Tomcat)

b. Starting with iteration 236 and  I see only two messages:

> [junit] INFO: Starting test case [testOptions[235]]
> [junit] INFO: Initializing ProtocolHandler ["http-nio-127.0.0.1-auto-236"]

There are no other startup or shutdown messages following.

c. After iteration 309 (out of total 312) the run is aborted with an OOME

> [junit] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space


5. I tried to investigate this as a regression and bisect.
My result is that this is not a regression. I was not able to find a non-failing version.

a. The test fails with current 7.0.x branch head (as of 2019-10-10), with 7.0.97 and 7.0.96.

b. The test fails on the commit when it was added to the source tree (aed9453c710bafce9d69c5d4ea02363d371b8a32 on 2019-06-29). This was between versions 7.0.94 and 7.0.95.

c. If I backport the test implementation (the test/** changes from aed9453c71, I observe the same OutOfMemoryError for versions 7.0.94, 7.0.93, 7.0.92 and 7.0.80.

6. A trivial attempt of a fix by patching TomcatBaseTest.tearDown() by explicitly releasing a reference to a Tomcat instance - does not help

--- a/test/org/apache/catalina/startup/TomcatBaseTest.java
+++ b/test/org/apache/catalina/startup/TomcatBaseTest.java
@@ -176,6 +176,7 @@ public abstract class TomcatBaseTest extends LoggingBaseTest {
             }
         } finally {
             super.tearDown();
+            tomcat = null;
         }
     }

I see the same OutOfMemoryError.

Judging from point 5.c. I think this is not a regression and not a showstopper. If it were, I think there would have been other evidences in the two years since 7.0.80.

It might be an issue with the test itself, but it is strange that it manifests itself only when running with a NIO connector. The overhead from JUnit, from test set up, from resources being served, from the size of a web application should be the same.

I think that a good next step from here would be to take a memory dump and to analyze it with a profiler.
Comment 1 Konstantin Kolinko 2019-10-11 12:12:14 UTC
Created attachment 36818 [details]
TEST-org.apache.catalina.servlets.TestWebdavServletOptions.NIO.txt

Output from a failing run (Java 8u202 32-bit, NIO)
Comment 2 Konstantin Kolinko 2019-10-11 12:13:12 UTC
Created attachment 36819 [details]
TEST-org.apache.catalina.servlets.TestWebdavServletOptions.APR.txt

Output from a successful run (Java 8u202 32-bit, APR)
Comment 3 Konstantin Kolinko 2019-10-11 12:19:10 UTC
Created attachment 36820 [details]
ant_test.log (zipped)

Apache Ant stdout/stderr from the run (Java 8u202 32-bit, all connectors: BIO, NIO and APR). A 1.8 MB file compressed with zip.

The OutOfMemoryError message can be seen in it.
Comment 4 Mark Thomas 2019-10-26 20:15:03 UTC
Oh, the irony.

The problem is that the 1MB reserved by the OOME parachute is not released.

Hopefully, I'll have a fix for this shortly.
Comment 5 Mark Thomas 2019-10-26 20:31:28 UTC
Fixed in 7.0.x for 7.0.98 onwards