Bug 62224 - SafeForkJoinWorkerThreadFactory breaks class loading on org.apache.naming.java.javaURLContextFactory
Summary: SafeForkJoinWorkerThreadFactory breaks class loading on org.apache.naming.jav...
Alias: None
Product: Tomcat 8
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 8.5.x-trunk
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2018-03-27 21:34 UTC by Behrooz Nobakht
Modified: 2021-02-04 08:04 UTC (History)
1 user (show)


Note You need to log in before you can comment on or make changes to this bug.
Description Behrooz Nobakht 2018-03-27 21:34:31 UTC
Tomcat 8.0.42 introduces a change to install a different ForkJoinWorkerThreadFactory into Java using a System property in an unconditional way.

Link to the change from 8.0.42: https://github.com/apache/tomcat80/compare/TOMCAT_8_0_41...TOMCAT_8_0_42#diff-738f0383c5c3a3e5bbd162aa59a114c6R470

We have tried to bisect Tomcat versions from 9.0.6 to 8.0.42 and all version upgrades have yielded to the following exception:

javax.naming.NoInitialContextException: Cannot instantiate class: org.apache.naming.java.javaURLContextFactory [Root exception is java.lang.ClassNotFoundException: org.apache.naming.java.javaURLContextFactory]
	at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:674)
	at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
	at javax.naming.InitialContext.init(InitialContext.java:244)
	at javax.naming.InitialContext.<init>(InitialContext.java:192)

When trying Tomcat 8.0.41, the exception does not appear anymore.

When trying Tomcat 8.0.42, when we remove the following Listener from server.xml, the exception does not occur any more:

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
Comment 1 Behrooz Nobakht 2018-03-27 22:21:35 UTC
Adding more context into this: The reason for this change seems to be this JDK bug: https://bugs.java.com/view_bug.do?bug_id=JDK-8172726
Comment 2 Remy Maucherat 2018-03-28 09:54:25 UTC
Ok so the revision is here: http://svn.apache.org/viewvc?rev=1778061&view=rev
It is supposedly no longer needed on Java 9+, so a subsequent revision could be made to avoid it in that case. Note that you also have a setting on JreMemoryLeakPreventionListener to disable it.

I don't get how this, by itself, then causes the naming context factory failure. Please explain how to reproduce it.
Comment 3 Behrooz Nobakht 2018-03-29 02:03:34 UTC
Extracting a minimal reproducible example is quite difficult for me. I will explain the current situation as is.

Tomcat is used with a custom server.xml:

<?xml version='1.0' encoding='utf-8'?>
<Server port="${shutdown.port}" shutdown="SHUTDOWN">
    <Listener className="org.apache.catalina.startup.VersionLoggerListener"/>
    <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"/>
    <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/>
    <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
    <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"/>
    <Service name="Catalina">
        <Connector port="${port}" protocol="HTTP/1.1"
                   maxHttpHeaderSize="8192" maxThreads="150" minSpareThreads="25" useBodyEncodingForURI="true"
                   enableLookups="false" redirectPort="8443" acceptCount="100" connectionTimeout="20000"
        <Engine name="Catalina" defaultHost="localhost">
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
                <Context containerSciFilter="org.apache.tomcat.websocket.server.WsSci" path="${app.context}"
                         docBase="${app.docbase}" workDir="${tomcat.workdir}" reloadable="false" useHttpOnly="true">
                    <Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction"
                              factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/>
                    <Manager pathname=""/>
                <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
                       prefix="localhost_access_log." suffix=".txt"
                       pattern="%h %l %u %t &quot;%r&quot; %s %b %D"/>

Please note the definition of `Resource` in above.

When accessing one of the HTTP endpoints on the app, a business objects wants for the first time to gain access to the JNDI resource `java:env/comp/UserTransaction`. 

Because there's no custom definition of NamingContext provider API, the default is picked by javax.naming.InitialContext. 

This is L313 of javax.naming.InitialContext in Java 1.8.0_172:

defaultInitCtx = NamingManager.getInitialContext(myProps);

`myProps` is filled by Catalina bootstrap to point to Apache Naming implementation.

Going further down the stack, you will reach com.sun.naming.internal.VersionHelper12 at L61:

return loadClass(className, getContextClassLoader());

The fix for ForkJoinWorkerThreadFactory means that there will be no context class loader, which means it will be "system" class loader and eventually leads to ClassNotFoundException.

Hope this clarifies further the issue.
Comment 4 Mark Thomas 2018-03-29 19:06:58 UTC
The JNDI lookup must be happening on the ForkJoin Thread.

The JNDI depends on the TCCL being set and it is no longer set on a ForkJoin thread (to avoid the memory leak).

I'll confirm the Java 9 fix and then disable the protection for Java 9 onwards.

Note that with the memory leak protection in place or when running on Java 9 onwards, the TCCL will have to be set on a ForkJoin thread.
Comment 5 Mark Thomas 2018-03-29 19:44:24 UTC
Protection disabled in Java 9 onwards in:
- trunk for 9.0.7 onwards
- 8.5.x for 8.5.30 onwards
- 8.0.x for 8.0.51 onwards
- 7.0.x for 7.0.86 onwards

Useful test cases for exploring memory leaks:
Comment 6 soheil rahsaz 2021-02-04 07:59:03 UTC
I'm experiencing the same issue on Tomcat 9.0.21 and JDK 14.0.1.
Comment 7 Mark Thomas 2021-02-04 08:04:39 UTC
This issue is fixed.

As per comment #4:

Note that with the memory leak protection in place or when running on Java 9 onwards, the TCCL will have to be set on a ForkJoin thread.

Please follow up on the users mailing list and include the simplest possible test case that demonstrates the problem you are seeing. If that discussion determines that you have found a bug then this issue can be re-opened or a new issue created as appropriate.