Bug 62830

Summary: Add static methods to load native libraries by the Common ClassLoader
Product: Tomcat 9 Reporter: Igal Sapir <isapir>
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: enhancement    
Priority: P2    
Version: unspecified   
Target Milestone: -----   
Hardware: PC   
OS: Linux   

Description Igal Sapir 2018-10-17 05:35:55 UTC
By design, a native library may not be loaded by more than one ClassLoader.  From [1]:

> In the JDK, each class loader manages its own set of native libraries. 
> The same JNI native library cannot be loaded into more than one class loader. 
> Doing so causes UnsatisfiedLinkError to be thrown. For example, 
> System.loadLibrary throws an UnsatisfiedLinkError when used to load a native
> library into two class loaders.

Due to that restriction, Native Libraries that are loaded by Webapp ClassLoaders may only be loaded in one Webapp.  Subsequent Webapps that attempt to load the same native library fail with an UnsatisfiedLinkError.

A working solution [2] proposes to create a small jar with a method that will load the native libraries, and place it in ${CATALINA_BASE}/lib so that it will be loaded by the Common ClassLoader.

A simpler solution would be to add the static methods `load(filename)` and `loadLibrary(libname)` that will simply call the respective System methods.  That will eliminate the need to create custom jar files and place them in the lib directory.

Patch coming shortly.

[1] https://docs.oracle.com/javase/1.5.0/docs/guide/jni/spec/invocation.html#library_version

[2] https://stackoverflow.com/questions/36936948/java-lang-unsatisfiedlinkerror-native-library-xxx-so-already-loaded-in-another
Comment 1 Igal Sapir 2018-10-17 05:41:56 UTC
Commit r1844065
Comment 2 Mark Thomas 2018-10-17 09:32:11 UTC
I think it would be worth updating the Javadoc for the new methods to make it clear that System.load[Library]() associates the loaded library not with the current thread context class loader but with the class loader of the class that calls System.load[Library](). Typically, this would be the Common class loader but that may vary depending on configuration and usage.
Comment 3 Remy Maucherat 2018-10-17 09:45:53 UTC
Yes, if it was using the thread context classloader, like is done nearly 100% of the time elsewhere, it would be useless. Weird stuff there.

Personally, I think native libraries can use extra care and if you're going to write a bit of Tomcat code, I would recommend writing a bit more and using a Catalina listener to load them (like for tomcat-native, basically).
Comment 4 Igal Sapir 2018-10-17 20:21:53 UTC
(In reply to Mark Thomas from comment #2)
> I think it would be worth updating the Javadoc for the new methods to make
> it clear that System.load[Library]() associates the loaded library not with
> the current thread context class loader but with the class loader of the
> class that calls System.load[Library](). Typically, this would be the Common
> class loader but that may vary depending on configuration and usage.

Please check r1844162 for accuracy.  I hope that I understood you correctly.  Text below for convenience:

System.loadLibrary() associates the
loaded library with the class loader of the class that called
the System method. A native library may not be loaded by more
than one class loader, so calling the System method from a class that
was loaded by a Webapp class loader will make it impossible for
other Webapps to load it.

Using this method will load the native library via a shared class
loader (typically the Common class loader, but may vary in some
configurations), so that it can be loaded by multiple Webapps.
Comment 5 Igal Sapir 2018-10-17 20:23:15 UTC
(In reply to Remy Maucherat from comment #3)
> Yes, if it was using the thread context classloader, like is done nearly
> 100% of the time elsewhere, it would be useless. Weird stuff there.
> 
> Personally, I think native libraries can use extra care and if you're going
> to write a bit of Tomcat code, I would recommend writing a bit more and
> using a Catalina listener to load them (like for tomcat-native, basically).

I like the idea of using a Listener.  I will look into it.
Comment 6 Igal Sapir 2018-10-22 05:05:12 UTC
(In reply to Remy Maucherat from comment #3)
> Personally, I think native libraries can use extra care and if you're going
> to write a bit of Tomcat code, I would recommend writing a bit more and
> using a Catalina listener to load them (like for tomcat-native, basically).

Since a Listener can not accept any arguments, the only way I can think of is to use a System property that will specify which native libraries should be loaded, and I know that we usually don't like to rely on those for configurations.

Are there other ideas?
Comment 7 Mark Thomas 2018-10-22 05:37:08 UTC
Listeners can use arguments. Look at the existing Tomcat listeners for examples.
Comment 8 Igal Sapir 2018-10-22 05:56:46 UTC
(In reply to Mark Thomas from comment #7)
> Listeners can use arguments. Look at the existing Tomcat listeners for
> examples.

Of course they can.  Don't know what I was thinking. _facepalm_
Comment 9 Igal Sapir 2018-10-22 08:03:54 UTC
Commit r1844531 adds JniLifecycleListener to trunk
Comment 10 Igal Sapir 2018-10-22 18:09:52 UTC
Added JniLifecycleListener to tc7.0.x and tc8.5.x as well
Comment 11 Igal Sapir 2018-10-22 18:14:21 UTC
JniLifecycleListener, Library.load(), and Library.loadLibrary() available in Tomcat 9.0.13, 8.5.35, and 7.0.92
Comment 12 Christopher Schultz 2018-10-23 00:06:05 UTC
Sorry... I must be missing something, here.

System.loadLibrary isn't ClassLoader-specific... once the library has been loaded, it can't be loaded again at all.

The code here is all fine, and using a Listener makes a lot of sense. But the documentation suggests that somehow loading a shared lib in a different ClassLoader changes something when it doesn't. Instead, you are asking Tomcat to load it *once* (at the server level) and then not again.

If you try to use this <Listener> at the <Context> level, it will fail when re-deploying the <Context>, just like if the application had used
Comment 13 Igal Sapir 2018-10-23 02:06:52 UTC
(In reply to Christopher Schultz from comment #12)
> Sorry... I must be missing something, here.
> 
> System.loadLibrary isn't ClassLoader-specific... once the library has been
> loaded, it can't be loaded again at all.
> 
> The code here is all fine, and using a Listener makes a lot of sense. But
> the documentation suggests that somehow loading a shared lib in a different
> ClassLoader changes something when it doesn't. Instead, you are asking
> Tomcat to load it *once* (at the server level) and then not again.
> 
> If you try to use this <Listener> at the <Context> level, it will fail when
> re-deploying the <Context>, just like if the application had used

It's counter-intuitive, I know. Here's what I tested right now, tell me know if covers the scenario you described above:

In conf/server.xml, I added the following snippet to the Server, Service, and in conf/Catalina/localhost/ROOT.xml, conf/Catalina/localhost/context_1.xml, conf/Catalina/localhost/context_2.xml to the Context element:  

> <Listener className="org.apache.catalina.core.JniLifecycleListener"  libraryName="opencv_java343" />

Below are the relevant log entries:

> 22-Oct-2018 18:45:04.733 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 412 ms
> 22-Oct-2018 18:45:04.739 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:45:04.759 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:45:04.759 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
> 22-Oct-2018 18:45:04.760 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.13-dev
> 22-Oct-2018 18:45:04.766 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/ROOT.xml]
> 22-Oct-2018 18:45:04.790 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> <snip/>
> 22-Oct-2018 18:45:06.277 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/ROOT.xml] has finished in [1,510] ms
> 22-Oct-2018 18:45:06.277 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_2.xml]
> 22-Oct-2018 18:45:06.279 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:45:06.379 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
> 22-Oct-2018 18:45:06.380 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_2.xml] has finished in [103] ms
> 22-Oct-2018 18:45:06.381 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_1.xml]
> 22-Oct-2018 18:45:06.382 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:45:06.473 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
> 22-Oct-2018 18:45:06.474 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_1.xml] has finished in [93] ms

Then I modified ROOT.xml, context_1.xml, and context_2.xml, one at a time:

> 22-Oct-2018 18:45:41.573 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/ROOT.xml]
> 22-Oct-2018 18:45:41.575 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:45:41.675 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
> 22-Oct-2018 18:45:41.677 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/ROOT.xml] has finished in [104] ms
> 22-Oct-2018 18:45:51.679 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/context_1]
> 22-Oct-2018 18:45:51.687 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_1.xml]
> 22-Oct-2018 18:45:51.692 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:45:51.787 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
> 22-Oct-2018 18:45:51.788 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_1.xml] has finished in [101] ms
> 22-Oct-2018 18:46:01.791 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/context_2]
> 22-Oct-2018 18:46:01.797 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_2.xml]
> 22-Oct-2018 18:46:01.803 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 22-Oct-2018 18:46:01.896 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
> 22-Oct-2018 18:46:01.897 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/home/user/Workspace/test/JniListener/conf/Catalina/localhost/context_2.xml] has finished in [99] ms
Comment 14 Konstantin Kolinko 2018-10-23 02:12:41 UTC
I think that this listener must be mentioned on "security-howto.xml".

http://tomcat.apache.org/tomcat-9.0-doc/security-howto.html#Listeners

It can be configured in any container (e.g. in context.xml) and it will load an arbitrary DLL, and I think that this will be done with only Tomcat code in the call stack. That means that it will run with Tomcat's "java.security.AllPermission" permissions.

This is not a problem, as server.xml/context.xml are edited by a trusted administrator. But whoever does a security audit of those files should be aware of this effect. Thus I think this listener should be mentioned.
Comment 15 Igal Sapir 2018-10-23 04:27:49 UTC
(In reply to Konstantin Kolinko from comment #14)
> I think that this listener must be mentioned on "security-howto.xml".
> 
> http://tomcat.apache.org/tomcat-9.0-doc/security-howto.html#Listeners
> 
> It can be configured in any container (e.g. in context.xml) and it will load
> an arbitrary DLL, and I think that this will be done with only Tomcat code
> in the call stack. That means that it will run with Tomcat's
> "java.security.AllPermission" permissions.

Added the following statement in r1844615:

The JNI Library Loading Listener may be used to load native code. It should 
only be used to load trusted libraries.
Comment 16 Christopher Schultz 2018-10-23 14:24:21 UTC
(In reply to Igal Sapir from comment #13)
> (In reply to Christopher Schultz from comment #12)
> > Sorry... I must be missing something, here.
> > 
> > System.loadLibrary isn't ClassLoader-specific... once the library has been
> > loaded, it can't be loaded again at all.
> > 
> > The code here is all fine, and using a Listener makes a lot of sense. But
> > the documentation suggests that somehow loading a shared lib in a different
> > ClassLoader changes something when it doesn't. Instead, you are asking
> > Tomcat to load it *once* (at the server level) and then not again.
> > 
> > If you try to use this <Listener> at the <Context> level, it will fail when
> > re-deploying the <Context>, just like if the application had used
> 
> It's counter-intuitive, I know. Here's what I tested right now, tell me know
> if covers the scenario you described above:
> 
> In conf/server.xml, I added the following snippet to the Server, Service,
> and in conf/Catalina/localhost/ROOT.xml,
> conf/Catalina/localhost/context_1.xml, conf/Catalina/localhost/context_2.xml
> to the Context element:  
> 
> > <Listener className="org.apache.catalina.core.JniLifecycleListener"  libraryName="opencv_java343" />
> 
> Below are the relevant log entries:
> 
> > 22-Oct-2018 18:45:04.739 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> > 22-Oct-2018 18:45:04.759 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343

Okay.


> Then I modified ROOT.xml, context_1.xml, and context_2.xml, one at a time:
> 
> > 22-Oct-2018 18:45:51.692 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> > [...]
> > 22-Oct-2018 18:46:01.803 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343

Interesting. I didn't realize that the JVM binds the native library to a specific ClassLoader and, does one of two things if the library has alreayd been loaded:

1. Ignores the call (if in the same ClassLoader; says so in javadoc for Runtime.loadLibrary)
2. Throws an UnsatisfiedLinkError complaining that the library is already loaded from a different ClassLoader.

The Java spec[1] has something odd to say about native lirbaries and ClassLoaders, though it's still a little unclear:

"
The programmer may use a single library to store all the native methods needed by any number of classes, as long as these classes are to be loaded with the same class loader. The VM internally maintains a list of loaded native libraries for each class loader.
"

I guess "same class loader" can also mean "or a child class loader" since it's evidently working for you.

But the point is that you need to consistently (re)load the native library into the same classloader if you don't want to throw any errors.

[1] https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.html#jni_interface_functions_and_pointers
Comment 17 Igal Sapir 2018-10-23 15:47:02 UTC
(In reply to Christopher Schultz from comment #16)
> (In reply to Igal Sapir from comment #13)
> > (In reply to Christopher Schultz from comment #12)
> > <snip/>
> > 
> > Below are the relevant log entries:
> > 
> > > 22-Oct-2018 18:45:04.739 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> > > 22-Oct-2018 18:45:04.759 INFO [main] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 
> Okay.
> 
> 
> > Then I modified ROOT.xml, context_1.xml, and context_2.xml, one at a time:
> > 
> > > 22-Oct-2018 18:45:51.692 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> > > [...]
> > > 22-Oct-2018 18:46:01.803 INFO [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.JniLifecycleListener.lifecycleEvent Loaded native library opencv_java343
> 
> Interesting. I didn't realize that the JVM binds the native library to a
> specific ClassLoader and, does one of two things if the library has alreayd
> been loaded:
> 
> 1. Ignores the call (if in the same ClassLoader; says so in javadoc for
> Runtime.loadLibrary)
> 2. Throws an UnsatisfiedLinkError complaining that the library is already
> loaded from a different ClassLoader.
> 
> The Java spec[1] has something odd to say about native lirbaries and
> ClassLoaders, though it's still a little unclear:

Oh, that is a much newer version of the link that I posted in the OP.  I'll give it a read.

> "The programmer may use a single library to store all the native methods
> needed by any number of classes, as long as these classes are to be loaded
> with the same class loader. The VM internally maintains a list of loaded
> native libraries for each class loader."
> 
> I guess "same class loader" can also mean "or a child class loader" since
> it's evidently working for you.

I believe that this is due to the delegation model [2] of class loaders (or else that would have really been useless as Remy pointed out above):

"
When requested to find a class or resource, a ClassLoader instance will delegate the search for the class or resource to its parent class loader before attempting to find the class or resource itself.
"

There is some variation in the Servlet spec for the order IIRC, but there is still delegation. 

> But the point is that you need to consistently (re)load the native library
> into the same classloader if you don't want to throw any errors.

It's not actually re-loading.  As you cited above, subsequent calls are ignored, but yes, they must be done in the same class loader or its lineage.  

That's the reason for adding the static methods to Library and adding the Listener.  Because these classes are in $CATALINA_HOME/lib, the native library to be loaded by a shared class loader which is in the lineage.

> [1]
> https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.
> html#jni_interface_functions_and_pointers

[2] https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html