Created attachment 31573 [details] Classes that hold a reference to WebappClassLoader I have a problem with unloading casses of a web application when I stop it. All threads are closed, but all 6.000 classes remain in memory. The GC cannot destroy WebappClassLoader because it is held by a static hashMap of the naming services which is hold by the VM. See attached image. The problem occurs only when the application uses the naming service. I identified the first method in my application that triggers the problem: public class MyBindAuthenticator extends BindAuthenticator { ... @Override public DirContextOperations authenticate(Authentication authentication) { ... List<String> userns=getUserDns(username); ... } ... } When I replace this line by a hardcoded list of strings, then the problem gets triggered by the next call of any naming service method. When I replace the whole authenticate() method by an empty one, then the problem disappears. But I need it for security reason. Unfortunately, the problematic hashmap (with name securityTokens) is not accessible to me, so I cannot remove the references. I seems that all related classes are part of Catalina and not reachable from outside. I would appreciate a workaround, if not bugfix is available.
1. Normally that reference should have been cleared by NamingContextListener The sequence of events is o.a.c.core.StandardContext.stopInternal() ... fireLifecycleEvent(Lifecycle.CONFIGURE_STOP_EVENT, null); -> o.a.c.core.NamingContextListener.lifecycleEvent(..) Is there any indication in your log files that your Context failed to start up, or failed to stop? 2. What is 'BindAuthenticator'? There is no such class in Tomcat. Source code for getUserDns() = ? 3. I think there is a bug in NamingContextListener.lifecycleEvent(..): If its processing of startup event fails, then its 'initialized' field remains to be 'false'. This causes its processing of stop event to exit immediately without proper cleanup. Is it what happened here?
Thanks for your quick answer. 1)While analyzing this problem, I found and solved two problem causes: I disabled the pooling of the LDAP interface because the pool starts a thread that remains running. Tomcat gave a related warning on application shutdown. I implemented a thread local ring buffer for log messages (using log4j). This buffer was not cleared on shutdown, so the web app classloader remained in memory. However, the problem with naming services is still present and I did not find out how to solve it without modifying Tomcat's source code. 2)BindAuthenticator is part of Spring Security. It is used to check the users name and password when he logs in. I had to override the authenticate method because the default method does not work for all users. 3) Good hint, I will check that.
Created attachment 31583 [details] ContextAccessController cleans up properly
Created attachment 31584 [details] New heap analysis showing classes that are keep WebAppClassloader in memory
Enabling FINEST logging for org.apache.catalina.core did not gave any helpful message. I still dont see any warning or error message in any logfile. But I have some few more debug messages now. I managed tro track the event handling and did not find any problem there. Lifecycle.CONFIGURE_START_EVENT and Lifecycle.CONFIGURE_STOP_EVENT are both completely processed by NamingContextLister without exception. Also the ContextAccessController removes the securityToken properly from the hashmap when I stop the program. So what I saw today does not match the heap dump that I created a few days ago. Sorry for that, I must have done something wrong without noticing it by myself. I attached a new screenshot from heap dump analysis, which looks much smaller now. And I see now totally other classes holding references to the classloader. But these classes are again all part of Tomcat or JVM, am I right? How can I analyze that further?
Created attachment 31585 [details] New heap analysis after lib upgrade I upgraded the cssparser.jar library from version 0.9.5 to 0.9.13. Now the picture has changed a lot again. The WebAppClassloader of my application has not path to GC Root anymore and is now hold by a soft or weak reference. But that did not finally solve my problem. All classes of my web application still remain in memory even when I force a garbage collection several times. I am still not able to restart the application (except by doubling the PermGenSpace size). My previous method to analyze the problem cause is not applicable anymore. There is no class anymore that holds a strong reference to the WebAppClassloader. What else can be the problem cause?
All the indications are that the root cause of this issue lies in application code or third party library code rather than Tomcat code. I am therefore resolving this as invalid. The users list is the best place to get help with this issue. One pointer that may help you move this a little further forward before you post to the users list is that I have seen cases where the JVM fails to correctly report all GC roots. Tracking down the memory leak in these cases is a complete pain. I hit a similar issue with a Spring app and had to build the relevant Spring libraries locally so I could do what was effectively a binary search to track down the line of code that triggered the issue. It turned out to be a memory leak somewhere in the JRE's XML parser but I never could track down exactly what as the JVM never reported the GC root.
Thanks for your helpful answer. And yes, I agree that the problem cause is obvisouly outside Tomcat.
(In reply to Konstantin Kolinko from comment #1) > 3. I think there is a bug in NamingContextListener.lifecycleEvent(..): > > If its processing of startup event fails, then its 'initialized' field > remains to be 'false'. This causes its processing of stop event to exit > immediately without proper cleanup. Is it what happened here? I fixed the 'initialized' field processing issue identified in the above comment. The fix will be in 8.0.6 and 7.0.54.
Created attachment 31671 [details] Screenshot of jvisualvm showing that webappclassloader has no gc root I tested again with version 7.0.54, the problem still occurs. The classloader of the stopped web application still remains in memory, but it has no gc root. The problem occurs only when the after the application did an LDAP communication. It does not happen, when I comment out the related lines of code or when I configure Spring Security to use dummy user accounts (in xml file) instead of LDAP authentication. Any ideas why the class loader remains in memory?