I have found that after deploying and removing my application using tomcat manager a few times, the JVM throws an out of memory exception. I found the following posts on tomcat-dev/tomcat-user mail archives but did not see any bugzilla entries for it. POST 1 http://www.mail-archive.com/tomcat-dev@jakarta.apache.org/msg42351.html -------------------------------------------------------------------------- I believe I am seeing a memory leak that occurs when deploying or more precisely undeploying a web application through the Tomcat manager. I've done some analysis using a stripped down web application, JProbe, and code inspection. I would not presume to know the Tomcat source nor have done a complete and thorough analysis, but I would like to share my observations and more importantly, solicit feedback from the Tomcat user/development community. Environment: RedHat 8.0, JDK 1.4.1, Tomcat 4.1.21 Beta Synopsis of problem: We are deploying and undeploying our web applications through the Tomcat Manager. In the case of one of our web applications, redeploying 3-4 times resulted in an OutOfMemoryError in Tomcat's JVM. Initially, we thought this was due to several daemon Threads that were not Servlet lifecycle aware. But even after fixing these, we were still running out of memory. Suspecting that our classes were not being garbage collected (note the distinction between object garbage collection and class garbage collection) and might be pinned by classes that exist higher in the ClassLoader hierarchy (in common, shared, or possibly even server), I decided to try profiling using JProbe (http://java.quest.com/jprobe/jprobe.shtml) and a VERY simple web application. This web application is composed of a single Servlet that does nothing but allocate a 1,000,000 element byte array in its init() method and nulls it in its destroy() method. I deployed and undeployed several times running under JProbe's memory debugger and did observe a small memory leak of org.apache.* classes/instances. Analysis: These are the org.apache instances that do not appear to be garbage collected after a deploy/undeploy cycle: Class Count --------------------------------------------------------------------- org.apache.catalina.LifecycleListener[] 4 org.apache.catalina.Valve[] 1 org.apache.catalina.core.ApplicationContext 1 org.apache.catalina.core.ApplicationContextFacade 1 org.apache.catalina.core.NamingContextListener 1 org.apache.catalina.core.StandardContext 1 org.apache.catalina.core.StandardContextMapper 1 org.apache.catalina.core.StandardContextValve 1 org.apache.catalina.core.StandardPipeline 1 org.apache.catalina.deploy.ApplicationParameter[] 1 org.apache.catalina.deploy.NamingResources 1 org.apache.catalina.deploy.SecurityConstraint[] 1 org.apache.catalina.deploy.FilterMap[] 1 org.apache.catalina.loader.WebappClassLoader 1 org.apache.catalina.loader.WebappLoader 1 org.apache.catalina.session.StandardManager 1 org.apache.catalina.startup.ContextConfig 1 org.apache.catalina.util.LifecycleSupport 4 org.apache.commons.collections.LRUMap 1 org.apache.commons.collections.SequencedHashMap$Entry 6 org.apache.naming.NameParserImpl 2 org.apache.naming.NamingContext 3 org.apache.naming.NamingEntry 4 org.apache.naming.TransactionRef 1 org.apache.naming.resources.ImmutableNameNotFoundException 1 org.apache.naming.resources.ProxyDirContext 1 org.apache.naming.resources.ProxyDirContext$CacheEntry 5 org.apache.naming.resources.ResourceAttributes 3 org.apache.naming.resources.WARDirContext 2 org.apache.naming.resources.WARDirContext$WARResource 2 org.apache.naming.resources.WARDirContext$Entry 2 org.apache.naming.resources.WARDirContext$Entry[] 2 Initially, I focused on the org.apache.catalina.core.StandardContext class. It seemed like a nice entry point that scopes the Catalina classes supporting a web application deployment. It appears that an instance is pinned in several locations: 1. org.apache.naming.ContextBindings.bindContext() is called (in org.apache.catalina.core.NamingContextListener.lifecycleEvent() given a org.apache.catalina.LifeCycleEvent whose getType() is org.apache.catalina.Lifecycle.START_EVENT). This puts StandardContext into a static Hashtable within ContextBindings. This Hashtable entry is removed by a call to ContextBindings.unbindContext(). unbindContext() appears to never be called. 2. An org.apache.jasper.logging.DefaultLogger instance (which implements org.apache.jasper.logging.Logger) is created in org.apache.jasper.servlet.JspServlet.init(). DefaultLogger's setName() method is called resulting in the Logger being placed into a static Hashtable. Entries in this Hashtable are removed via the Logger.close() and Logger.removeLogger(...) methods, neither of which appear to be called. DefaultLogger refers to a StandardContext via the following chain of references: DefaultLogger -> org.apache.catalina.core.ApplicationContextFacade -> org.apache.catalina.core.ApplicationContext -> StandardContext 3. The DefaultLogger created in JspServlet is also referenced by a static field in org.apache.jasper.Constants. This field does not appear to be cleared. 4. org.apache.catalina.core.StandardHostDeployer has a static org.apache.catalina.Context field that is set to the suspect StandardContext after a call to its addChild() method (called reflectively by org.apache.commons.digester.Digester). This field is not unset unless the install(URL, URL) method is called (but not if the other install(String, URL) or either of the remove() methods is called). This context can be replaced if addChild() is called again, but would this happen only if another deployment occured? If so, StandardContext is still "pinned" if the web application is left undeployed and a subsequent deployment does not occur. 5. One of the Digester instances can also periodically hold onto a StandardContext (as its root), but this reference can and appears to be replaced (via Digester.push() when its stack is empty). I have not analyzed when this may be the case, so similar to (4), it seems possible that StandardContext will be pinned if the Digester instance is not "reset." At this point in time, I have not analyzed memory leaks beyond references to StandardContext. Many other "leaky" instances can be traced back to StandardContext (ApplicationContext, ApplicationContextFacade, StandardManager, etc...). The number of instances that appear to be leaking and the size of these instances is fairly small. I would guess less than 2K. However, I suspect that these instances are pinnning classes. By instrumenting our code, I have been able to determine that instances are indeed being garbage collected. On the other hand, I have been able to instrument WebappClassLoader and have not seen it finalize. In our production environment, we are deploying a Jetspeed portal which contains hundreds of classes which I believe can explain the limited number of times we are able to re-deploy before running out of memory. If you've managed to read through to here, any ideas or pointers would be greatly appreciated. Thanks, Ted Chen POST 2: http://www.mail-archive.com/tomcat-user@jakarta.apache.org/msg90811.html ---------------------------------------------------------------------- I believe I'm seeing a memory leak as a result of a Manager deploy/undeploy. I have a very simple test case: a Servlet that has a static field that refers to an object (Foo) that allocates a large chunk of memory. I've instrumented both the Servlet (init(), destroy(), and finalize()) and Foo (ctor and finalize()). The Servlet has been configured to load on startup. On a deploy, I see: Foo.ctor (during class initialization of the Servlet) Servlet.init(): On an undeploy, I see: Servlet.destroy() Servlet.finalize() I never see Foo.finalize(). If I continue to deploy/undeploy repeatedly, eventually the VM reports an OutOfMemoryError when I try to deploy. Running the VM with -verbose:gc and "encouraging" GC whenever possible, I see that after each undeploy, memory usage goes up roughly by what I've allocated in Foo. Any ideas?
bug #11128 could be related (or the same)
*** Bug 11128 has been marked as a duplicate of this bug. ***
Thanks for summarizing the issue, that I will try to address. That's typically is needed for TC 5.
Thanks for the summary on this memory leak Tony. I think this should be added to the list of changes/bug fixes for the next Tomcat 4.1.XX release.
http://nagoya.apache.org/eyebrowse/ReadMsg?listName=tomcat- dev@jakarta.apache.org&msgId=776903 seems to be adding to the topic: Using OptimizeIt, I tried to understand the problem and I found that my application ClassLoader was never totally released when I stop my application. There are always some strange references on it and especially the "classLoader of java.security.ProtectionDomain", which comes from the org.apache.catalina.core.StandardContext.start() method.
We too are getting out of memory erros in Tomcat 4.1.24 JDK 1.4.x Has this been fixed in a new 4.1 verion of tomcat?
The same problem still exists for 5.0.27. Can the version be changed to reflect this. I have some rather large singletons that do not get garbage collect on undeploy. This renders the undeploy/deploy features utterly useless. Bug 11128 has a suggested fix. Why have they not been commented on or implemented?
In summary it looks like there are three issues here: 1. StandardContext 2. Static references in servlets 3. ClassLoader The issues may or may be not be inter-related. I have just downloaded a profiler and started to have a look at these. I'll post results as I get them. On the subject of changing version, lets leave it as TC4 for now. If I make any changes I'll make them to TC5 as well.
The profiler shows that the static references in servlets are being cleaned up. Therefore 2 is not an issue. On re-reading this report it looks like there might be a fourth issue. Steve - can you provide me with a simple test case for your singleton issue? In summary: 1. Still to investigate 2. No issue 3. Still to investigate 4. Need more info
Is it possible that a static reference to the webapp's classloader could keep the servlet context components from being garbage collected? I've seen hints that this might be the case, particularly w.r.t. webapps that use cglib and/or hibernate.
Created attachment 12622 [details] Singleton Test Servlet
Created attachment 12623 [details] Singleton Test Class
Created attachment 12624 [details] Large object to load via reflection
I applogize. My previous posting was incorrect and does not reproduce the problem. It grows until the garbage collector hits its first limit. However, I have a new one that definately reproduces the problem. It is related to using ObjectOutputStreams. There are many posting related to using reset() with ObjectOutputStreams which I do. I also have a simple test code that repeatidly writes Object to a file with no problem. However, if put it in a servlet and continualy reload, it leaks. Could it be related to the ClassLoader?
Created attachment 12638 [details] Revised BigSingleton, which writes objects to disk.
Created attachment 12639 [details] Revised Singleton Servlet.
Quick update: 1. Worst offenders patched in TC4 (already pacthed in TC5). Need to work out if the remaining potential offenders are really an issue. 2. No issue. 3. Directly related to 1. 4. Still to investigate.
All should now be fixed. For the record: 1. Problem exactly as described in original report. All StandardContext issues fixed in TC4. Nearlly all had already been applied to TC5. The remaining one has been applied. 2. Could not reproduce. 3. Same problem as 1. 4. Unable to reproduce after fixing 1.
*** Bug 3888 has been marked as a duplicate of this bug. ***