Bug 20758

Summary: Memory Leak in Classloader/Manager deploy/undeploy
Product: Tomcat 4 Reporter: Tony Chao <sys>
Component: Webapps:ManagerAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal CC: ernst.matthias, jon, kutzi, thomas.cataldo
Priority: P3    
Version: 4.1.24   
Target Milestone: ---   
Hardware: Other   
OS: other   
Attachments: Singleton Test Servlet
Singleton Test Class
Large object to load via reflection
Revised BigSingleton, which writes objects to disk.
Revised Singleton Servlet.

Description Tony Chao 2003-06-13 18:34:09 UTC
I have found that after deploying and removing my application using tomcat 
manager a few times, the JVM throws an out of memory exception.

I found the following posts on tomcat-dev/tomcat-user mail archives but did 
not see any bugzilla entries for it.

POST 1
http://www.mail-archive.com/tomcat-dev@jakarta.apache.org/msg42351.html
--------------------------------------------------------------------------
I believe I am seeing a memory leak that occurs when deploying or
more precisely undeploying a web application through the Tomcat manager.
I've done some analysis using a stripped down web application, JProbe,
and code inspection.  I would not presume to know the Tomcat source nor
have done a complete and thorough analysis, but I would like to share
my observations and more importantly, solicit feedback from the Tomcat
user/development community.

Environment:

  RedHat 8.0, JDK 1.4.1, Tomcat 4.1.21 Beta

Synopsis of problem:

  We are deploying and undeploying our web applications through
  the Tomcat Manager.  In the case of one of our web applications,
  redeploying 3-4 times resulted in an OutOfMemoryError in
  Tomcat's JVM.  Initially, we thought this was due to several daemon
  Threads that were not Servlet lifecycle aware.  But even after
  fixing these, we were still running out of memory.

  Suspecting that our classes were not being garbage collected
  (note the distinction between object garbage collection and
  class garbage collection) and might be pinned by classes that
  exist higher in the ClassLoader hierarchy (in common, shared,
  or possibly even server), I decided to try profiling using JProbe
  (http://java.quest.com/jprobe/jprobe.shtml) and a VERY simple
  web application.  This web application is composed of a single
  Servlet that does nothing but allocate a 1,000,000 element
  byte array in its init() method and nulls it in its destroy() method.
  I deployed and undeployed several times running under JProbe's
  memory debugger and did observe a small memory leak of org.apache.*
  classes/instances.

Analysis:

  These are the org.apache instances that do not appear to be garbage
  collected after a deploy/undeploy cycle:


  Class                                                         Count
  ---------------------------------------------------------------------
  org.apache.catalina.LifecycleListener[]                       4
  org.apache.catalina.Valve[]                                   1
  org.apache.catalina.core.ApplicationContext                   1
  org.apache.catalina.core.ApplicationContextFacade             1
  org.apache.catalina.core.NamingContextListener                1
  org.apache.catalina.core.StandardContext                      1
  org.apache.catalina.core.StandardContextMapper                1
  org.apache.catalina.core.StandardContextValve                 1
  org.apache.catalina.core.StandardPipeline                     1
  org.apache.catalina.deploy.ApplicationParameter[]             1
  org.apache.catalina.deploy.NamingResources                    1
  org.apache.catalina.deploy.SecurityConstraint[]               1
  org.apache.catalina.deploy.FilterMap[]                        1
  org.apache.catalina.loader.WebappClassLoader                  1
  org.apache.catalina.loader.WebappLoader                       1
  org.apache.catalina.session.StandardManager                   1
  org.apache.catalina.startup.ContextConfig                     1
  org.apache.catalina.util.LifecycleSupport                     4
  org.apache.commons.collections.LRUMap                         1
  org.apache.commons.collections.SequencedHashMap$Entry         6
  org.apache.naming.NameParserImpl                              2
  org.apache.naming.NamingContext                               3
  org.apache.naming.NamingEntry                                 4
  org.apache.naming.TransactionRef                              1
  org.apache.naming.resources.ImmutableNameNotFoundException    1
  org.apache.naming.resources.ProxyDirContext                   1
  org.apache.naming.resources.ProxyDirContext$CacheEntry        5
  org.apache.naming.resources.ResourceAttributes                3
  org.apache.naming.resources.WARDirContext                     2
  org.apache.naming.resources.WARDirContext$WARResource         2
  org.apache.naming.resources.WARDirContext$Entry               2
  org.apache.naming.resources.WARDirContext$Entry[]             2

  Initially, I focused on the org.apache.catalina.core.StandardContext
  class.  It seemed like a nice entry point that scopes the Catalina
  classes supporting a web application deployment.

  It appears that an instance is pinned in several locations:


  1.  org.apache.naming.ContextBindings.bindContext() is called (in 
      org.apache.catalina.core.NamingContextListener.lifecycleEvent()
      given a org.apache.catalina.LifeCycleEvent whose getType() is
      org.apache.catalina.Lifecycle.START_EVENT).  This puts
      StandardContext into a static Hashtable within ContextBindings.
      This Hashtable entry is removed by a call to
      ContextBindings.unbindContext().  unbindContext() appears to never
      be called.

  2.  An org.apache.jasper.logging.DefaultLogger instance (which
      implements org.apache.jasper.logging.Logger) is created in
      org.apache.jasper.servlet.JspServlet.init().  DefaultLogger's
      setName() method is called resulting in the Logger being placed
      into a static Hashtable.  Entries in this Hashtable are removed
      via the Logger.close() and Logger.removeLogger(...) methods,
      neither of which appear to be called.

      DefaultLogger refers to a StandardContext via the following chain
      of references:

      DefaultLogger -> org.apache.catalina.core.ApplicationContextFacade
      -> org.apache.catalina.core.ApplicationContext -> StandardContext

  3.  The DefaultLogger created in JspServlet is also referenced
      by a static field in org.apache.jasper.Constants.  This field
      does not appear to be cleared.

  4.  org.apache.catalina.core.StandardHostDeployer has a
      static org.apache.catalina.Context field that is set to the
      suspect StandardContext after a call to its addChild() method
      (called reflectively by org.apache.commons.digester.Digester).
      This field is not unset unless the install(URL, URL) method
      is called (but not if the other install(String, URL) or either
      of the remove() methods is called).

      This context can be replaced if addChild() is called again,
      but would this happen only if another deployment occured?
      If so, StandardContext is still "pinned" if the web application
      is left undeployed and a subsequent deployment does not occur.

  5.  One of the Digester instances can also periodically hold onto
      a StandardContext (as its root), but this reference can and 
      appears to be replaced (via Digester.push() when its stack
      is empty). I have not analyzed when this may be the case,
      so similar to (4), it seems possible that StandardContext
      will be pinned if the Digester instance is not "reset."


  At this point in time, I have not analyzed memory leaks beyond
  references to StandardContext.  Many other "leaky" instances
  can be traced back to StandardContext (ApplicationContext,
  ApplicationContextFacade, StandardManager, etc...).

  The number of instances that appear to be leaking and the size
  of these instances is fairly small.  I would guess less than 2K.
  However, I suspect that these instances are pinnning classes.

  By instrumenting our code, I have been able to determine that
  instances are indeed being garbage collected.  On the other hand,
  I have been able to instrument WebappClassLoader and have not seen
  it finalize.  In our production environment, we are deploying a 
  Jetspeed portal which contains hundreds of classes which I believe
  can explain the limited number of times we are able to re-deploy
  before running out of memory.

  If you've managed to read through to here, any ideas or pointers
  would be greatly appreciated.

  Thanks,
  Ted Chen




POST 2:
http://www.mail-archive.com/tomcat-user@jakarta.apache.org/msg90811.html
----------------------------------------------------------------------
I believe I'm seeing a memory leak as a result of a Manager
deploy/undeploy.  I have a very simple test case:  a Servlet that has a
static field that refers to an object (Foo) that allocates a large chunk
of memory.  I've instrumented both the Servlet (init(), destroy(), and
finalize()) and Foo (ctor and finalize()).  The Servlet has been
configured to load on startup.

On a deploy, I see:

    Foo.ctor (during class initialization of the Servlet)
    Servlet.init():

On an undeploy, I see:

    Servlet.destroy()
    Servlet.finalize()

I never see Foo.finalize().  If I continue to deploy/undeploy
repeatedly, eventually the VM reports an OutOfMemoryError when I try to
deploy.  Running the VM with -verbose:gc and "encouraging" GC whenever
possible, I see that after each undeploy, memory usage goes up roughly
by what I've allocated in Foo.

Any ideas?
Comment 1 Tony Chao 2003-06-13 18:36:42 UTC
bug #11128 could be related (or the same)
Comment 2 Remy Maucherat 2003-06-14 07:50:59 UTC
*** Bug 11128 has been marked as a duplicate of this bug. ***
Comment 3 Remy Maucherat 2003-06-14 08:02:33 UTC
Thanks for summarizing the issue, that I will try to address. That's typically
is needed for TC 5.
Comment 4 Glenn Nielsen 2003-06-16 12:20:48 UTC
Thanks for the summary on this memory leak Tony.

I think this should be added to the list of changes/bug fixes for the
next Tomcat 4.1.XX release.
Comment 5 tagunov 2003-07-27 17:07:55 UTC
http://nagoya.apache.org/eyebrowse/ReadMsg?listName=tomcat-
dev@jakarta.apache.org&msgId=776903

seems to be adding to the topic:

Using 
OptimizeIt, I tried to understand the problem and I found that my application ClassLoader was 
never totally released
when I stop my application. There are always some strange references on 
it and especially the "classLoader of
java.security.ProtectionDomain", which comes from 
the org.apache.catalina.core.StandardContext.start() method.
Comment 6 Tim Golden 2004-03-23 14:07:45 UTC
We too are getting out of memory erros in Tomcat 4.1.24 JDK 1.4.x

Has this been fixed in a new 4.1 verion of tomcat?
Comment 7 Steve Menke 2004-09-02 18:54:14 UTC
The same problem still exists for 5.0.27. Can the version be changed to 
reflect this.

I have some rather large singletons that do not get garbage collect on 
undeploy. This renders the undeploy/deploy features utterly useless.

Bug 11128 has a suggested fix. Why have they not been commented on or 
implemented?
Comment 8 Mark Thomas 2004-09-02 21:47:12 UTC
In summary it looks like there are three issues here:
1. StandardContext
2. Static references in servlets
3. ClassLoader

The issues may or may be not be inter-related.

I have just downloaded a profiler and started to have a look at these. I'll 
post results as I get them.

On the subject of changing version, lets leave it as TC4 for now. If I make 
any changes I'll make them to TC5 as well.
Comment 9 Mark Thomas 2004-09-02 22:36:28 UTC
The profiler shows that the static references in servlets are being cleaned 
up. Therefore 2 is not an issue.

On re-reading this report it looks like there might be a fourth issue. Steve - 
can you provide me with a simple test case for your singleton issue?

In summary:
1. Still to investigate
2. No issue
3. Still to investigate
4. Need more info
Comment 10 Donald Ball 2004-09-03 01:57:02 UTC
Is it possible that a static reference to the webapp's classloader could keep
the servlet context components from being garbage collected? I've seen hints
that this might be the case, particularly w.r.t. webapps that use cglib and/or
hibernate.
Comment 11 Steve Menke 2004-09-03 02:11:48 UTC
Created attachment 12622 [details]
Singleton Test Servlet
Comment 12 Steve Menke 2004-09-03 02:12:25 UTC
Created attachment 12623 [details]
Singleton Test Class
Comment 13 Steve Menke 2004-09-03 02:13:30 UTC
Created attachment 12624 [details]
Large object to load via reflection
Comment 14 Steve Menke 2004-09-03 16:25:53 UTC
I applogize. My previous posting was incorrect and does not reproduce the 
problem. It grows until the garbage collector hits its first limit. However, I 
have a new one that definately reproduces the problem. It is related to using 
ObjectOutputStreams. There are many posting related to using reset() with 
ObjectOutputStreams which I do. I also have a simple test code that repeatidly 
writes Object to a file with no problem. However, if put it in a servlet and 
continualy reload, it leaks. Could it be related to the ClassLoader?
Comment 15 Steve Menke 2004-09-03 16:27:21 UTC
Created attachment 12638 [details]
Revised BigSingleton, which writes objects to disk.
Comment 16 Steve Menke 2004-09-03 16:28:21 UTC
Created attachment 12639 [details]
Revised Singleton Servlet.
Comment 17 Mark Thomas 2004-09-05 23:26:34 UTC
Quick update:
1. Worst offenders patched in TC4 (already pacthed in TC5). Need to work out 
if the remaining potential offenders are really an issue.
2. No issue.
3. Directly related to 1.
4. Still to investigate.
Comment 18 Mark Thomas 2004-09-06 21:17:09 UTC
All should now be fixed. For the record:
1. Problem exactly as described in original report. All StandardContext issues 
fixed in TC4. Nearlly all had already been applied to TC5. The remaining one 
has been applied.
2. Could not reproduce.
3. Same problem as 1.
4. Unable to reproduce after fixing 1.
Comment 19 Mark Thomas 2005-03-24 23:57:23 UTC
*** Bug 3888 has been marked as a duplicate of this bug. ***