Bug 3888 - WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
Status: RESOLVED DUPLICATE of bug 20758
Product: Tomcat 4
Classification: Unclassified
Component: Catalina
4.0 Final
All All
: P1 blocker with 21 votes (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
: 24847 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2001-09-30 11:39 UTC by Jon Stevens
Modified: 2005-03-24 14:57 UTC (History)
12 users (show)



Attachments
sample webapp to test log4j jar being locked even after Tomcat manager remove (12.94 KB, application/octet-stream)
2002-10-22 02:53 UTC, Jacob Kjome
Details
See the README file. (4.47 KB, application/octet-stream)
2002-10-22 15:03 UTC, Ceki Gulcu
Details
A new version of log4j that solves the log4j.jar locking problem. (340.53 KB, application/octet-stream)
2002-10-22 17:41 UTC, Ceki Gulcu
Details
ant buld of a webapp reproducing the CL error....requires libraries in next attachment... (268.01 KB, application/octet-stream)
2002-10-30 06:47 UTC, Jacob Kjome
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jon Stevens 2001-09-30 11:39:19 UTC
Every once and a while I will get this...

WebappClassLoader:   Resource 
'/WEB-INF/classes/org/tigris/scarab/screens/Default.class' was modified; 
Date is now: Sun Sep 30 10:09:30 PDT 2001 Was: Sun Sep 30
 10:07:31 PDT 2001
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
...

Here is the log file...

2001-09-30 10:07:49 invoker: init
2001-09-30 10:07:49 scarab: init
2001-09-30 10:09:35 StandardContext[/scarab]: Reloading this Context 
has started
2001-09-30 10:09:37 StandardManager[/scarab]: Stopping
2001-09-30 10:09:37 StandardManager[/scarab]: Unloading persisted 
sessions
2001-09-30 10:09:37 StandardManager[/scarab]: Saving persisted 
sessions to SESSI
ONS.ser
2001-09-30 10:09:37 StandardManager[/scarab]: Unloading 1 sessions
2001-09-30 10:09:37 StandardManager[/scarab]: writeObject() storing 
session 8544
C68035258287B9D54201D235D0F2
2001-09-30 10:09:37 StandardManager[/scarab]: Expiring 1 persisted 
sessions
2001-09-30 10:09:37 StandardManager[/scarab]: Unloading complete
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploying class 
repositories to work
directory /Users/jon/checkout/scarab/target/work/localhost/scarab
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/activation-1.
0.1.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib
/activation-1.0.1.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/commons-colle
ctions.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/
lib/commons-collections.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/commons-email
-0.1-dev.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-IN
F/lib/commons-email-0.1-dev.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/commons-util-
0.1-dev.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF
/lib/commons-util-0.1-dev.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/ecs-1.4.1.jar
 to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/ec
s-1.
4.1.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/flux-3.0-dev.
jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/flu
x-3.0-dev.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/fulcrum-1.0.j
ar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/ful
c
rum-1.0.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/jakarta-regex
p-1.3-dev.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-I
NF/lib/jakarta-regexp-1.3-dev.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/jdbc2_0-stdex
t.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/j
dbc2_0-stdext.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/junit-3.7.jar
 to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/jun
it-
3.7.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/log4j-1.1.jar
 to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/log
4j-
1.1.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/lucene.jar to
 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/luc
ene.ja
r
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/mail-1.2.jar
to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/m
ail-1.
2.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/mm.mysql-2.0.
4.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/m
m.mysql-2.0.4.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/torque-1.0.ja
r to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/tor
qu
e-1.0.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/turbine-3.0-d
ev.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/
turbine-3.0-dev.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/velocity-1.2-
dev.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib
/velocity-1.2-dev.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/village-1.5.2
.jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/vi
llage-1.5.2.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Deploy JAR 
/WEB-INF/lib/xerces-1.3.0.
jar to 
/Users/jon/checkout/scarab/target/./bin/../webapps/scarab/WEB-INF/lib/xer
ces-1.3.0.jar
2001-09-30 10:09:37 WebappLoader[/scarab]: Reloading checks are 
enabled for this
 Context
2001-09-30 10:09:37 StandardWrapper[/scarab:default]: Loading 
container servlet
default
2001-09-30 10:09:37 default: init
2001-09-30 10:09:37 StandardWrapper[/scarab:invoker]: Loading 
container servlet
invoker
2001-09-30 10:09:37 invoker: init
2001-09-30 10:09:37 scarab: init
2001-09-30 10:09:42 StandardManager[/scarab]: Starting
2001-09-30 10:09:42 StandardManager[/scarab]: Force random number 
initialization
 starting
2001-09-30 10:09:42 StandardManager[/scarab]: Seeding random 
number generator cl
ass java.security.SecureRandom
2001-09-30 10:09:42 StandardManager[/scarab]: Seeding of random 
number generator
 has been completed
2001-09-30 10:09:42 StandardManager[/scarab]: Force random number 
initialization
 completed
2001-09-30 10:09:42 StandardManager[/scarab]: Start: Loading persisted 
sessions
2001-09-30 10:09:42 StandardManager[/scarab]: Loading persisted 
sessions from SE
SSIONS.ser
2001-09-30 10:09:42 StandardManager[/scarab]: Creating custom object 
input strea
m for class loader WebappClassLoader
  available:
    Extension[org.apache.commons.collections, 
implementationVendor=Apache Softwa
re Foundation, implementationVersion=1.1-dev, 
specificationVendor=Apache Softwar
e Foundation, specificationVersion=1.0]
    Extension[@name@, implementationVendor=Apache Software 
Foundation, implement
ationVersion=0.1-dev, specificationVendor=Apache Software Foundation, 
specificat
ionVersion=1.0]
    Extension[@name@, implementationVendor=Apache Software 
Foundation, implement
ationVersion=0.1-dev, specificationVendor=Apache Software Foundation, 
specificat
ionVersion=1.0]
    Extension[javax.mail, implementationVendor=Sun Microsystems, Inc., 
implement
ationVendorId=com.sun, implementationVersion=1.2, 
specificationVendor=Sun Micros
ystems, Inc., specificationVersion=1.2]
  delegate: false
  repositories:
    /WEB-INF/classes/
  required:
----------> Parent Classloader:
StandardClassLoader
  available:
  delegate: false
  repositories:
    file:/Users/jon/checkout/scarab/target/lib/naming-factory.jar
  required:
----------> Parent Classloader:
StandardClassLoader
  available:
  delegate: false
  repositories:
    file:/Users/jon/checkout/scarab/target/common/lib/crimson.jar
    file:/Users/jon/checkout/scarab/target/common/lib/jaxp.jar
    file:/Users/jon/checkout/scarab/target/common/lib/naming-common.jar
    
file:/Users/jon/checkout/scarab/target/common/lib/naming-resources.jar
    file:/Users/jon/checkout/scarab/target/common/lib/servlet.jar
  required:
----------> Parent Classloader:
sun.misc.Launcher$AppClassLoader@33056f



2001-09-30 10:09:42 StandardManager[/scarab]: Loading 1 persisted 
sessions
2001-09-30 10:09:42 StandardManager[/scarab]: readObject() loading 
session 8544C
68035258287B9D54201D235D0F2
2001-09-30 10:09:42 StandardManager[/scarab]: Finish: Loading 
persisted sessions
2001-09-30 10:09:42 StandardContext[/scarab]: Reloading this Context is 
complete
d
Comment 1 Jon Stevens 2001-09-30 11:40:04 UTC
According to Remy, this one isn't going to be easy to fix because it is so 
hard to duplicate. I'm putting it in the system in the hopes that others may 
help duplicate this error.
Comment 2 Pier Fumagalli 2001-10-08 11:38:48 UTC
I am going to check out this bug against Mac OS X 10.1 in the next few days.
Comment 3 Remy Maucherat 2001-10-10 01:02:01 UTC
Aftre more studying, it's extremely unlikely it's a Mac OS X specific problem 
(so reassigning to tomcat-dev). OTOH, it could be a case of static abuse (not 
Jon's fault, but maybe in one of the library used).

I've added some info in the release notes. Maybe they would apply to your 
situation:

<quote>
Some shared libraries (many are part of the JDK) keep references to objects
instantiated by the web application. To avoid class loading related problems
(ClassCastExceptions, messages indicating that the classloader 
is stopped, ...), the shared libraries state should be reinitialized.

Something which could help is to avoid putting classes which would be 
referenced by a shared static field in the web application classloader, 
and put them in the shared classloader instead (the JARs should be put in the 
"lib" folder, and classes should be put in the "classes" folder).
</quote>

Here, a candidate to cause trouble may be the JDBC driver (which gets 
registered in a static field in the system CL).

There are no workarounds, except:
- Reset the affected static field(s) on destroy() (once the problem has been 
isolated)
- Move away the libraries and put them in a shared loader
Comment 4 Remy Maucherat 2001-10-12 19:32:37 UTC
After much investigation, I think it should come from a "user error", caused by 
the libraries used (more details on what may cause this in the release notes).
Comment 5 Yan Li 2002-09-11 15:53:26 UTC
I couldn't find details about this bug in any release notes. Could you tell me
which release note I should look into? Or copy the details here so that it is
easier to find.

Thanks.
Comment 6 Martin van Dijken 2002-10-10 08:19:06 UTC
Getting this error on FreeBSD with Tomcat 4.0.3. Can't really tell why this is
going on. I have four webapps running. Three websites of our clients and the
default webapp. I can give many more details about this situation if you'd like,
but I can't really tell what is significant in this situation, so please let me
know what kind of details you need. I'd like to have it noted that in
catalina.out, about a 1/4 of the entire 65000 line log file consists of
WebappClassLoader: Lifecycle error : CL stopped. 
Comment 7 Remy Maucherat 2002-10-10 08:24:42 UTC
I cannot add more than what I already wrote in that report. I am convinced
(until actually proven wrong) that this is a user or application error.
Comment 8 Martin van Dijken 2002-10-10 09:51:32 UTC
Hey Remy,

You say it is a user error. The only reasonable user error I can find, is that
the placing of our lib files might leave something to question. Let me explain:

We have one big web application. A Content Management System we're developing.
Whenever we build a project using this Content Management System, we use the
latest stable release of the thing. Since new versions of libraries we have not
created come out all the time, we have the libraries in our web application
also. This so that when we build our project, we have those versions of certain
libraries that are compatible with our project. 

This does mean, that when we use for example log4j on one server, we do not
place it in $Tomcat/common/lib or $tomcat/lib but in every web-app's WEB-INF/lib
directory. 

As said, I know this is not a usage that the Tomcat documentation promotes. I do
however feel, that IF this is the reason Tomcat's classloaders are crashing,
this should not be possible to happen.

If this is NOT the user error you're talking about, then please explain what you
do mean. I've read the servlet spec, and read the tomcat documentation. If I
don't know, that means a whole hell of a lot of people don't know either.
Comment 9 Remy Maucherat 2002-10-10 11:36:25 UTC
About the library placement, you're free to do whatever you want, and we're not
promoting that you move it somewhere else.

You have to make sure that the lifecycle of the object you create and resources
you allocate (threads for example) are synced with the webapp lifecycle. When
there's a restart you have to destroy all leftover objects, and recreate them
when the webapp is initialized again.
You get the errors when some object which was loaded by the old webapp
classloader  tries to run code.
Comment 10 Jon Stevens 2002-10-10 17:28:47 UTC
>You get the errors when some object which was loaded by the old 
> webapp classloader  tries to run code.

Ah...this is the first time you have specified when the error occurs. Now, if 
the classloader has been properly destroyed (generally you need to set it 
to be null), why is the old code even able to run? Since the old code is 
loaded in the now null classloader, then the JVM should not allow that 
code to run. Maybe there is classloader caching going on (ie: the 
reference to the old classloader is not being properly destoryed)?

Also, I still say this is a bug in Tomcat's classloader system because I 
have been doing servlet development for years (this bug is now more 
than a year old) with several different servlet containers and have never 
seen this bug. In fact, it didn't start happening until I had started using that 
version of Tomcat 4.

Also, I still see this bug still happens all the time for me...even with the 
4.0.6 version of Tomcat.

Remy, something is broken and it isn't user error.

-jon
Comment 11 Remy Maucherat 2002-10-10 17:37:34 UTC
Hhmm, sorry, but I can't force finalization of objects, and the VM will indeed
allow code to continue running (which is normal since the old CL is not actually
destroyed). The old CL will not get finalized either since the objects in
question still reference it through their classdef.

Waiting for your patch to fix it (and I'll be handling some other issues
meanwhile) :)
Comment 12 Jon Stevens 2002-10-10 17:55:52 UTC
Well, that is the problem...the old CL is not destroyed. Why is it not set to 
null?

-jon
Comment 13 Remy Maucherat 2002-10-10 17:59:44 UTC
Catalina replaces the pointer from the old loader to the new one, removes
sessions and things. However, you could have a thread or some other component
holding the pointer to the objects, and preventing finalization.
Basically, if you can find some component in Catalina holding a pointer to
either one of the old object/classes/classloader, then the bug is valid.
Otherwise, it's not.
Comment 14 Jon Stevens 2002-10-10 18:10:30 UTC
If you null out the classloader, then all of the pointers become invalid 
because the classloader is the top pointer. Again, the problem is the fact 
that you don't null out the classloader.

I would like you to ask on jsr-053 if any other vendors have this problem. I 
bet you some beers at my nightclub that they don't. This is a tomcat 
specific problem.

-jon
Comment 15 Remy Maucherat 2002-10-10 20:35:50 UTC
I'll need some help finding out where I should null it out (WebappLoader.stop()
should be doing that, and the CL is also removed from the bindings table). Sorry
Jon, I'm just too stupid to figure it out :-(
Comment 16 Martin van Dijken 2002-10-11 07:39:15 UTC
There there, Remy, hush now:) 
Now see what you've done Jon? You've made him upset! That won't help him fix the
problem and we both want the problem fixed don't we?:)

OK without the BS: I've talked a little with Remy over private mail yesterday,
and with a friend of mine. In our server we'd forgotten to remove the examples
webapp. If there ever was a webapp which causes problems it's that bugger. I
removed it, and am getting a lot less errors. I still however did get some CL
errors overnight... A lot less, but they still exist. 

Now here's a question for you Remy, I have no webapps left that are set to
reloadable. Examples was the only one. You say this bug happens when a
classloader is replaced by a new classloader. Why is the classloader replaced in
the first place? Because of an attempt to conserve memory usage or something?

Furthermore, I think Jon is correct to say that when the Classloader is stopped,
it doesn't truly matter whether or not anything within this classloader had
references to anything within this classloader itself. These references must be
broken by VM. Otherwise the whole concept of the singleton (which has a static
reference to an instance of itself) is an instant memory leak. So, if this holds
true, it must mean, that something within a classloader holds a reference to
something in another classloader (I think this is the point you're trying to
make Remy?) Now, the only code I use from within my webapps, is the Servlet API
classes, which is in the common classloader I believe, and I use a resource pool
with mysql connections. However, I am fairly certain that I keep no connections
open. Jon, you're getting this problem as well, do you use Tomcat's resource
pooling? Maybe that's where we should start fishing. Anyway, let me know what
you think about this novel I've just written:)
Comment 17 Jon Stevens 2002-10-11 07:52:43 UTC
I think that part of the problem might be that when the classloader is 
dumped, there is a few cases where objects within the classloader are 
not first nulled out. I sent Remy a patch which I have not heard back from 
him on. It is worth a try at least.

Also, I don't have any of the example webapp's...and I can't tell where 
Scarab might be messing up because it is a very complex application 
(Scarab has already uncovered about 4-5 odd bugs in Tomcat so far...).

I also don't see how putting jar files in different locations should be a 
requirement of the user (ala Remy's claim of "user" error).

-jon
Comment 18 Jacob Kjome 2002-10-12 05:18:51 UTC
Would the fact that Tomcat is not nulling out the classloader fully upon 
application stop/remove/undeploy be the reason that the log4j jar in my 
webapp's WEB-INF/lib directory is being locked even after a 
stop/remove/undeploy?  I only get this behavior with log4j.  All other 
resources are let go.

If this bug *is* the reason the log4j jar is being locked until after a full 
Tomcat shutdown, then it resolves a question that the Log4j developers and I 
have been asking about where the problem lies with the locked log4j jar file.  
If the culprit is Tomcat, then, I guess, we'd just wait for this bug to be 
fixed.  If it is not, then I'd like to know that so we can direct our attention 
to finding the problem in Log4j.

thanks,

Jake
Comment 19 Carsten Woelk 2002-10-18 14:36:21 UTC
Hi.

I experienced the same problem and have been trying to find out something about
it. What I did is I modified the
org.apache.catalina.loader.WebappClassLoader.java and provoke an Exception in
it. Actually there are two places where this 'error' can occur, but in my case
it happens while trying to load a class.

So my debug part starts in line 1304 (tomcat 4.1.12) and looks like:
public Class loadClass(String name, boolean resolve)
    throws ClassNotFoundException {

    if (debug >= 2)
        log("loadClass(" + name + ", " + resolve + ")");
    Class clazz = null;

    String test = null;

    // Don't load classes if class loader is stopped
    if (!started) {
        log("###CLASS[" + name + "]### Lifecycle error : CL stopped");
        try {
          test.length();
        }
        catch(Exception e) {
          e.printStackTrace();
        }
    ...

When the error occurs (strangely not on the first touch of a jar package,
instead only every after the first) it produces (in my case) the following output:

WebappClassLoader: ###CLASS[org.apache.log4j.helpers.NullEnumeration]###
Lifecycle error : CL stopped
java.lang.NullPointerException
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1317)
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1274)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:322)
        at org.apache.log4j.Category.getAllAppenders(Category.java:394)
        at
org.apache.velocity.runtime.log.SimpleLog4JLogSystem.shutdown(SimpleLog4JLogSystem.java:200)
        at
org.apache.velocity.runtime.log.SimpleLog4JLogSystem.finalize(SimpleLog4JLogSystem.java:194)
        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

So it looks like that the Finalizer calls (in my case) velocity's
SimpleLog4JLogSystem's finalize method which it self calls Log4J's Category's
getAllAppenders method calls. This uses the static method from NullEnumeration
to return an instance variable. The problem is that this instance
is created by new NullEnumeration() so I guess that then this class has to be
found with the help of catalina's WebappClassLoader.

Please find here an extract:

public class NullEnumeration implements Enumeration {
  private static final NullEnumeration instance = new NullEnumeration();
...
  public static NullEnumeration getInstance() {
    return instance;
  }
...

Perhaps this NullEnumeration should be instantiated once in Log4J's init
procedure in order to have an instance already loaded in memory and the class
already resolved or velocity should not try to close all appenders in its
finalize. In my case I have a start-up servlet in my webapp which now contains
the following line in it's init method:

logger.debug("Log4J instance now in memory: " +
org.apache.log4j.helpers.NullEnumeration.getInstance());

And this works fine for me... So I am not an expert, but I hope that helps
everybody to sort out their problems because after all I don't think that this
is a proper bug. Due to the fact that is my first active part, comments and
critics of this posts are very welcome.

Best regards
Carsten Woelk
Comment 20 Remy Maucherat 2002-10-18 17:28:35 UTC
If that is effectively the cause of the error, maybe log4j or Velocity could
take care of the needed instanciation.
Comment 21 Jon Stevens 2002-10-18 18:47:30 UTC
added ceki for his input.
Comment 22 Ceki Gulcu 2002-10-20 20:13:27 UTC
Indeed, Mark Womack and I have conversed with Jacob Kjome about log4j.jar being 
locked. Jacob supplied a test case reproducing the problem which I found to be 
somewhat complicated. I would very much like to see a simpler test case to 
address the bug dicovered by Jacob.

As for the problem reported by Carsten Woelk, as far as I can tell,  

public class X {
  private static final X instance = new X();
...
  public static X getInstance() {
    return instance;
  }

is just a classical pattern. What is unsafe with it?
Comment 23 Jacob Kjome 2002-10-20 23:05:00 UTC
Hi Ceki,

Did you check out the testcase that Mark Womack created.  Was that one not 
simplified enough?

http://marc.theaimsgroup.com/?l=log4j-dev&m=103371229121352&w=2

Look at the bottom of that message and you'll see a zip file containing a 
webapp to try out.

Note that some of the stuff in this message is a bit dated.  I tested Mark's 
findings and found that both the static and non-static test resulted in the 
log4j jar being locked and after testing again, Mark found that result as 
well.  Other than that, you can follow Mark's instructions on testing the app 
under Tomcat.

One thing to think about is whether commons-logging is getting a handle on 
log4j.  It really shouldn't because the parent classloader shouldn't be able to 
see the webapp classloaders, but I'm beginning to wonder if I can make that 
assumption?  The isssue could be either Tomcat not nulling out some objects in 
the classloader before Tomcat stops and app or commons-logging gaining a handle 
on log4j and not letting go until server shutdown or maybe a combination of 
both.

So far, the only viable solution I see is to use a RepositorySelector and keep 
log4j in one of the parent classloaders.  That shouldn't be necessary, though.  
Anyone have any bright ideas?

Jake
Comment 24 Ceki Gulcu 2002-10-21 08:33:53 UTC
Mark's message does not include source code for Log4jApplicationWatch and 
Log4jInit classes. Moreover, Mark reported that he could not reproduce the 
static-reference-jar-locking problem.
 
I need a simple test case, with source code, capabable of reproducing the 
problem. Is that too much to ask for? 

Ceki

ps: Mark CCed for reference.
Comment 25 Jacob Kjome 2002-10-22 02:53:25 UTC
Created attachment 3561 [details]
sample webapp to test log4j jar being locked even after Tomcat manager remove
Comment 26 Jacob Kjome 2002-10-22 03:05:03 UTC
The testcase I just uploaded provides Mark's testcase and includes a simplified 
version of my Log4jInit servlet.  Logging is done via both a console appender 
and a FileAppender.  The file appender is set up with a dynamic variable which 
points to where to write the file.  As long as you install the app with the 
path "/locktest", you will find the file "main.log" in WEB-INF/logs of this 
webapp.  No need to configure anything.  It is all set up and ready to run.

All source code including the classes that Mark didn't provide source to are 
provided (I decompiled his classes.  I'm sure he doesn't mind).  All source 
resides in the root of the webapp as well as the .jsp files which are what you 
need to run to see logging output.  There is also a "running.txt" file which 
describes the URL to use to install/remove the webapp.  Modify those example 
URL's to point to the "log4j_locktest" directory, which is the root of the 
webapp, on your system.

Make sure this is installed *outside* of Tomcat's "webapps" directory.  Any 
other place on your file system is appropriate.

Any questions, let me know.

Jake
Comment 27 Jacob Kjome 2002-10-22 03:09:15 UTC
Sorry for the spam,

Forgot to mention to put a recent version of log4j.jar in WEB-INF/lib before 
installing this in Tomcat.

You should notice after the removal of the webapp that the log file is able to 
be deleted and the static_test.jar is able to be deleted.  However the 
log4j.jar is locked until you shutdown Tomcat.  That is the problem we are 
trying to solve.

Jake
Comment 28 Ceki Gulcu 2002-10-22 15:03:59 UTC
Created attachment 3567 [details]
See the README file.
Comment 29 Ceki Gulcu 2002-10-22 15:10:53 UTC
I added a new tarball lock.tar.gz as an attachement. It contains a README file 
explaining its usage. 

My preliminary conclusion is that there is indeed an undesired interaction 
between log4j and Tomcat when the DOMConfigurator is used but not otherwise. 

The problem has been reduced to log4j/DOMConfigurator.

I don't think the indetified bug is directly linked to the bug reported by Jon. 
Jon are you using the DOMConfigurator?
Comment 30 Jon Stevens 2002-10-22 17:39:14 UTC
I don't believe so. All of my log4j configuration is done with properties 
files.
Comment 31 Ceki Gulcu 2002-10-22 17:41:53 UTC
Created attachment 3569 [details]
A new version of log4j that solves the log4j.jar locking problem.
Comment 32 Martin van Dijken 2002-10-23 07:36:35 UTC
I don't really want to further spam the discussion, but I'd just like it noted
that after removing the examples webapp from my Tomcat, I've not experienced any
problems any more. I DO use log4j, maybe a slightly older version, but I still
do use it. I have not experienced any CL errors any more.
Comment 33 Jacob Kjome 2002-10-30 06:47:37 UTC
Created attachment 3657 [details]
ant buld of a webapp reproducing the CL error....requires libraries in next attachment...
Comment 34 Jacob Kjome 2002-10-30 07:25:03 UTC
Ok, well, as it turns out, the second attachment seems too big for Bugzilla to 
handle.  It is moderately large, but not that big; about 2,386k.

Here is a link to it:
ftp://ftp.visi.com/users/hoju/pub/barracuda-libs.zip

Here is a link to the Ant build, for good measure:
ftp://ftp.visi.com/users/hoju/pub/barracuda-pages.sample_test.zip

Instructions on what everything is and how to use it is in a README.txt file in 
the root directory of the ant build in barracuda-pages.sample_test.zip.

Basically, extract barracuda-pages.sample_test.zip to any directory and extract 
barracuda-libs.zip to $CATALINA_HOME.  It will put the jar files where they 
need to go.  Again, more instructions are in the README.txt and they detail 
what libraries go where.

This setup is meant to be a completely reproduceable test showing the 
"...CL stopped" error.  It runs the first time the app is installed, but after 
removal and then another install, trying to pull up a dynamic page causes a 
java.lang.ClassNotFoundException and the "...CL stopped" error shows up in the 
stdout.log.

A similar configuration, but with a few of the libraries copied to WEB-INF/lib 
makes everything work flawlessly no matter how many install/remove cycles are 
done.

Step-by-step instructions are in README.txt of the Ant build.

let me know if you have any questions or difficulties.

Jake
Comment 35 Jacob Kjome 2002-11-01 01:53:47 UTC
Just wanted to note that I updated the libraries at the ftp addresses that I 
mentioned in my previous comment.

You can follow the instructions like normal, but if you want to see it working, 
you only have to copy barracuda-core.jar to WEB-INF/lib.  all other jars can 
stay where they are.  With barracuda-core.jar in WEB-INF/lib, the "...CL 
stopped" error doesn't ever happen, even after multiple remove/install cycles.

Oh, and Ceki asked me before whether the testcase was meant to show problems 
with log4j.  Just to clarify, this is *not* a log4j issue as far as I can tell.

Jake
Comment 36 Chris Forbis 2002-12-06 20:38:08 UTC
I just wanted to add a little on this bug.  I have a large webapplication that 
runs in tomcat 4.0.3 and 4.1 fine, but if I go to run in on JBoss (with tomcat 
inside jboss) I get this error ALL the time.  It happeneds as soon as tomcat 
reloads my web app (when jboss figures out my web.xml was touched and tells 
tomcat to reload my app).

Just figured I would add this because it might be anotehr thing worth testing.
I will look into it a little more myself, but wanted to let people know what I 
have seen.
Comment 37 Jon Stevens 2002-12-08 22:02:27 UTC
I can confirm that this still occurs with 4.1.12 and Scarab.
Comment 38 Boris Maras 2002-12-09 17:23:39 UTC
I also confirm that it occurs with Tomcat 4.0.3 and 4.1.12. I don't use log4j. 
My platform is a PC with Windows NT 4.
It seems to happen only when the webapp is restarted (because of a class or 
property file changed)

I'll try to track what has been done when it happens
Comment 39 Boris Maras 2002-12-10 13:11:26 UTC
I tried to remove the directory "examples" of Tomcat, as suggested in this 
thread : same behaviour

It seems like the lifecycle error occurs each time a .properties file is 
modified

Hope this helps
Comment 40 Jon Stevens 2002-12-20 05:02:59 UTC
I can now say that I most commonly see this error when I double compile. 

In other words, I will re-compile and it will cause a classloader change. 
Then, if I re-compile once more before loading the servlet, I will see this 
error.

I also see this error happen more often when the files that change are not 
.class files. Say for example, I have a resource bundle .properties file in 
my classpath that requires a change...for example:

WebappClassLoader:   Resource '/WEB-INF/classes/
ScarabBundle.properties' was modified; Date is now: Thu Dec 19 
20:57:57 PST 2002 Was: Thu Dec 19 20:09:44 PST 2002
WebappClassLoader:   Resource '/WEB-INF/classes/org/tigris/scarab/
om/Issue.class' was modified; Date is now: Thu Dec 19 20:59:04 PST 
2002 Was: Thu Dec 19 20:10:42 PST 2002
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped

Anyway, maybe that is more hints.
Comment 41 Jon Stevens 2002-12-20 05:03:43 UTC
adding remy to this issue since he is the only one out there that can fix it.
Comment 42 Jacob Kjome 2003-02-24 04:14:15 UTC
Note sure if this is still being investigated or not but I noticed some pretty
bad behavior related to this bug under Tomcat-5.  The symptom is triggered the
same way for both Tomcat-4.1.x and tomcat-5 (with slight differences...read on).
 However, after getting the lifecycle error under Tomcat4, the context continues
to work.  After gettting the error under Tomcat5, the context quits working. 
The manager app reports it as not being started.

Here is how I reproduce the problem...

1.  Start Tomcat (4 or 5)
2.  Use the catalina-ant task to install/remove my app.  I go through the
following separate steps...
    ant catalina-install
    ant catalina-remove catalina-install
    ant catalina-remove catalina-install
    ant catalina-remove superclean catalina-install

The first 3 separate steps result in the context starting just fine.  However,
after the fourth step, the context fails to start.  Note that "superclean"
cleans up everything forcing a full clean rebuild.

I actually am not getting the "INFO: Lifecycle error : CL stopped" error unless
I do Log4j configuration.  However, I've somewhat ruled out Log4j as the culprit
because, although I don't get said error printed out when I don't explicitly
configure Log4j, I get the same behavior of the context not starting after the
4th step in either case.  Doing the configure must just trigger the code that
actually prints out that debugging where that same debugging code isn't
triggered otherwise.

Below is what is printed out in stdout.log in Tomcat5 when I do Log4j
configuration.  Notice that, somewhat curiously, the number of classloader
errors matches the number of times that the app was installed/removed (not
including the initial install).  Also note that I end up with an
"java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl" error in Tomcat5 where in
Tomcat4, I get   "javax.xml.parsers.FactoryConfigurationError: Provider
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found" ...


Starting service Tomcat-Standalone
Apache Tomcat/5.0
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Feb 23, 2003 9:00:25 PM org.apache.catalina.loader.WebappClassLoader
findResourceInternal
INFO: Lifecycle error : CL stopped
java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1199)
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1159)
	at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:93)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:172)
	at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:93)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:644)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:616)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:584)
	at org.apache.log4j.xml.XMLWatchdog.doOnChange(DOMConfigurator.java:815)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:80)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:99)
Feb 23, 2003 9:00:25 PM org.apache.catalina.loader.WebappClassLoader
findResourceInternal
INFO: Lifecycle error : CL stopped
java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1199)
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1159)
	at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:93)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:172)
	at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:93)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:644)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:616)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:584)
	at org.apache.log4j.xml.XMLWatchdog.doOnChange(DOMConfigurator.java:815)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:80)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:99)
Feb 23, 2003 9:00:26 PM org.apache.catalina.loader.WebappClassLoader
findResourceInternal
INFO: Lifecycle error : CL stopped
java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1199)
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1159)
	at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:93)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:172)
	at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:93)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:644)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:616)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:584)
	at org.apache.log4j.xml.XMLWatchdog.doOnChange(DOMConfigurator.java:815)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:80)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:99)
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.


BTW, I am using Log4j-1.2.8 in WEB-INF/lib and have removed the Xerces that
comes with Barracuda in favor of Xerces-1.4.4 in common/endorsed because,
currently, XMLC-2.1 (xmlc.enhydra.org) requires that version of Xerces.  I have
noticed no issues with this setup other than when I do a clean build after
having installed the app in Tomcat.  Removing the app, then doing a clean build,
and then installing it again is when I get the bad behavior...and, of course,
worse behavior in Tomcat5.

So, does this help?  Any new ideas on why this might be happening?  Any fixes in
the works?

Jake
Comment 43 scott 2003-02-24 18:58:29 UTC
ClassLoader stopped on java.lang.DoubleI'm using OS X Server (latest rev), jav 1.3.1 and Tomcat 4.1.18. We have a thread that runs once/day. It will run for maybe a day or two successfully, but then the ClassLoader gives the following:WebappClassLoader: Lifecycle error : CL stoppedjava.lang.NoClassDefFoundError: java/lang/Double        at com.attask.AtProjects.setProj_invoiceAmount(AtProjects.java:1049)        at com.attask.AtProjects.populateFromResultSet(AtProjects.java:1210)        at com.attask.AtListBean.createListItem(AtListBean.java:238)        at com.attask.AtListBean.setList(AtListBean.java:127)        at com.attask.AtListBean.setList(AtListBean.java:67)        at com.attask.AtThreadRecalcTimelines.run(AtThreadRecalcTimelines.java:55)        at com.attask.AtThread.run(AtThread.java:101)This wreaks havoc on the data. Interesting that java.lang.NoClassDefFoundError is in the same package as java.lang.Double, but Double could not be found.-Scott
Comment 44 Curtis Paris 2003-05-06 18:28:05 UTC
Just to chime in on this ticket, we ran into this exception today as well.  
Although it wasn't with log4j or anything else.  It occured during a load test 
on the server.  It doesn't appear we ran out of Heap or Stack.  Our -Xmx/ms 
settings were our norm for a buffer during the test.

The only simularity is that we do run all of the JAR's out of the application 
WEB-INF/lib path.  There was no new jars/classes installed during this test.  
We were not using JSP's during this load test, rather just using Tomcat as a 
servlet engine for now.  Although I think we may still have reloadable="true" 
on it.

We had this exception occur while trying to perform some encryption within our 
application, using the SunJCE provider.

WebappClassLoader: Lifecycle error : CL stopped
java.security.NoSuchAlgorithmException: Algorithm DES not available
        at javax.crypto.SunJCE_b.a(DashoA6275)
        at javax.crypto.SecretKeyFactory.getInstance(DashoA6275)
        at com.metro1.encryption.CryptFactory.decryptKey(CryptFactory.java:155)
        at com.metro1.encryption.CryptFactory.<init>(CryptFactory.java:124)
        ... rest of the stack.


Is there anything that I can set in my server to give more information if we 
can reproduce this more regularly, other than during the load test?

I'm going to have our QA department turn of reloadable first.  If it 
continues, I'll have them move the encryption JAR's into common/lib and see if 
that resolves anything.
Comment 45 Remy Maucherat 2003-05-06 20:47:13 UTC
Your comments more or less prove that this bug is invalid (your classes are
referenced from a shared static object, and thus remain loaded after the webapp
reload). I will not attempt to close, resolve, or look further into this report,
which is an application problem, not a Tomcat bug.
Comment 46 Jon Stevens 2003-05-06 20:53:47 UTC
Remy,

How does the "applications WEB-INF/lib" consitute a "(your classes are
referenced from a shared static object, and thus remain loaded after the webapp
reload)"?????

Feel free to keep your head buried in the sand, however, this is a Tomcat
problem as I have never seen this problem before I used Tomcat or with any other
containers.

The way I currently duplicate it (using Scarab) is that it happens *every single
time* after exactly 3 classloader reloads. I suspect it is that there is a
memory leak with reloading which fills the memory up after 3 reloads and the JVM
blows up.
Comment 47 scott 2003-05-06 21:03:59 UTC
Agreed, this is a tomcat bug.
We are getting this error using 4.1.18 and jdk 1.4.1 the classloader is failing on java.lang.Double.
Comment 48 Eric Schwarzenbach 2003-06-13 23:10:27 UTC
I'm seeing this error with Tomcat 4.1.24 (LE jdk14), but under what seems a
somewhat different circumstance from what I'm seeing described here.

I have a servlet that performs a rather lengthy action (from doPost())--it takes
several minutes to complete. If I update the class while the doPost() is still
in the midst of processing, I see my servlet's destroy method called, then see
its init method called as it is reloaded. However the doPost method appears to
continue to run, judging by the output. It completes its task normally and
successfully, except that this "WebappClassLoader: Lifecycle error : CL stopped"
error keeps occuring.

I thought that destroy() was not supposed to be called until the service()
method has exited, which definitely has not happened in my case.

The reason I ran into this is I was testing the ability to take this servlet out
of service without risking interrupting current tasks...
Comment 49 Remy Maucherat 2003-06-14 07:49:46 UTC
Well, there's a timeout (basically, it will wait for 50ms 10 times, and then
give up and proceed with the stop; so in your case, I'd say it's not enough). I
think the number of retry attempts could be made to be configurable (hacking the
code is very simple - look in o.a.c.core.StandardWrapper.unload()). If you try
to run code on the discarded CL after the webapp has been reloaded, you'll get
the log messages, it's normal.
Comment 50 Eric Schwarzenbach 2003-06-16 15:21:20 UTC
I noticed the mention of a timeout in the serlvet docs. I take it this timeout 
isn't explicitly configurable (except by hacking) in Tomcat?

This behavior seems kind of unclean... If it's going to time out shouldn't the 
engine kill the thread(s) rather than letting it(them) continue to run and 
throw errors? (Incidentally I realized my post() didn't totally run to 
completion...in the middle of one of the last steps it throws a 
java.lang.NoClassDefFoundError: java/lang/reflect/InvocationTargetException
attempting to instantiate a object via pluggable factory-style method.)

I could write my own synchronization between destroy() and service() but it 
seems like the kind of functionality that the framework should provide. 
Comment 51 Norbert Pabis 2003-10-07 14:32:04 UTC
In 4.1.27 the bug still bites.
If someone wants to replicate the bug, try using jcoverage.
When the library was in WEB-INF/lib the bug occured.
When I moved the jar to TOMCAT/commons/lib everything was ok.

Anyway, thanks to Martin van Dijken and Remy Maucherat for insightful comments,
that helped me to find workaround.
Comment 52 Andre Schild 2003-10-07 16:37:41 UTC
Another way to reporduce it is to use swarmcache.
https://sourceforge.net/projects/swarmcache/

On stop of a context it can occure that Tomcat
throws this exception.

java.lang.NoClassDefFoundError: org/javagroups/stack/NakReceiverWindow$Entry
at org.javagroups.stack.NakReceiverWindow.add(NakReceiverWindow.java:199)
at org.javagroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:357)
at org.javagroups.protocols.pbcast.NAKACK.up(NAKACK.java:223)
at org.javagroups.stack.UpHandler.run(Protocol.java:50)
Comment 53 Mark Thomas 2003-11-23 17:00:55 UTC
*** Bug 24847 has been marked as a duplicate of this bug. ***
Comment 54 Dan Machak 2004-02-26 19:41:24 UTC
We have been seeing this same problem with 4.1.29. A jar file was still locked
after doing an undeploy. As a work-around, we found that if this jar file were
expanded into the WEB-INF/classes directory, the the undeploy was successful. 

It seems that this points to a problem with the ClassLoader closing the jar
file. Perhaps an exception during undeploy is causing the closing of the jar
file to be bypassed.
Comment 55 Jacob Kjome 2004-02-27 05:37:26 UTC
Hi Dan,

If you are seeing a jar that is locked, it means that some thread is still has a
handle to some resource in that jar (or the thread is created from a class in
the jar).  The fact that this isn't happening when you put the jar in
WEB-INF/classes should be obvious.  The jar was never loaded from
WEB-INF/classes.  Plain classes are loaded there, not jars.

You need to make sure what is still holding a handle to a resource in that jar
and make sure it gets shut down at webapp shutdown.  Remy is correct in saying
that most of the issues here are due to user error, not a problem in Tomcat. 
That is not to say that this is necessarily true in all cases, but certainly most.

Jake
Comment 56 Dan Machak 2004-02-27 17:31:15 UTC
Hi Jacob,
I think you misunderstood what I was saying. We expanded the jar file into
WEB-INF/classes. That is, "jar -xf my.jar". We then removed the jar file from
WEB-INF/lib. Therefore, any resources that were previously being loaded from the
jar file would now be getting loaded from the classes directory. However,
whereas the jar file would be locked after undeploy in the previous case, we now
see undeploy happen successfully when the jar file is expanded. If it were any
of our threads holding a resource open, it still would have it open in the
classes directory. The only difference we should have seen were that individual
files would have been locked.

We arrived at this point after having eliminated several cases where our threads
were holding resources open. We've now verified this by getting a thread dump on
threads still around after the undeploy. Also, we have no problem in running the
exact same scenario in WebLogic.

We're still investigating the problem and should have more info from a profiling
tool pretty soon.
Comment 57 Mark Thomas 2005-03-24 23:57:10 UTC
I have reviewed the multiple issues raised in this bug report and believe they
are all either resolved or invalid. Siince the original report is actually a
duplicate, that is how this bug will be resolved.

A summary follows:

Issue 1
WebappClassLoader: Lifecycle error : CL stopped
As Remy commented (#13) "...Basically, if you can find some component in
Catalina holding a pointer to either one of the old object/classes/classloader,
then the bug is valid. Otherwise, it's not."
Although there was no evidence of this in this bug report see Bug 20758 for a
detailed examination of where this was happening.

Issue 2
log4j jar locking. As per Ceki's comment (#28), this has been fixed in log4j.

Other issues
A variety of libraries and applciations using shared static objects.

*** This bug has been marked as a duplicate of 20758 ***