Bug 3888

According to Remy, this one isn't going to be easy to fix because it is so 
hard to duplicate. I'm putting it in the system in the hopes that others may 
help duplicate this error.

I am going to check out this bug against Mac OS X 10.1 in the next few days.

Aftre more studying, it's extremely unlikely it's a Mac OS X specific problem 
(so reassigning to tomcat-dev). OTOH, it could be a case of static abuse (not 
Jon's fault, but maybe in one of the library used).

I've added some info in the release notes. Maybe they would apply to your 
situation:

<quote>
Some shared libraries (many are part of the JDK) keep references to objects
instantiated by the web application. To avoid class loading related problems
(ClassCastExceptions, messages indicating that the classloader 
is stopped, ...), the shared libraries state should be reinitialized.

Something which could help is to avoid putting classes which would be 
referenced by a shared static field in the web application classloader, 
and put them in the shared classloader instead (the JARs should be put in the 
"lib" folder, and classes should be put in the "classes" folder).
</quote>

Here, a candidate to cause trouble may be the JDBC driver (which gets 
registered in a static field in the system CL).

There are no workarounds, except:
- Reset the affected static field(s) on destroy() (once the problem has been 
isolated)
- Move away the libraries and put them in a shared loader

After much investigation, I think it should come from a "user error", caused by 
the libraries used (more details on what may cause this in the release notes).

I couldn't find details about this bug in any release notes. Could you tell me
which release note I should look into? Or copy the details here so that it is
easier to find.

Thanks.

Getting this error on FreeBSD with Tomcat 4.0.3. Can't really tell why this is
going on. I have four webapps running. Three websites of our clients and the
default webapp. I can give many more details about this situation if you'd like,
but I can't really tell what is significant in this situation, so please let me
know what kind of details you need. I'd like to have it noted that in
catalina.out, about a 1/4 of the entire 65000 line log file consists of
WebappClassLoader: Lifecycle error : CL stopped. 

I cannot add more than what I already wrote in that report. I am convinced
(until actually proven wrong) that this is a user or application error.

Hey Remy,

You say it is a user error. The only reasonable user error I can find, is that
the placing of our lib files might leave something to question. Let me explain:

We have one big web application. A Content Management System we're developing.
Whenever we build a project using this Content Management System, we use the
latest stable release of the thing. Since new versions of libraries we have not
created come out all the time, we have the libraries in our web application
also. This so that when we build our project, we have those versions of certain
libraries that are compatible with our project. 

This does mean, that when we use for example log4j on one server, we do not
place it in $Tomcat/common/lib or $tomcat/lib but in every web-app's WEB-INF/lib
directory. 

As said, I know this is not a usage that the Tomcat documentation promotes. I do
however feel, that IF this is the reason Tomcat's classloaders are crashing,
this should not be possible to happen.

If this is NOT the user error you're talking about, then please explain what you
do mean. I've read the servlet spec, and read the tomcat documentation. If I
don't know, that means a whole hell of a lot of people don't know either.

About the library placement, you're free to do whatever you want, and we're not
promoting that you move it somewhere else.

You have to make sure that the lifecycle of the object you create and resources
you allocate (threads for example) are synced with the webapp lifecycle. When
there's a restart you have to destroy all leftover objects, and recreate them
when the webapp is initialized again.
You get the errors when some object which was loaded by the old webapp
classloader  tries to run code.

>You get the errors when some object which was loaded by the old 
> webapp classloader  tries to run code.

Ah...this is the first time you have specified when the error occurs. Now, if 
the classloader has been properly destroyed (generally you need to set it 
to be null), why is the old code even able to run? Since the old code is 
loaded in the now null classloader, then the JVM should not allow that 
code to run. Maybe there is classloader caching going on (ie: the 
reference to the old classloader is not being properly destoryed)?

Also, I still say this is a bug in Tomcat's classloader system because I 
have been doing servlet development for years (this bug is now more 
than a year old) with several different servlet containers and have never 
seen this bug. In fact, it didn't start happening until I had started using that 
version of Tomcat 4.

Also, I still see this bug still happens all the time for me...even with the 
4.0.6 version of Tomcat.

Remy, something is broken and it isn't user error.

-jon

Hhmm, sorry, but I can't force finalization of objects, and the VM will indeed
allow code to continue running (which is normal since the old CL is not actually
destroyed). The old CL will not get finalized either since the objects in
question still reference it through their classdef.

Waiting for your patch to fix it (and I'll be handling some other issues
meanwhile) :)

Well, that is the problem...the old CL is not destroyed. Why is it not set to 
null?

-jon

Catalina replaces the pointer from the old loader to the new one, removes
sessions and things. However, you could have a thread or some other component
holding the pointer to the objects, and preventing finalization.
Basically, if you can find some component in Catalina holding a pointer to
either one of the old object/classes/classloader, then the bug is valid.
Otherwise, it's not.

If you null out the classloader, then all of the pointers become invalid 
because the classloader is the top pointer. Again, the problem is the fact 
that you don't null out the classloader.

I would like you to ask on jsr-053 if any other vendors have this problem. I 
bet you some beers at my nightclub that they don't. This is a tomcat 
specific problem.

-jon

I'll need some help finding out where I should null it out (WebappLoader.stop()
should be doing that, and the CL is also removed from the bindings table). Sorry
Jon, I'm just too stupid to figure it out :-(

There there, Remy, hush now:) 
Now see what you've done Jon? You've made him upset! That won't help him fix the
problem and we both want the problem fixed don't we?:)

OK without the BS: I've talked a little with Remy over private mail yesterday,
and with a friend of mine. In our server we'd forgotten to remove the examples
webapp. If there ever was a webapp which causes problems it's that bugger. I
removed it, and am getting a lot less errors. I still however did get some CL
errors overnight... A lot less, but they still exist. 

Now here's a question for you Remy, I have no webapps left that are set to
reloadable. Examples was the only one. You say this bug happens when a
classloader is replaced by a new classloader. Why is the classloader replaced in
the first place? Because of an attempt to conserve memory usage or something?

Furthermore, I think Jon is correct to say that when the Classloader is stopped,
it doesn't truly matter whether or not anything within this classloader had
references to anything within this classloader itself. These references must be
broken by VM. Otherwise the whole concept of the singleton (which has a static
reference to an instance of itself) is an instant memory leak. So, if this holds
true, it must mean, that something within a classloader holds a reference to
something in another classloader (I think this is the point you're trying to
make Remy?) Now, the only code I use from within my webapps, is the Servlet API
classes, which is in the common classloader I believe, and I use a resource pool
with mysql connections. However, I am fairly certain that I keep no connections
open. Jon, you're getting this problem as well, do you use Tomcat's resource
pooling? Maybe that's where we should start fishing. Anyway, let me know what
you think about this novel I've just written:)

I think that part of the problem might be that when the classloader is 
dumped, there is a few cases where objects within the classloader are 
not first nulled out. I sent Remy a patch which I have not heard back from 
him on. It is worth a try at least.

Also, I don't have any of the example webapp's...and I can't tell where 
Scarab might be messing up because it is a very complex application 
(Scarab has already uncovered about 4-5 odd bugs in Tomcat so far...).

I also don't see how putting jar files in different locations should be a 
requirement of the user (ala Remy's claim of "user" error).

-jon

Would the fact that Tomcat is not nulling out the classloader fully upon 
application stop/remove/undeploy be the reason that the log4j jar in my 
webapp's WEB-INF/lib directory is being locked even after a 
stop/remove/undeploy?  I only get this behavior with log4j.  All other 
resources are let go.

If this bug *is* the reason the log4j jar is being locked until after a full 
Tomcat shutdown, then it resolves a question that the Log4j developers and I 
have been asking about where the problem lies with the locked log4j jar file.  
If the culprit is Tomcat, then, I guess, we'd just wait for this bug to be 
fixed.  If it is not, then I'd like to know that so we can direct our attention 
to finding the problem in Log4j.

thanks,

Jake

Hi.

I experienced the same problem and have been trying to find out something about
it. What I did is I modified the
org.apache.catalina.loader.WebappClassLoader.java and provoke an Exception in
it. Actually there are two places where this 'error' can occur, but in my case
it happens while trying to load a class.

So my debug part starts in line 1304 (tomcat 4.1.12) and looks like:
public Class loadClass(String name, boolean resolve)
    throws ClassNotFoundException {

    if (debug >= 2)
        log("loadClass(" + name + ", " + resolve + ")");
    Class clazz = null;

    String test = null;

    // Don't load classes if class loader is stopped
    if (!started) {
        log("###CLASS[" + name + "]### Lifecycle error : CL stopped");
        try {
          test.length();
        }
        catch(Exception e) {
          e.printStackTrace();
        }
    ...

When the error occurs (strangely not on the first touch of a jar package,
instead only every after the first) it produces (in my case) the following output:

WebappClassLoader: ###CLASS[org.apache.log4j.helpers.NullEnumeration]###
Lifecycle error : CL stopped
java.lang.NullPointerException
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1317)
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1274)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:322)
        at org.apache.log4j.Category.getAllAppenders(Category.java:394)
        at
org.apache.velocity.runtime.log.SimpleLog4JLogSystem.shutdown(SimpleLog4JLogSystem.java:200)
        at
org.apache.velocity.runtime.log.SimpleLog4JLogSystem.finalize(SimpleLog4JLogSystem.java:194)
        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

So it looks like that the Finalizer calls (in my case) velocity's
SimpleLog4JLogSystem's finalize method which it self calls Log4J's Category's
getAllAppenders method calls. This uses the static method from NullEnumeration
to return an instance variable. The problem is that this instance
is created by new NullEnumeration() so I guess that then this class has to be
found with the help of catalina's WebappClassLoader.

Please find here an extract:

public class NullEnumeration implements Enumeration {
  private static final NullEnumeration instance = new NullEnumeration();
...
  public static NullEnumeration getInstance() {
    return instance;
  }
...

Perhaps this NullEnumeration should be instantiated once in Log4J's init
procedure in order to have an instance already loaded in memory and the class
already resolved or velocity should not try to close all appenders in its
finalize. In my case I have a start-up servlet in my webapp which now contains
the following line in it's init method:

logger.debug("Log4J instance now in memory: " +
org.apache.log4j.helpers.NullEnumeration.getInstance());

And this works fine for me... So I am not an expert, but I hope that helps
everybody to sort out their problems because after all I don't think that this
is a proper bug. Due to the fact that is my first active part, comments and
critics of this posts are very welcome.

Best regards
Carsten Woelk

If that is effectively the cause of the error, maybe log4j or Velocity could
take care of the needed instanciation.

added ceki for his input.

Indeed, Mark Womack and I have conversed with Jacob Kjome about log4j.jar being 
locked. Jacob supplied a test case reproducing the problem which I found to be 
somewhat complicated. I would very much like to see a simpler test case to 
address the bug dicovered by Jacob.

As for the problem reported by Carsten Woelk, as far as I can tell,  

public class X {
  private static final X instance = new X();
...
  public static X getInstance() {
    return instance;
  }

is just a classical pattern. What is unsafe with it?

Hi Ceki,

Did you check out the testcase that Mark Womack created.  Was that one not 
simplified enough?

http://marc.theaimsgroup.com/?l=log4j-dev&m=103371229121352&w=2

Look at the bottom of that message and you'll see a zip file containing a 
webapp to try out.

Note that some of the stuff in this message is a bit dated.  I tested Mark's 
findings and found that both the static and non-static test resulted in the 
log4j jar being locked and after testing again, Mark found that result as 
well.  Other than that, you can follow Mark's instructions on testing the app 
under Tomcat.

One thing to think about is whether commons-logging is getting a handle on 
log4j.  It really shouldn't because the parent classloader shouldn't be able to 
see the webapp classloaders, but I'm beginning to wonder if I can make that 
assumption?  The isssue could be either Tomcat not nulling out some objects in 
the classloader before Tomcat stops and app or commons-logging gaining a handle 
on log4j and not letting go until server shutdown or maybe a combination of 
both.

So far, the only viable solution I see is to use a RepositorySelector and keep 
log4j in one of the parent classloaders.  That shouldn't be necessary, though.  
Anyone have any bright ideas?

Jake

Mark's message does not include source code for Log4jApplicationWatch and 
Log4jInit classes. Moreover, Mark reported that he could not reproduce the 
static-reference-jar-locking problem.
 
I need a simple test case, with source code, capabable of reproducing the 
problem. Is that too much to ask for? 

Ceki

ps: Mark CCed for reference.

Created attachment 3561 [details]
sample webapp to test log4j jar being locked even after Tomcat manager remove

The testcase I just uploaded provides Mark's testcase and includes a simplified 
version of my Log4jInit servlet.  Logging is done via both a console appender 
and a FileAppender.  The file appender is set up with a dynamic variable which 
points to where to write the file.  As long as you install the app with the 
path "/locktest", you will find the file "main.log" in WEB-INF/logs of this 
webapp.  No need to configure anything.  It is all set up and ready to run.

All source code including the classes that Mark didn't provide source to are 
provided (I decompiled his classes.  I'm sure he doesn't mind).  All source 
resides in the root of the webapp as well as the .jsp files which are what you 
need to run to see logging output.  There is also a "running.txt" file which 
describes the URL to use to install/remove the webapp.  Modify those example 
URL's to point to the "log4j_locktest" directory, which is the root of the 
webapp, on your system.

Make sure this is installed *outside* of Tomcat's "webapps" directory.  Any 
other place on your file system is appropriate.

Any questions, let me know.

Jake

Sorry for the spam,

Forgot to mention to put a recent version of log4j.jar in WEB-INF/lib before 
installing this in Tomcat.

You should notice after the removal of the webapp that the log file is able to 
be deleted and the static_test.jar is able to be deleted.  However the 
log4j.jar is locked until you shutdown Tomcat.  That is the problem we are 
trying to solve.

Jake

Created attachment 3567 [details]
See the README file.

I added a new tarball lock.tar.gz as an attachement. It contains a README file 
explaining its usage. 

My preliminary conclusion is that there is indeed an undesired interaction 
between log4j and Tomcat when the DOMConfigurator is used but not otherwise. 

The problem has been reduced to log4j/DOMConfigurator.

I don't think the indetified bug is directly linked to the bug reported by Jon. 
Jon are you using the DOMConfigurator?

I don't believe so. All of my log4j configuration is done with properties 
files.

Created attachment 3569 [details]
A new version of log4j that solves the log4j.jar locking problem.

I don't really want to further spam the discussion, but I'd just like it noted
that after removing the examples webapp from my Tomcat, I've not experienced any
problems any more. I DO use log4j, maybe a slightly older version, but I still
do use it. I have not experienced any CL errors any more.

Created attachment 3657 [details]
ant buld of a webapp reproducing the CL error....requires libraries in next attachment...

Ok, well, as it turns out, the second attachment seems too big for Bugzilla to 
handle.  It is moderately large, but not that big; about 2,386k.

Here is a link to it:
ftp://ftp.visi.com/users/hoju/pub/barracuda-libs.zip

Here is a link to the Ant build, for good measure:
ftp://ftp.visi.com/users/hoju/pub/barracuda-pages.sample_test.zip

Instructions on what everything is and how to use it is in a README.txt file in 
the root directory of the ant build in barracuda-pages.sample_test.zip.

Basically, extract barracuda-pages.sample_test.zip to any directory and extract 
barracuda-libs.zip to $CATALINA_HOME.  It will put the jar files where they 
need to go.  Again, more instructions are in the README.txt and they detail 
what libraries go where.

This setup is meant to be a completely reproduceable test showing the 
"...CL stopped" error.  It runs the first time the app is installed, but after 
removal and then another install, trying to pull up a dynamic page causes a 
java.lang.ClassNotFoundException and the "...CL stopped" error shows up in the 
stdout.log.

A similar configuration, but with a few of the libraries copied to WEB-INF/lib 
makes everything work flawlessly no matter how many install/remove cycles are 
done.

Step-by-step instructions are in README.txt of the Ant build.

let me know if you have any questions or difficulties.

Jake

Just wanted to note that I updated the libraries at the ftp addresses that I 
mentioned in my previous comment.

You can follow the instructions like normal, but if you want to see it working, 
you only have to copy barracuda-core.jar to WEB-INF/lib.  all other jars can 
stay where they are.  With barracuda-core.jar in WEB-INF/lib, the "...CL 
stopped" error doesn't ever happen, even after multiple remove/install cycles.

Oh, and Ceki asked me before whether the testcase was meant to show problems 
with log4j.  Just to clarify, this is *not* a log4j issue as far as I can tell.

Jake

I just wanted to add a little on this bug.  I have a large webapplication that 
runs in tomcat 4.0.3 and 4.1 fine, but if I go to run in on JBoss (with tomcat 
inside jboss) I get this error ALL the time.  It happeneds as soon as tomcat 
reloads my web app (when jboss figures out my web.xml was touched and tells 
tomcat to reload my app).

Just figured I would add this because it might be anotehr thing worth testing.
I will look into it a little more myself, but wanted to let people know what I 
have seen.

I can confirm that this still occurs with 4.1.12 and Scarab.

I also confirm that it occurs with Tomcat 4.0.3 and 4.1.12. I don't use log4j. 
My platform is a PC with Windows NT 4.
It seems to happen only when the webapp is restarted (because of a class or 
property file changed)

I'll try to track what has been done when it happens

I tried to remove the directory "examples" of Tomcat, as suggested in this 
thread : same behaviour

It seems like the lifecycle error occurs each time a .properties file is 
modified

Hope this helps

I can now say that I most commonly see this error when I double compile. 

In other words, I will re-compile and it will cause a classloader change. 
Then, if I re-compile once more before loading the servlet, I will see this 
error.

I also see this error happen more often when the files that change are not 
.class files. Say for example, I have a resource bundle .properties file in 
my classpath that requires a change...for example:

WebappClassLoader:   Resource '/WEB-INF/classes/
ScarabBundle.properties' was modified; Date is now: Thu Dec 19 
20:57:57 PST 2002 Was: Thu Dec 19 20:09:44 PST 2002
WebappClassLoader:   Resource '/WEB-INF/classes/org/tigris/scarab/
om/Issue.class' was modified; Date is now: Thu Dec 19 20:59:04 PST 
2002 Was: Thu Dec 19 20:10:42 PST 2002
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped
WebappClassLoader: Lifecycle error : CL stopped

Anyway, maybe that is more hints.

adding remy to this issue since he is the only one out there that can fix it.

Note sure if this is still being investigated or not but I noticed some pretty
bad behavior related to this bug under Tomcat-5.  The symptom is triggered the
same way for both Tomcat-4.1.x and tomcat-5 (with slight differences...read on).
 However, after getting the lifecycle error under Tomcat4, the context continues
to work.  After gettting the error under Tomcat5, the context quits working. 
The manager app reports it as not being started.

Here is how I reproduce the problem...

1.  Start Tomcat (4 or 5)
2.  Use the catalina-ant task to install/remove my app.  I go through the
following separate steps...
    ant catalina-install
    ant catalina-remove catalina-install
    ant catalina-remove catalina-install
    ant catalina-remove superclean catalina-install

The first 3 separate steps result in the context starting just fine.  However,
after the fourth step, the context fails to start.  Note that "superclean"
cleans up everything forcing a full clean rebuild.

I actually am not getting the "INFO: Lifecycle error : CL stopped" error unless
I do Log4j configuration.  However, I've somewhat ruled out Log4j as the culprit
because, although I don't get said error printed out when I don't explicitly
configure Log4j, I get the same behavior of the context not starting after the
4th step in either case.  Doing the configure must just trigger the code that
actually prints out that debugging where that same debugging code isn't
triggered otherwise.

Below is what is printed out in stdout.log in Tomcat5 when I do Log4j
configuration.  Notice that, somewhat curiously, the number of classloader
errors matches the number of times that the app was installed/removed (not
including the initial install).  Also note that I end up with an
"java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl" error in Tomcat5 where in
Tomcat4, I get   "javax.xml.parsers.FactoryConfigurationError: Provider
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found" ...


Starting service Tomcat-Standalone
Apache Tomcat/5.0
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Feb 23, 2003 9:00:25 PM org.apache.catalina.loader.WebappClassLoader
findResourceInternal
INFO: Lifecycle error : CL stopped
java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1199)
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1159)
	at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:93)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:172)
	at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:93)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:644)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:616)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:584)
	at org.apache.log4j.xml.XMLWatchdog.doOnChange(DOMConfigurator.java:815)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:80)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:99)
Feb 23, 2003 9:00:25 PM org.apache.catalina.loader.WebappClassLoader
findResourceInternal
INFO: Lifecycle error : CL stopped
java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1199)
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1159)
	at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:93)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:172)
	at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:93)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:644)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:616)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:584)
	at org.apache.log4j.xml.XMLWatchdog.doOnChange(DOMConfigurator.java:815)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:80)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:99)
Feb 23, 2003 9:00:26 PM org.apache.catalina.loader.WebappClassLoader
findResourceInternal
INFO: Lifecycle error : CL stopped
java.lang.IncompatibleClassChangeError:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1199)
	at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1159)
	at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:93)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:172)
	at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:93)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:644)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:616)
	at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:584)
	at org.apache.log4j.xml.XMLWatchdog.doOnChange(DOMConfigurator.java:815)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:80)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:99)
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.
Configuring GlobalOR from File: WEB-INF/object-repository.xml
Enhydra Java Application Server
Copyright 1997-2000 Lutris Technologies, Inc.
All rights reserved.


BTW, I am using Log4j-1.2.8 in WEB-INF/lib and have removed the Xerces that
comes with Barracuda in favor of Xerces-1.4.4 in common/endorsed because,
currently, XMLC-2.1 (xmlc.enhydra.org) requires that version of Xerces.  I have
noticed no issues with this setup other than when I do a clean build after
having installed the app in Tomcat.  Removing the app, then doing a clean build,
and then installing it again is when I get the bad behavior...and, of course,
worse behavior in Tomcat5.

So, does this help?  Any new ideas on why this might be happening?  Any fixes in
the works?

Jake

ClassLoader stopped on java.lang.DoubleI'm using OS X Server (latest rev), jav 1.3.1 and Tomcat 4.1.18. We have a thread that runs once/day. It will run for maybe a day or two successfully, but then the ClassLoader gives the following:WebappClassLoader: Lifecycle error : CL stoppedjava.lang.NoClassDefFoundError: java/lang/Double        at com.attask.AtProjects.setProj_invoiceAmount(AtProjects.java:1049)        at com.attask.AtProjects.populateFromResultSet(AtProjects.java:1210)        at com.attask.AtListBean.createListItem(AtListBean.java:238)        at com.attask.AtListBean.setList(AtListBean.java:127)        at com.attask.AtListBean.setList(AtListBean.java:67)        at com.attask.AtThreadRecalcTimelines.run(AtThreadRecalcTimelines.java:55)        at com.attask.AtThread.run(AtThread.java:101)This wreaks havoc on the data. Interesting that java.lang.NoClassDefFoundError is in the same package as java.lang.Double, but Double could not be found.-Scott

Just to chime in on this ticket, we ran into this exception today as well.  
Although it wasn't with log4j or anything else.  It occured during a load test 
on the server.  It doesn't appear we ran out of Heap or Stack.  Our -Xmx/ms 
settings were our norm for a buffer during the test.

The only simularity is that we do run all of the JAR's out of the application 
WEB-INF/lib path.  There was no new jars/classes installed during this test.  
We were not using JSP's during this load test, rather just using Tomcat as a 
servlet engine for now.  Although I think we may still have reloadable="true" 
on it.

We had this exception occur while trying to perform some encryption within our 
application, using the SunJCE provider.

WebappClassLoader: Lifecycle error : CL stopped
java.security.NoSuchAlgorithmException: Algorithm DES not available
        at javax.crypto.SunJCE_b.a(DashoA6275)
        at javax.crypto.SecretKeyFactory.getInstance(DashoA6275)
        at com.metro1.encryption.CryptFactory.decryptKey(CryptFactory.java:155)
        at com.metro1.encryption.CryptFactory.<init>(CryptFactory.java:124)
        ... rest of the stack.


Is there anything that I can set in my server to give more information if we 
can reproduce this more regularly, other than during the load test?

I'm going to have our QA department turn of reloadable first.  If it 
continues, I'll have them move the encryption JAR's into common/lib and see if 
that resolves anything.

Your comments more or less prove that this bug is invalid (your classes are
referenced from a shared static object, and thus remain loaded after the webapp
reload). I will not attempt to close, resolve, or look further into this report,
which is an application problem, not a Tomcat bug.

Remy,

How does the "applications WEB-INF/lib" consitute a "(your classes are
referenced from a shared static object, and thus remain loaded after the webapp
reload)"?????

Feel free to keep your head buried in the sand, however, this is a Tomcat
problem as I have never seen this problem before I used Tomcat or with any other
containers.

The way I currently duplicate it (using Scarab) is that it happens *every single
time* after exactly 3 classloader reloads. I suspect it is that there is a
memory leak with reloading which fills the memory up after 3 reloads and the JVM
blows up.

Agreed, this is a tomcat bug.
We are getting this error using 4.1.18 and jdk 1.4.1 the classloader is failing on java.lang.Double.

I'm seeing this error with Tomcat 4.1.24 (LE jdk14), but under what seems a
somewhat different circumstance from what I'm seeing described here.

I have a servlet that performs a rather lengthy action (from doPost())--it takes
several minutes to complete. If I update the class while the doPost() is still
in the midst of processing, I see my servlet's destroy method called, then see
its init method called as it is reloaded. However the doPost method appears to
continue to run, judging by the output. It completes its task normally and
successfully, except that this "WebappClassLoader: Lifecycle error : CL stopped"
error keeps occuring.

I thought that destroy() was not supposed to be called until the service()
method has exited, which definitely has not happened in my case.

The reason I ran into this is I was testing the ability to take this servlet out
of service without risking interrupting current tasks...

Well, there's a timeout (basically, it will wait for 50ms 10 times, and then
give up and proceed with the stop; so in your case, I'd say it's not enough). I
think the number of retry attempts could be made to be configurable (hacking the
code is very simple - look in o.a.c.core.StandardWrapper.unload()). If you try
to run code on the discarded CL after the webapp has been reloaded, you'll get
the log messages, it's normal.

I noticed the mention of a timeout in the serlvet docs. I take it this timeout 
isn't explicitly configurable (except by hacking) in Tomcat?

This behavior seems kind of unclean... If it's going to time out shouldn't the 
engine kill the thread(s) rather than letting it(them) continue to run and 
throw errors? (Incidentally I realized my post() didn't totally run to 
completion...in the middle of one of the last steps it throws a 
java.lang.NoClassDefFoundError: java/lang/reflect/InvocationTargetException
attempting to instantiate a object via pluggable factory-style method.)

I could write my own synchronization between destroy() and service() but it 
seems like the kind of functionality that the framework should provide. 

In 4.1.27 the bug still bites.
If someone wants to replicate the bug, try using jcoverage.
When the library was in WEB-INF/lib the bug occured.
When I moved the jar to TOMCAT/commons/lib everything was ok.

Anyway, thanks to Martin van Dijken and Remy Maucherat for insightful comments,
that helped me to find workaround.

Another way to reporduce it is to use swarmcache.
https://sourceforge.net/projects/swarmcache/

On stop of a context it can occure that Tomcat
throws this exception.

java.lang.NoClassDefFoundError: org/javagroups/stack/NakReceiverWindow$Entry
at org.javagroups.stack.NakReceiverWindow.add(NakReceiverWindow.java:199)
at org.javagroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:357)
at org.javagroups.protocols.pbcast.NAKACK.up(NAKACK.java:223)
at org.javagroups.stack.UpHandler.run(Protocol.java:50)

*** Bug 24847 has been marked as a duplicate of this bug. ***

We have been seeing this same problem with 4.1.29. A jar file was still locked
after doing an undeploy. As a work-around, we found that if this jar file were
expanded into the WEB-INF/classes directory, the the undeploy was successful. 

It seems that this points to a problem with the ClassLoader closing the jar
file. Perhaps an exception during undeploy is causing the closing of the jar
file to be bypassed.

Hi Dan,

If you are seeing a jar that is locked, it means that some thread is still has a
handle to some resource in that jar (or the thread is created from a class in
the jar).  The fact that this isn't happening when you put the jar in
WEB-INF/classes should be obvious.  The jar was never loaded from
WEB-INF/classes.  Plain classes are loaded there, not jars.

You need to make sure what is still holding a handle to a resource in that jar
and make sure it gets shut down at webapp shutdown.  Remy is correct in saying
that most of the issues here are due to user error, not a problem in Tomcat. 
That is not to say that this is necessarily true in all cases, but certainly most.

Jake

Hi Jacob,
I think you misunderstood what I was saying. We expanded the jar file into
WEB-INF/classes. That is, "jar -xf my.jar". We then removed the jar file from
WEB-INF/lib. Therefore, any resources that were previously being loaded from the
jar file would now be getting loaded from the classes directory. However,
whereas the jar file would be locked after undeploy in the previous case, we now
see undeploy happen successfully when the jar file is expanded. If it were any
of our threads holding a resource open, it still would have it open in the
classes directory. The only difference we should have seen were that individual
files would have been locked.

We arrived at this point after having eliminated several cases where our threads
were holding resources open. We've now verified this by getting a thread dump on
threads still around after the undeploy. Also, we have no problem in running the
exact same scenario in WebLogic.

We're still investigating the problem and should have more info from a profiling
tool pretty soon.

I have reviewed the multiple issues raised in this bug report and believe they
are all either resolved or invalid. Siince the original report is actually a
duplicate, that is how this bug will be resolved.

A summary follows:

Issue 1
WebappClassLoader: Lifecycle error : CL stopped
As Remy commented (#13) "...Basically, if you can find some component in
Catalina holding a pointer to either one of the old object/classes/classloader,
then the bug is valid. Otherwise, it's not."
Although there was no evidence of this in this bug report see Bug 20758 for a
detailed examination of where this was happening.

Issue 2
log4j jar locking. As per Ceki's comment (#28), this has been fixed in log4j.

Other issues
A variety of libraries and applciations using shared static objects.

*** This bug has been marked as a duplicate of 20758 ***