| Summary: | Infinite redeploy loop with undeployOldVersions="true" | ||
|---|---|---|---|
| Product: | Tomcat 8 | Reporter: | Lauri Lehtinen <lauri.h.lehtinen> |
| Component: | Catalina | Assignee: | Tomcat Developers Mailing List <dev> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | lauri.h.lehtinen |
| Priority: | P2 | ||
| Version: | 8.0.x-trunk | ||
| Target Milestone: | ---- | ||
| Hardware: | PC | ||
| OS: | Mac OS X 10.1 | ||
| Attachments: |
JConsole screen while observing this behavior
catalina.out with HostConfig logging set to finest |
||
Created attachment 33472 [details]
catalina.out with HostConfig logging set to finest
Easy fix for this particular problem would be to remove the getCanonicalFile call in HostConfig (https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/startup/HostConfig.java#L1416), but I'm afraid it's there for some good reason. I think the root issue is that your application leaks a ClassLoader for some reason (and you should definitely fix that), but Tomcat should also not be endlessly re-deploying itself. There is a less pathological use-case here: downgrading an application after pushing-out a broken update. For example, version A is deployed and working well, then version B is deployed. Version B is determined to be flawed and so version A will be re-deployed as version C. Symlinks are used to link version C to version A for clarity for an admin. Assuming versions A and C are deployed simultaneously (e.g. version A was not decommissioned before version C was deployed) and you have the same situation as described in comment #0. +1 to Chris's comment regarding the memory leak. 99% of those turn out to be bugs in applications or 3rd party libraries. The redeploy loop is definitely a Tomcat bug. getCanonicalPath() is used handle docBases such as "../../someapp.war" which will be resolved relative to the appBase but should not be deleted because they are outside the appBase. In this case the context would probably have been configured via a someapp.xml file (which does need to be removed). There are likely to be similar issues with any code that triggers an undeploy if the WAR, DIR or XML files are symlinked into the appBase or configBase. The fix will be to remove those symlinks. The checks are going to have to be more sophisticated than the current getCanonicalPath(). I plan to look at this today. You're right, the memory leak was caused by separate issue. Thanks Fixed for 9.0.0.M2 onwards, 8.0.31 onwards and 7.0.68 onwards. |
Created attachment 33471 [details] JConsole screen while observing this behavior ## Reproduction steps ## To reproduce this issue: 1. Add undeployOldVersions="true" to the otherwise default Host configuration in server.xml 2. Do a parallel deployment of two versions of the same war by creating a symlink to the actual war file(s): $ ln -s /path/to/app.war /path/to/webapps/app#001.war $ ln -s /path/to/app.war /path/to/webapps/app#002.war Shortly after startup, Tomcat undeploys app##001 (the "old version"). However, the war file is not deleted, and ends up getting redeployed. This happens over and over again, until at some point the number of loaded classes grows too big and OutOfMemoryError happens. ## Root cause ## What's preventing the symlinked war from getting deleted is this line in org.apache.catalina.startup.HostConfig#deleteRedeployResources: current = current.getCanonicalFile(); The canonical file is war file the symlink is pointing to, and the isDeletableResource method determines that it should not be deleted. ## Context ## This issue has been plaguing me on a CentOS/6 server running 7.0.55, but appears to affect 8.x and 9.x as well. I reproduced it locally on OSX. My Struts2/Spring/Hibernate application died within ~10 minutes with -Xmx256m. Screenshot from jconsole attached, undeploy/redeploy cycle can be seen clearly in the CPU usage and number of classes loaded.