Bug 57251

Summary: WAR deployment unbelievably slow when unpackWARs=false
Product: Tomcat 8 Reporter: Axel Fontaine <axel>
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal CC: flozano, guillermo.grandes, williamleung2006
Priority: P2    
Version: 8.0.17   
Target Milestone: ----   
Hardware: PC   
OS: All   

Description Axel Fontaine 2014-11-24 12:47:40 UTC
I have a small webapp I deploy to test Flyway (flyway-sample-webapp)

Tomcat 8.0.15, extracted from zip, unmodified on Java 1.8.0_20-b26 x64.

If server.xml contains unpackWARs="true" all is well and it deploys instantly.

If I set the value to false, it takes several minutes and this is the output I get:

24-Nov-2014 13:35:02.200 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
24-Nov-2014 13:35:02.238 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
24-Nov-2014 13:35:02.240 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]
24-Nov-2014 13:35:02.242 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
24-Nov-2014 13:35:02.242 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 391 ms
24-Nov-2014 13:35:02.258 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
24-Nov-2014 13:35:02.259 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.15
24-Nov-2014 13:35:02.274 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive C:\Programs\apache-tomcat-8.0.15\webapps\ROOT.war
24-Nov-2014 13:37:44.119 SEVERE [localhost-startStop-1] org.apache.catalina.startup.ContextConfig.processAnnotationsJar Unable to process Jar entry [com/google/cloud/sql/jdbc/internal/ConnectionProperty.class] from Jar [jar:file:/C:/Programs/apache-tomcat-8.0.15/webapps/ROOT.war!/WEB-INF/lib/appengine-api-1.0-sdk-1.9.10.jar] for annotations
 java.io.EOFException
        at org.apache.tomcat.util.bcel.classfile.FastDataInputStream.readUnsignedShort(FastDataInputStream.java:120)
        at org.apache.tomcat.util.bcel.classfile.ClassParser.readAttributes(ClassParser.java:110)
        at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:94)
        at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:1994)
        at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1944)
        at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1919)
        at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1880)
        at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1149)
        at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:771)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5095)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:917)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1701)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

24-Nov-2014 13:37:58.057 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
24-Nov-2014 13:37:58.220 INFO [localhost-startStop-1] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [144] milliseconds.
24-Nov-2014 13:37:58.232 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive C:\Programs\apache-tomcat-8.0.15\webapps\ROOT.war has finished in 175,958 ms
24-Nov-2014 13:37:58.235 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
24-Nov-2014 13:37:58.240 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
24-Nov-2014 13:37:58.241 INFO [main] org.apache.catalina.startup.Catalina.startServer startup in 175998 ms
24-Nov-2014 13:38:08.236 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.HostConfig.deployWARs The directory [C:\Programs\apache-tomcat-8.0.15\webapps\ROOT] will be ignored because the WAR [C:\Programs\apache-tomcat-8.0.15\webapps\ROOT.war] takes priority and unpackWARs is false

Switching unpackWARs to true, makes the deployment instant and error-free again.

Cheers
Axel
Comment 1 Mark Thomas 2014-11-24 12:59:20 UTC

*** This bug has been marked as a duplicate of bug 57173 ***
Comment 2 Francisco A. Lozano 2015-01-26 16:31:17 UTC
I can see the "EOF" error is duplicate of 57173, but - is the deployment speed also? To me it looks like there is a speed-related regression between 8 and 7.

I have an app which is webapp-3.0, with metadata-complete=true and an empty absolute-ordering element. The WAR is around 80MB.

I am getting:
 - Tomcat 7.0.57, unpackWARs=false: deploys in 30-50 seconds.
 - Tomcat 8.0.17, unpackWARs=false: deploys in 350 seconds.
 - Tomcat 8.0.17, unpackWARs=true: deploys in 30-50 seconds.

Can you please consider reopening?
Comment 3 Francisco A. Lozano 2015-01-26 16:37:31 UTC
- JDK 1.8.0_20 and 31
- Linux and OS X
- Slow on Tomcat 8.0.15, 8.0.17
- OK on Tomcat 7.0.57
Comment 4 Francisco A. Lozano 2015-01-26 17:00:47 UTC
I've collected several stack-traces during slow startup times and, when the startup gets "stalled", the stacks look like this:

"localhost-startStop-1" #15 daemon prio=5 os_prio=0 tid=0x0000000001954000 nid=0x52d1 runnable [0x00007fb50c625000]
   java.lang.Thread.State: RUNNABLE
	at java.util.zip.Inflater.inflateBytes(Native Method)
	at java.util.zip.Inflater.inflate(Inflater.java:259)
	- locked <0x00000000dab5ab88> (a java.util.zip.ZStreamRef)
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
	at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
	at java.util.jar.JarInputStream.read(JarInputStream.java:207)
	at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:140)
	at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:118)
	at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:142)
	at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:179)
	at org.apache.catalina.webresources.JarWarResource.getJarInputStreamWrapper(JarWarResource.java:59)
	at org.apache.catalina.webresources.AbstractArchiveResource.getContent(AbstractArchiveResource.java:175)
	at org.apache.catalina.loader.WebappClassLoaderBase.findResourceInternal(WebappClassLoaderBase.java:2579)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2405)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:854)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1274)
	- locked <0x0000000081c173f8> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2472)
	- locked <0x0000000081c173f8> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:854)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1274)
	- locked <0x0000000081c173f8> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)


It happens from different places:
 - netty loading some classes in io.netty.util.internal.PlatformDependent.getSystemClassLoader
 - spring/aspectj startup in org.aspectj.weaver.tools.PointcutParser.resolvePointcutExpression

and I guess many others
Comment 5 Mark Thomas 2015-01-26 19:42:43 UTC
Stalled suggests that the strtup stops which doesn't appear to be the case.

A web application that demonstrated the issue would be a big help.
Comment 6 Francisco A. Lozano 2015-01-27 06:14:32 UTC
Apologies for my poor usage of English with regard to the word "stalled" - you're right, it doesn't stop at all, just takes a lot of time.

I've prepared a WAR file which shows the issue.

https://github.com/flozano/greenhouse/blob/dev/show_tomcat8_problem/greenhouse-1.0.0.BUILD-SNAPSHOT.war

- Tomcat 8 deployment speed with unpackWARs="false": 24-26 seconds.
- Tomcat 8 deployment speed with unpackWARs="true": 5-6 seconds.
- Tomcat 7 deployment speed with unpackWARS="false": 4-5 seconds.
- Tomcat 7 deployment speed with unpackWARS="false": 4-5 seconds.

Hope it helps.

PS: 
The WAR is a small modification of a pre-existing Spring sample project, don't expect it to work properly, but the startup part works well.
Comment 7 Francisco A. Lozano 2015-01-27 06:15:44 UTC
Correction:

- Tomcat 8 deployment speed with unpackWARs="false": 24-26 seconds.
- Tomcat 8 deployment speed with unpackWARs="true": 5-6 seconds.
- Tomcat 7 deployment speed with unpackWARS="false": 4-5 seconds.
- Tomcat 7 deployment speed with unpackWARS="true": 4-5 seconds.
Comment 8 Mark Thomas 2015-01-27 10:50:50 UTC
Thanks for the test case. I'm currently working on a potentially related issue (BZ 57472) so I'll look at this once I have fixed that. I suspect similar root causes - performance issues in the new (in 8.0.x) resources implementation when accessing resources in archives.
Comment 9 Mark Thomas 2015-01-27 16:03:21 UTC
The fix for bug 57472 might shave a few seconds of the deployment time but it doesn't appear to make a significant difference.

The fundamental problem when running from a packed WAR is that to access any resource in a JAR, Tomcat has to do the following:
- open the WAR
- get the entry for the JAR
- get the InputStream for the JAR entry
- Create a JarInputStream
- Read the JarInputStream until it finds the entry it wants

This is always going to be slow.

The reason that it is fast in Tomcat 7 and earlier took some digging. In unpackWARs is false in Tomcat 7, it unpacks the JARs anyway into the work directory and uses them from there. Performance is therefore comparable with unpackWARs="true".

I haven't made my mind up what is the right thing to do here.

On one hand, unpackWARS="false" really should mean exactly that. Not "don't unpack the WAR into the appBase but so unpack the JARs in to the work dir". You might as well just use unpackWARs="true".

On the other hand, the performance is clearly worse. It was a factor of 3 on my machine but clearly it varies from machine to machine. It looks to be in the range 3 to 10 times slower - which is not good.

To throw something else into the mix, static resources will be cached in memory in both cases for improved performance.

As I ponder what to do about this I do have one question. Why do you want to run with unpackWARs="false"? What is the use case?
Comment 10 Francisco A. Lozano 2015-01-27 21:29:21 UTC
The deploy/redeploy tooling we have had around since tomcat 6 works that way, and it's easier to manage just a WAR file in webapps than a WAR file and a exploded directory. I seem to recall that keeping them in sync was problematic, as tomcat may be down  when the WAR is reinstalled.

I think there is value in existing tomcat6/7 behaviour - the "interface" that users have to deal with is just simpler, and it makes it more difficult to shoot yourself in the foot without paying performance penalty.

In T8, the current options are either large performance penalty paid or having to deal with a slightly more complex deployment.

I understand the 'unpackWARS="false" really should mean exactly that' statement, but existing behaviour in T6/7 is valuable and is relied upon - so maybe the new 'really don't unpack anything' behaviour could be a separate feature, but existing one in my opinion should still be available.
Comment 11 Mark Thomas 2015-01-27 22:17:23 UTC
(In reply to Francisco A. Lozano from comment #10)
> The deploy/redeploy tooling we have had around since tomcat 6 works that
> way, and it's easier to manage just a WAR file in webapps than a WAR file
> and a exploded directory. I seem to recall that keeping them in sync was
> problematic, as tomcat may be down  when the WAR is reinstalled.

Yes, that can be a problem. Tomcat doesn't detect that the WAR is newer than the exploded directory.

> I think there is value in existing tomcat6/7 behaviour - the "interface"
> that users have to deal with is just simpler, and it makes it more difficult
> to shoot yourself in the foot without paying performance penalty.

Keep in mind you are still paying the performance penalty for at least the first load all your static resources (although they are normally then cached in memory which will help until the cache expires).

> In T8, the current options are either large performance penalty paid or
> having to deal with a slightly more complex deployment.
> 
> I understand the 'unpackWARS="false" really should mean exactly that'
> statement, but existing behaviour in T6/7 is valuable and is relied upon -
> so maybe the new 'really don't unpack anything' behaviour could be a
> separate feature, but existing one in my opinion should still be available.

That would mean adding back in the "unpack the JARs anyway if unpackWARs is false" feature which is do-able but would add complexity to an area where I have been trying to reduce it.

Given the use case, I'm wondering if making Tomcat's auto-deployment code smart enough to detect that the WAR has been updated even if Tomcat is not running might be a better solution. Something along the lines of a file in the work directory configured with the same last modified time as the WAR. A simple comparison of the last modified time of the two files tells Tomcat if a redploy is required on start.

I did think about caching the JARs in memory (probably only during web application start) but for some applications that could mean significant increases in the memory footprint just to start. I suspect this might cause as many problems as it solves.

Overall, I'm leaning towards a position of finding a better solution to the use cases that mean folks opt to run with unpackWARs set to false. I'd be interested in hearing if there are any other use cases.
Comment 12 Francisco A. Lozano 2015-01-28 01:38:10 UTC
One question about your reasoning - what's the point of having unpackWARS="false" option if it's so unusably slow in T8?
Comment 13 Mark Thomas 2015-01-28 07:54:42 UTC
Indeed. Hence my question about use cases.

I would note that there are many folks that are less concerned about slow deployment as long as performance once deployed is acceptable.
Comment 14 Wolfgang Imig 2015-01-28 14:51:41 UTC
"... I'd be interested in hearing if there are any other use cases..."

Here is my use case for unpackWARs="false":
Our software product installs for each repository the same set of web applications. If the customer wants to update an application, she usually wants to have it updated for all repositories. To make the update process as simple as possible, the WARs are not stored at Tomcat/webapps but in a directory outside of tomcat. Each WAR is placed there only once. Context container files in Tomcat/conf/Catalina/localhost represent the required web applications and link in their "docBase" attribute to the WAR.

With unpackWARs="false", the startup time in Tomcat 8 is unacceptably slow, usually several minutes. Setting unpackWARs="true" results in an acceptable performance, but it has the downside that Tomcat does not recognize if the WAR was updated and always starts the exploded WAR. 

In our case, WARs are only updated when Tomcat is stopped. I think your suggestion about comparing the last modified times is a reasonable solution (timestamp of WAR compared to timestamp of a file that is created during unpacking).
Comment 15 Christopher Schultz 2015-01-28 14:55:08 UTC
(In reply to Francisco A. Lozano from comment #12)
> One question about your reasoning - what's the point of having
> unpackWARS="false" option if it's so unusably slow in T8?

One valid use case: read-only filesystem (from Tomcat's perspective). unpackWars="false" allows you to run with the host's appBase directory non-writable by Tomcat.

Having the webapps directory writable by Tomcat is a security concern, especially if Tomcat were exploited in some way... the attacker could deploy an application by dropping a WAR file into that directory. Obviously, there are other ways to attack Tomcat, but this is a legitimate layer of protection.

The old behavior of unpackWars="false" expanding WAR files into the work/ directory was acceptable from a security perspective, since nothing in the work/ directory could be auto-deployed. Again, there are other security concerns here with the work directory outside the scope of the original question.
Comment 16 Mark Thomas 2015-01-28 15:13:07 UTC
(In reply to Christopher Schultz from comment #15)
> (In reply to Francisco A. Lozano from comment #12)
> > One question about your reasoning - what's the point of having
> > unpackWARS="false" option if it's so unusably slow in T8?
> 
> One valid use case: read-only filesystem (from Tomcat's perspective).
> unpackWars="false" allows you to run with the host's appBase directory
> non-writable by Tomcat.

There is nothing stopping users copying an exploded directory into the appBase in the same way a WAR is copied. The ASF's JIRA instance runs this way for exactly the security concerns you cite.

I do not see any security benefits that are unique to unpackWARs="false"
Comment 17 Christopher Schultz 2015-01-28 15:45:43 UTC
(In reply to Mark Thomas from comment #16)
> There is nothing stopping users copying an exploded directory into the
> appBase in the same way a WAR is copied. The ASF's JIRA instance runs this
> way for exactly the security concerns you cite.

Yes, but those WARs are being copied locally and can work by using a user other than Tomcat's uid.

> I do not see any security benefits that are unique to unpackWARs="false"

If Tomcat itself can be remotely exploited to drop a WAR file into webapps/ then it might be auto-deployed without local access (which is what you describe above).
Comment 18 Remy Maucherat 2015-01-29 09:38:49 UTC
This "feature" should not have been added in the first place, that's for sure. The [huge] amount of time that was spent on this should have been spent on a better deployer back then.
Comment 19 Mark Thomas 2015-01-29 09:47:14 UTC
(In reply to Christopher Schultz from comment #17)
> (In reply to Mark Thomas from comment #16)
> > There is nothing stopping users copying an exploded directory into the
> > appBase in the same way a WAR is copied. The ASF's JIRA instance runs this
> > way for exactly the security concerns you cite.
> 
> Yes, but those WARs are being copied locally and can work by using a user
> other than Tomcat's uid.

Nothing stops this other user from copying an exploded directory to the appBase rather than an unexploded WAR.

> > I do not see any security benefits that are unique to unpackWARs="false"
> 
> If Tomcat itself can be remotely exploited to drop a WAR file into webapps/
> then it might be auto-deployed without local access (which is what you
> describe above).

Either the appBase is writeable (in which case there is a small security risk) or it isn't. A writeable (by the Tomcat user) appBase is independent of whether you deploy applications as WARs or exploded directories.
Comment 20 Francisco A. Lozano 2015-01-29 12:52:38 UTC
(In reply to Mark Thomas from comment #19)
> (In reply to Christopher Schultz from comment #17)
> > (In reply to Mark Thomas from comment #16)
> > > There is nothing stopping users copying an exploded directory into the
> > > appBase in the same way a WAR is copied. The ASF's JIRA instance runs this
> > > way for exactly the security concerns you cite.
> > 
> > Yes, but those WARs are being copied locally and can work by using a user
> > other than Tomcat's uid.
> 
> Nothing stops this other user from copying an exploded directory to the
> appBase rather than an unexploded WAR.

But this other user can be more tightly controlled, because it doesn't execute anything. The user that writes doesn't execute, and the user that executes doesn't write. It's a pretty common security pattern
http://en.wikipedia.org/wiki/W%5EX

> 
> > > I do not see any security benefits that are unique to unpackWARs="false"
> > 
> > If Tomcat itself can be remotely exploited to drop a WAR file into webapps/
> > then it might be auto-deployed without local access (which is what you
> > describe above).
> 
> Either the appBase is writeable (in which case there is a small security
> risk) or it isn't. A writeable (by the Tomcat user) appBase is independent
> of whether you deploy applications as WARs or exploded directories.

But when you use WARs you hit this issue in Tomcat 8 and not in Tomcat 7/6.
Comment 21 Francisco A. Lozano 2015-01-29 16:28:49 UTC
From documentation (http://tomcat.apache.org/tomcat-8.0-doc/config/context.html):

"Note that WAR files located outside of a Host's appBase are never unpacked."

From that comment, not fixing this would mean that there would be no way to deploy WARs out of appBase at an acceptable speed?
Comment 22 Mark Thomas 2015-01-29 17:06:57 UTC
(In reply to Francisco A. Lozano from comment #20)
> (In reply to Mark Thomas from comment #19)
> > (In reply to Christopher Schultz from comment #17)
> > > (In reply to Mark Thomas from comment #16)
> > > > There is nothing stopping users copying an exploded directory into the
> > > > appBase in the same way a WAR is copied. The ASF's JIRA instance runs this
> > > > way for exactly the security concerns you cite.
> > > 
> > > Yes, but those WARs are being copied locally and can work by using a user
> > > other than Tomcat's uid.
> > 
> > Nothing stops this other user from copying an exploded directory to the
> > appBase rather than an unexploded WAR.
> 
> But this other user can be more tightly controlled, because it doesn't
> execute anything. The user that writes doesn't execute, and the user that
> executes doesn't write. It's a pretty common security pattern
> http://en.wikipedia.org/wiki/W%5EX

Which is the point I was making. Copying in a WAR that Tomcat doesn't expand or copying in an exploded directory that Tomcat doesn't need to expand, the security benefits are exactly the same (assuming permissions are set correctly).

> > > > I do not see any security benefits that are unique to unpackWARs="false"
> > > 
> > > If Tomcat itself can be remotely exploited to drop a WAR file into webapps/
> > > then it might be auto-deployed without local access (which is what you
> > > describe above).
> > 
> > Either the appBase is writeable (in which case there is a small security
> > risk) or it isn't. A writeable (by the Tomcat user) appBase is independent
> > of whether you deploy applications as WARs or exploded directories.
> 
> But when you use WARs you hit this issue in Tomcat 8 and not in Tomcat 7/6.

Which is why we are dicussing whether or not there is any need to run directly from a WAR. The best argument made so far is that it is easier to move around a WAR than an exploded directory but - given expanding a WAR is a one-line script on any platform where you can run Tomcat (you can use jar to unpack the WAR) - that use case doesn't strike me as a particularly strong one.

(In reply to Francisco A. Lozano from comment #21)
> From documentation
> (http://tomcat.apache.org/tomcat-8.0-doc/config/context.html):
> 
> "Note that WAR files located outside of a Host's appBase are never unpacked."
> 
> From that comment, not fixing this would mean that there would be no way to
> deploy WARs out of appBase at an acceptable speed?

That comment is out of date. Tomcat 8 (and possibly 7 - I'd need to check) will unpack it now the various edge cases in the deployer have been cleaned up.
Comment 23 Remy Maucherat 2015-01-29 18:17:23 UTC
+1 for dropping that unpackWars feature, if the new resources won't support it that well. It is not very useful, adds complexity, etc, and maybe it allows improving the deployer further.
Comment 24 William Leung 2015-02-03 10:14:56 UTC
Hi Mark Thomas 

We have two virtual hosts hosting different versions of same names
And they shares some common apps drop in 'common_webapps'

<Host name="HOST1" unpackWARs="false" appBase="common_webapps">
  <Context path=""     docBase="HOST1_web/ROOT.war"/>
  <Context path="app1" docBase="HOST1_web/app1.war"/>
  <Context path="app2" docBase="HOST1_web/app2.war"/>
</Host>

<Host name="HOST2" unpackWARs="false" appBase="common_webapps">
  <Context path=""     docBase="HOST2_web/ROOT.war"/>
  <Context path="app1" docBase="HOST2_web/app1.war"/>
  <Context path="app2" docBase="HOST2_web/app2.war"/>
</Host>


If I change the server.xml unpackWARs to true
Host2 deployment will keep overwriting HOST1's applications

Can you tell me how to avoid this?
Comment 25 Francisco A. Lozano 2015-02-16 17:34:04 UTC
Any decision taken about this issue?
Comment 26 Mark Thomas 2015-03-03 12:00:07 UTC
Short term, I'm working on adding a feature that detects if a WAR is updated while Tomcat is shut down. Currently, I view that as sufficient to resolve this bug.

Medium to long term I'm thinking about removing the unpackWARs feature and always unpacking into the appBase. Any move in that direction will be preceeded by a discussion on one of the mailing lists - probably the users list since that is where we'd get best feedback on the impact of such a change.
Comment 27 Francisco A. Lozano 2015-03-03 12:14:05 UTC
Does that address Comment 24 (https://bz.apache.org/bugzilla/show_bug.cgi?id=57251#c24)?
Comment 28 Mark Thomas 2015-03-03 12:40:37 UTC
(In reply to Francisco A. Lozano from comment #27)
> Does that address Comment 24
> (https://bz.apache.org/bugzilla/show_bug.cgi?id=57251#c24)?

Short version:
- use unique appBase values per Host
- place the WARs outside all appBases
- use context.xml files to add apps to a Host
- leave unpackWARs as the default of true

The new feature will ensure each Host sees any updates to the WARs even if they are made while Tomcat isn't running.

If you need any more advice, use the users mailing list.
Comment 29 Konstantin Kolinko 2015-03-03 12:43:47 UTC
(In reply to Mark Thomas from comment #26)
> Medium to long term I'm thinking about removing the unpackWARs feature and
> always unpacking into the appBase.

I am -1 on removing this feature, as

1) It is a showcase that ServletContext.getRealPath() may return null.

2) Even if we do not fix the slowness, it can be used if there are no jars in the war.

E.g. if all jars are elsewhere outside of the webapp (e.g. mapped from external locations via context.xml).
Comment 30 Mark Thomas 2015-03-03 14:28:20 UTC
(In reply to Konstantin Kolinko from comment #29)
> (In reply to Mark Thomas from comment #26)
> > Medium to long term I'm thinking about removing the unpackWARs feature and
> > always unpacking into the appBase.
> 
> I am -1 on removing this feature, as

Fair enough. I'm not that set on removing it at the moment.

Given the performance issues, my main concern is that for each use case for using unpackWARs=false there is an alternative confiuration approach that doesn't add too much burden to the end users.
Comment 31 Mark Thomas 2015-03-03 20:52:18 UTC
I've added a section to the migration guide for 7.0.x -> 8.0.x covering deployment and the performance drop if unpackWARs="false" is used.

I've added the 'detect WAR has been modified while Tomcat isn't running' feature to trunk and 8.0.x (will be in 8.0.21 onwards).

As previosuly stated, I believe we have done enough at this point to resolve this issue.
Comment 32 Kim 2015-03-04 12:54:58 UTC
Will this change also make sure that if the context XML file is later removed from conf/Catalina/localhost then the app will not be started?

One of my reasons for specifying unpackWARs="false" is to have very tight control of which apps are started when Tomcat starts.
Comment 33 William Leung 2015-08-06 04:51:25 UTC
I had make a workaround for this issue

https://github.com/lwr/tomcat8-custom-loader

because the some fields are not accessible from tomcat source
1. org.apache.catalina.webresources.StandardRoot#classResources for writing
2. org.apache.catalina.webresources.JarWarResourceSet#archivePath for reading

a hack (using reflection) must be used.
If these fields are renamed in future, the workaround has to fix.

I really hope that tomcat should keep this feature (unpackWARs=false)
it is useful and meaningful

And the problem here is very easy to solve, just unpack the jars to working directory (as TC6 or TC7 does), and leave webapps alone!
Comment 34 Guillermo Grandes 2017-04-30 03:03:20 UTC
(In reply to Mark Thomas from comment #9)
> The fix for bug 57472 might shave a few seconds of the deployment time but
> it doesn't appear to make a significant difference.
> 
> [...cut...]
> 
> The reason that it is fast in Tomcat 7 and earlier took some digging. In
> unpackWARs is false in Tomcat 7, it unpacks the JARs anyway into the work
> directory and uses them from there. Performance is therefore comparable with
> unpackWARs="true".

Question: If unpack of war in work directory works well, why change?

> [...cut...]
> 
> As I ponder what to do about this I do have one question. Why do you want to
> run with unpackWARs="false"? What is the use case?

More use cases for unpackWARs=false, our is:

Multiple Tomcats with appBase pointing to a NFS directory (mounted read-only). All WARs are versioned (acme##712.war), come from a Nexus server and SHA1/signature is checked before stored in NFS.

Unpacked directories (unpackWARs=true) broken "inmutable" infraestruture (humans are prone to "touch files" manually), in these cases "md5sum xxx.war -tomcat-" vs "md5sum xxx.war -nexus-" help to find the problem.

+1 to maintain feature unpackWARs=false.