Bug 63078 - autodeploy in Tomcat 9.0.14 is not working anymore
Summary: autodeploy in Tomcat 9.0.14 is not working anymore
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 9.0.14
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-01-15 11:44 UTC by Peter
Modified: 2019-01-22 15:57 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Peter 2019-01-15 11:44:18 UTC
When copying a new war file into tomcats webapps directory, Tomcat 9.0.14 does not autodeploy it anymore. 
Up to 9.0.13 the warfile is unpacked and the webapp is startet. With 9.0.14 tomcat has to be restartet in order to unpack and start the war file.
My host element:
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true" startStopThreads="0">
Comment 1 Remy Maucherat 2019-01-15 13:36:33 UTC
That works for me with the current trunk (place a new war into the host, it is expanded and deployed), but there are many relevant changes in 9.0.14 so I'm not closing it outright. Please try to investigate further.

BTW, startStopThreads="0" might not be doing what you think it does (it will use parallel deployment).
Comment 2 Peter 2019-01-15 14:25:40 UTC
Hi Remy,

I know that startStopThreads="0" uses parallel deployment (as many as cores are available).
I testet on Debian Linux (64 bit, 4 cores, ext4 filesystem) with openjdk11, openjdk8 and oracle jdk8.
Tomcat 9.0.13 and Tomcat 9.0.14 have the same configuration, and same war files.

If I start the server, all war files get deployed.
Then I rename one war file into war.backup. On both Tomcat versions the webapp gets undeployed.
Afterwards I rename the war file back to .war. On Tomcat 9.0.13 the Webapp gets redeployed, on Tomcat 9.0.14 not. A touch to the war file does not change anything. This is independant from the jdk.

Best regards
Comment 3 Remy Maucherat 2019-01-15 14:34:44 UTC
(In reply to Peter from comment #2)
> I know that startStopThreads="0" uses parallel deployment (as many as cores
> are available).
> I testet on Debian Linux (64 bit, 4 cores, ext4 filesystem) with openjdk11,
> openjdk8 and oracle jdk8.
> Tomcat 9.0.13 and Tomcat 9.0.14 have the same configuration, and same war
> files.

Ok, so now it doesn't do exactly that anymore, the new behavior is documented.

> If I start the server, all war files get deployed.
> Then I rename one war file into war.backup. On both Tomcat versions the
> webapp gets undeployed.
> Afterwards I rename the war file back to .war. On Tomcat 9.0.13 the Webapp
> gets redeployed, on Tomcat 9.0.14 not. A touch to the war file does not
> change anything. This is independant from the jdk.

This is a more complex scenario than what is in the BZ description, however it still worked for me on trunk. You should check the logs, filesystem state, deployment state in JMX to investigate this.
Comment 4 Peter 2019-01-16 07:54:36 UTC
There are no entries in the logfile when I try to redeploy a warfile.

However, I found, that if I turn parallel deployment off, autodeploy is working with Tomcat 9.0.14 again. (but my tomcat needs significantly longer to start with a couple webapps)

With parallel deployment (startStopThreads="0") autodeployment works only until Tomcat 9.0.13.
Comment 5 Mark Thomas 2019-01-16 09:26:34 UTC
Related to the server executor refactoring?
Comment 6 Mark Thomas 2019-01-16 09:26:34 UTC
Related to the server executor refactoring?
Comment 7 Remy Maucherat 2019-01-16 09:47:07 UTC
(In reply to Mark Thomas from comment #5)
> Related to the server executor refactoring?

That sounds like it, but I cannot reproduce the issue using the steps and configuration given.
Comment 8 Mark Thomas 2019-01-16 12:27:05 UTC
Just a thought. I can't reproduce it either. Neither did a code inspection suggest any potential configuration that would cause it.

Peter - can you provide the server.xml file for the 9.0.14 install that exhibits this issue please. xxxxxx out any sensitive values such as passwords.
Comment 9 Peter 2019-01-16 12:45:44 UTC
Hi,

her is my server.xml file. Like mentioned before, I do use exactly the same with Tomcat 9.0.13 and 9.0.14. 
If I remove the attribute startStopThreads="0" from the host element, autodeployment is working on 9.014.

Best regard
Peter
P.S. Thank's a lot for the support. I really do appreciate your work. Tomcat is a great pice of software which I am using already for many years.

--
<?xml version="1.0" encoding="UTF-8"?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <!-- Security listener. Documentation at /docs/config/listeners.html
  <Listener className="org.apache.catalina.security.SecurityListener" />
  -->
  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <GlobalNamingResources>

  <!-- here are some Oracle DB resources which I removed ->

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <Service name="Catalina">

    <Connector port="8080" protocol="HTTP/1.1" maxHttpHeaderSize="16384"
               compressableMimeType="text/html,text/xml,text/css,text/javascript, application/x-javascript,application/javascript"
               compression="on"
               connectionTimeout="20000"
               >
    </Connector>

    <Engine name="Catalina" defaultHost="localhost" jvmRoute="limas02">

      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>

      <!-- startStopThreads=0 set deployment Thread to number of available cores -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true" startStopThreads="0">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="access_log" suffix=".txt"
               pattern="%h %l %u %t %D &quot;%r&quot; %s %b" resolveHosts="false" />

      </Host>
    </Engine>
  </Service>
</Server>
Comment 10 Remy Maucherat 2019-01-16 15:05:53 UTC
Except the use of startStopThreads="0" which uses parallel deployment, this looks like a default configuration.

There is nothing which radically changes in the deployer with parallel deployment here. And the executor refactoring, although the prime suspect since it's a big change, simply uses an executor somewhere else.

You can also test simpler scenarios. Does dropping a war into the host appBase auto deploys it ? If so, the most likely problem is the undeployment, if it is not clean then the same webapp will never be deployed again until a restart. Can you verify if the undeploy is clean using JMX ? (you should see beans for the deployed webapps, and other beans will give out their state)
Comment 11 Peter 2019-01-17 07:53:03 UTC
If I copy a new war in the appBase directory,nothing happens. No message in the log file. The war does not get unpacked and started until I restart the tomcat.

If I upload a new war-file with the Tomcat Manager Webapp, the warfile gets unpacked and started.

This behavior is consistant with all my debian based servers. 

I just did some tests in different environments:
- unpack original apache-tomcat-9.0.14.tar.gz
- add startStopThreads="0" to the host element of server.xml
- start tomcat with openjdk11 (on all systems the same one)
- copy a war-file to the appBase directory.

I was using two environments:
1) my Laptop (Ubuntu with Kernel 4.18, Intel Core i7)
2) Debian server with Kernel 3.16.39 (on blade with AMD Opteron 6376 Processors)

On the Laptop deployment of the war file is working, on the server not.

The same test without startStopThreads="0": 
=> war on both systems get deployed

Same test with apache-tomcat-9.0.13.tar.gz and startStopThreads="0" 
=> war on both systems get deployed
Comment 12 Remy Maucherat 2019-01-17 13:40:55 UTC
Ok, so this turns out rather mysterious. The executor used has an unlimited queue and any execution rejection or error is supposed to be logged anyway, so there shouldn't be a problem.
Can you try another value for startStopThreads on your opteron server, like "16" ?
Comment 13 Peter 2019-01-17 13:59:10 UTC
We are getting closer.
I testet with startStopThreads="2",  startStopThreads="4" and  startStopThreads="8"
In these cases autodeploy is working fine. Only startStopThreads="0" has the problem.
Comment 14 Konstantin Kolinko 2019-01-17 14:26:26 UTC
1. Have you looked into Tomcat log files?

2. A comment says:
<!-- startStopThreads=0 set deployment Thread to number of available cores -->

This means that the actual number depends on your hardware.
I wonder what is the actual number of cores that Java reports for you.
Comment 15 Remy Maucherat 2019-01-17 14:43:24 UTC
(In reply to Konstantin Kolinko from comment #14)
> 1. Have you looked into Tomcat log files?
> 
> 2. A comment says:
> <!-- startStopThreads=0 set deployment Thread to number of available cores
> -->
> 
> This means that the actual number depends on your hardware.
> I wonder what is the actual number of cores that Java reports for you.

That was before, I changed it (and documented) to Runtime.getRuntime().availableProcessors() / 2 now since CPUs have smt and I thought the previous default was way too high for the tasks at hand. Not this opteron though ;) Runtime.getRuntime().availableProcessors() is supposed to return 16, so the new default should be 8, and before it was 16.
Comment 16 Coty Sutherland 2019-01-17 21:03:47 UTC
Are you able to reproduce the problem 100% of the time? How large are your webapps and how long does it take to deploy/undeploy? Does the same webapp fail repeatedly, or are you able to reproduce with different webapps? Could you try reproducing with the webapps provided with vanilla tomcat (i.e. examples, ROOT)? Using a vanilla webapp will help rule out/pinpoint anything weird happening with the webapp you're currently reproducing with.
Comment 17 Remy Maucherat 2019-01-17 21:53:15 UTC
Negative or zero values will now again use (Runtime.getRuntime().availableProcessors() + value) in Tomcat 9.0.15.

It's not really related to the mysterious behavior, but it shouldn't have been changed as this default value had been in place for a long while (I approached the item with a "all my CPUs of the past 10 years have SMT so this is wasteful" thinking).
Comment 18 Peter 2019-01-18 06:11:38 UTC
We operate more than 30 of tomcat server with many different webapps. All server run in a big blade environments in virtual maschines.
New versions of these webapps from many developer gets automatically deployed in the test environment. The number of cores Runtime.getRuntime().availableProcessors() reports on these test maschines is allways between 2 and 8 (depending on the VM settings).
After I updated Tomcat to Version 9.0.14 the automatic deployment did not work anymore.
Some webapps are very small with only a few libraries, others are rather large (spring boot with many libs). It was a generell Problem of all maschines.

Without parallel deployment, the tomcats needed significantly longer to start, since a view of them host more than 20 different webapps. Now I changed the value to startStopThreads="4" and my problem is solved. 

Thanks to all of you who helped to find the reason and for providing the tomcat server software.
Comment 19 Mark Thomas 2019-01-21 15:46:29 UTC
We still haven't got to the bottom of what is going on here.

Could you reproduce then and then take a thread dump? I am wondering if the utility thread pool ends up with a single thread and that thread gets blocked during a task.
Comment 20 Peter 2019-01-22 08:12:29 UTC
Hi,
for the test I used a vanilla tomcat, added startStopThreads="0" startet it only with the manager webapp and one smnall war-file (Echo.war)

1) after startup I took the first threadump
2) undeployed the the war file and made the second threadump
3) waited a minute and copied the war file back into the webapp directory.
4) waited a minute (nothing happend) and made the thrid threaddump
5) shutdown tomcat

below you'll find the last threaddump as well as the catalina.out entries for stopping the server. The Exception at the end from shutdown might be interesting for you. 

Let me know if  you also need the first two thread dumps.

best regards
Peter

2019-01-22 08:45:03
Full thread dump OpenJDK 64-Bit Server VM (11.0.1+13 mixed mode):

Threads class SMR info:
_java_thread_list=0x000000000228ddf0, length=40, elements={
0x00000000020fb800, 0x00000000021f7000, 0x0000000002201800, 0x0000000002219000,
0x000000000221b800, 0x000000000221d800, 0x0000000002220000, 0x00007f62940a6000,
0x00007f62940ad800, 0x00007f629412c000, 0x00007f6294165000, 0x00007f628c130800,
0x00007f628c135000, 0x00007f628c15e800, 0x00007f628c1ab800, 0x00007f628c1ac800,
0x00007f628c1ae000, 0x00007f628c3ae000, 0x00007f628c3b0000, 0x00007f628c3b8000,
0x00007f628c3ba000, 0x00007f628c3c0000, 0x00007f628c3c4800, 0x00007f628c3c7800,
0x00007f628c3c8800, 0x00007f628c3cb000, 0x00007f628c3d0800, 0x00007f628c3d2000,
0x00007f628c3d6000, 0x00007f628c3d8000, 0x00007f628c3da000, 0x00007f628c3df000,
0x00007f628c3e4000, 0x00007f628c3e6000, 0x000000000228b000, 0x000000000228c000,
0x000000000228f800, 0x0000000002290800, 0x0000000002291800, 0x000000000230c800
}

"main" #1 prio=5 os_prio=0 cpu=2599.92ms elapsed=489.68s tid=0x00000000020fb800 nid=0x37a6d1 runnable  [0x00007f62b9d5c000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(java.base@11.0.1/Native Method)
	at java.net.AbstractPlainSocketImpl.accept(java.base@11.0.1/AbstractPlainSocketImpl.java:458)
	at java.net.ServerSocket.implAccept(java.base@11.0.1/ServerSocket.java:551)
	at java.net.ServerSocket.accept(java.base@11.0.1/ServerSocket.java:519)
	at org.apache.catalina.core.StandardServer.await(StandardServer.java:612)
	at org.apache.catalina.startup.Catalina.await(Catalina.java:725)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:671)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.1/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.1/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.1/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.1/Method.java:566)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=5.31ms elapsed=489.61s tid=0x00000000021f7000 nid=0x37a6d8 waiting on condition  [0x00007f6298c50000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.1/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.1/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.1/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=1.23ms elapsed=489.61s tid=0x0000000002201800 nid=0x37a6d9 in Object.wait()  [0x00007f6298b4f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.1/Native Method)
	- waiting on <0x00000000b243d178> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000000b243d178> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.1/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.64ms elapsed=489.59s tid=0x0000000002219000 nid=0x37a6da waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=3847.63ms elapsed=489.59s tid=0x000000000221b800 nid=0x37a6db waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=2047.40ms elapsed=489.59s tid=0x000000000221d800 nid=0x37a6dc waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #7 daemon prio=9 os_prio=0 cpu=83.64ms elapsed=489.59s tid=0x0000000002220000 nid=0x37a6dd runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #8 daemon prio=8 os_prio=0 cpu=5.75ms elapsed=489.43s tid=0x00007f62940a6000 nid=0x37a6de in Object.wait()  [0x00007f62984e2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.1/Native Method)
	- waiting on <0x00000000b25a46a0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000000b25a46a0> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.1/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.1/InnocuousThread.java:134)

"Service Thread" #9 daemon prio=9 os_prio=0 cpu=0.13ms elapsed=489.40s tid=0x00007f62940ad800 nid=0x37a6df runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"AsyncFileHandlerWriter-1632392469" #12 daemon prio=5 os_prio=0 cpu=199.67ms elapsed=489.23s tid=0x00007f629412c000 nid=0x37a6e1 waiting on condition  [0x00007f62981ae000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b240f5a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(java.base@11.0.1/LinkedBlockingDeque.java:513)
	at java.util.concurrent.LinkedBlockingDeque.poll(java.base@11.0.1/LinkedBlockingDeque.java:675)
	at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:159)

"FileHandlerLogFilesCleaner-1" #13 daemon prio=5 os_prio=0 cpu=94.13ms elapsed=489.04s tid=0x00007f6294165000 nid=0x37a6e2 waiting on condition  [0x00007f6293bce000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b240eab8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"NioBlockingSelector.BlockPoller-1" #14 daemon prio=5 os_prio=0 cpu=156.28ms elapsed=486.80s tid=0x00007f628c130800 nid=0x37a6e6 runnable  [0x00007f6292ab6000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
	- locked <0x00000000b5eaefd0> (a sun.nio.ch.Util$2)
	- locked <0x00000000b5eaee70> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
	at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)

"NioBlockingSelector.BlockPoller-2" #15 daemon prio=5 os_prio=0 cpu=122.18ms elapsed=486.78s tid=0x00007f628c135000 nid=0x37a6e7 runnable  [0x00007f62929b5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
	- locked <0x00000000b5ead6a0> (a sun.nio.ch.Util$2)
	- locked <0x00000000b5ead540> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
	at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)

"Catalina-utility-1" #16 prio=1 os_prio=0 cpu=1575.38ms elapsed=486.57s tid=0x00007f628c15e800 nid=0x37a6e8 waiting on condition  [0x00007f62928b3000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8142908> (a java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.FutureTask.awaitDone(java.base@11.0.1/FutureTask.java:447)
	at java.util.concurrent.FutureTask.get(java.base@11.0.1/FutureTask.java:190)
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
	at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1621)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1149)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1351)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1355)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1333)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.1/Executors.java:515)
	at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.1/FutureTask.java:305)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.1/ScheduledThreadPoolExecutor.java:305)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-1" #17 daemon prio=5 os_prio=0 cpu=0.78ms elapsed=485.31s tid=0x00007f628c1ab800 nid=0x37a6ea waiting on condition  [0x00007f62923b3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-2" #18 daemon prio=5 os_prio=0 cpu=1.41ms elapsed=485.31s tid=0x00007f628c1ac800 nid=0x37a6eb waiting on condition  [0x00007f62922b2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-3" #19 daemon prio=5 os_prio=0 cpu=0.51ms elapsed=485.31s tid=0x00007f628c1ae000 nid=0x37a6ec waiting on condition  [0x00007f62921b1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-4" #20 daemon prio=5 os_prio=0 cpu=0.50ms elapsed=485.31s tid=0x00007f628c3ae000 nid=0x37a6ed waiting on condition  [0x00007f62920b0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-5" #21 daemon prio=5 os_prio=0 cpu=0.42ms elapsed=485.31s tid=0x00007f628c3b0000 nid=0x37a6ee waiting on condition  [0x00007f6291faf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-9" #25 daemon prio=5 os_prio=0 cpu=0.45ms elapsed=485.31s tid=0x00007f628c3b8000 nid=0x37a6f2 waiting on condition  [0x00007f6291bab000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-10" #26 daemon prio=5 os_prio=0 cpu=0.68ms elapsed=485.31s tid=0x00007f628c3ba000 nid=0x37a6f3 waiting on condition  [0x00007f6291aaa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-ClientPoller-0" #27 daemon prio=5 os_prio=0 cpu=130.28ms elapsed=485.30s tid=0x00007f628c3c0000 nid=0x37a6f4 runnable  [0x00007f62919a9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
	- locked <0x00000000b888a468> (a sun.nio.ch.Util$2)
	- locked <0x00000000b888a318> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:744)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-ClientPoller-1" #28 daemon prio=5 os_prio=0 cpu=134.09ms elapsed=485.30s tid=0x00007f628c3c4800 nid=0x37a6f5 runnable  [0x00007f62918a8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
	- locked <0x00000000b888ac68> (a sun.nio.ch.Util$2)
	- locked <0x00000000b888ab18> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:744)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-Acceptor-0" #29 daemon prio=5 os_prio=0 cpu=1.02ms elapsed=485.29s tid=0x00007f628c3c7800 nid=0x37a6f6 runnable  [0x00007f62917a7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(java.base@11.0.1/Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.1/ServerSocketChannelImpl.java:533)
	at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.1/ServerSocketChannelImpl.java:285)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
	at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-1" #30 daemon prio=5 os_prio=0 cpu=0.41ms elapsed=485.28s tid=0x00007f628c3c8800 nid=0x37a6f7 waiting on condition  [0x00007f62916a6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-2" #31 daemon prio=5 os_prio=0 cpu=0.35ms elapsed=485.27s tid=0x00007f628c3cb000 nid=0x37a6f8 waiting on condition  [0x00007f62915a5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-5" #34 daemon prio=5 os_prio=0 cpu=0.40ms elapsed=485.26s tid=0x00007f628c3d0800 nid=0x37a6fb waiting on condition  [0x00007f62912a2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-6" #35 daemon prio=5 os_prio=0 cpu=0.47ms elapsed=485.26s tid=0x00007f628c3d2000 nid=0x37a6fc waiting on condition  [0x00007f62911a1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-8" #37 daemon prio=5 os_prio=0 cpu=0.43ms elapsed=485.22s tid=0x00007f628c3d6000 nid=0x37a6fe waiting on condition  [0x00007f6290f9f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-9" #38 daemon prio=5 os_prio=0 cpu=0.43ms elapsed=485.22s tid=0x00007f628c3d8000 nid=0x37a6ff waiting on condition  [0x00007f6290e9e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-10" #39 daemon prio=5 os_prio=0 cpu=0.48ms elapsed=485.22s tid=0x00007f628c3da000 nid=0x37a700 waiting on condition  [0x00007f6290d9d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.1/LinkedBlockingQueue.java:458)
	at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:97)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-ClientPoller-0" #40 daemon prio=5 os_prio=0 cpu=191.54ms elapsed=485.22s tid=0x00007f628c3df000 nid=0x37a701 runnable  [0x00007f6290c9c000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
	- locked <0x00000000b889b930> (a sun.nio.ch.Util$2)
	- locked <0x00000000b889b7e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:744)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-ClientPoller-1" #41 daemon prio=5 os_prio=0 cpu=110.88ms elapsed=485.22s tid=0x00007f628c3e4000 nid=0x37a702 runnable  [0x00007f6290b9b000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
	- locked <0x00000000b889c068> (a sun.nio.ch.Util$2)
	- locked <0x00000000b889bf18> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:744)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-Acceptor-0" #42 daemon prio=5 os_prio=0 cpu=0.19ms elapsed=485.22s tid=0x00007f628c3e6000 nid=0x37a703 runnable  [0x00007f6290a9a000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(java.base@11.0.1/Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.1/ServerSocketChannelImpl.java:533)
	at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.1/ServerSocketChannelImpl.java:285)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
	at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-11" #44 daemon prio=5 os_prio=0 cpu=0.33ms elapsed=174.59s tid=0x000000000228b000 nid=0x37ab22 waiting on condition  [0x00007f6290999000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-11" #45 daemon prio=5 os_prio=0 cpu=0.26ms elapsed=174.59s tid=0x000000000228c000 nid=0x37ab23 waiting on condition  [0x00007f6290898000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-12" #46 daemon prio=5 os_prio=0 cpu=0.68ms elapsed=114.59s tid=0x000000000228f800 nid=0x37ab5d waiting on condition  [0x00007f62910a0000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-12" #47 daemon prio=5 os_prio=0 cpu=0.22ms elapsed=114.59s tid=0x0000000002290800 nid=0x37ab5e waiting on condition  [0x00007f6291cac000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"http-nio-8082-exec-13" #48 daemon prio=5 os_prio=0 cpu=0.45ms elapsed=54.59s tid=0x0000000002291800 nid=0x37ada9 waiting on condition  [0x00007f62914a4000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b8848a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"ajp-nio-8009-exec-13" #49 daemon prio=5 os_prio=0 cpu=0.26ms elapsed=54.59s tid=0x000000000230c800 nid=0x37adaa waiting on condition  [0x00007f6291dad000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x00000000b88993b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.1/LinkedBlockingQueue.java:433)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"VM Thread" os_prio=0 cpu=279.93ms elapsed=489.65s tid=0x00000000021ef000 nid=0x37a6d7 runnable  

"GC Thread#0" os_prio=0 cpu=139.54ms elapsed=489.69s tid=0x0000000002123000 nid=0x37a6d2 runnable  

"GC Thread#1" os_prio=0 cpu=123.15ms elapsed=488.62s tid=0x0000000002574800 nid=0x37a6e3 runnable  

"G1 Main Marker" os_prio=0 cpu=3.29ms elapsed=489.69s tid=0x0000000002143800 nid=0x37a6d3 runnable  

"G1 Conc#0" os_prio=0 cpu=49.41ms elapsed=489.69s tid=0x0000000002145800 nid=0x37a6d4 runnable  

"G1 Refine#0" os_prio=0 cpu=11.20ms elapsed=489.69s tid=0x000000000218e000 nid=0x37a6d5 runnable  

"G1 Refine#1" os_prio=0 cpu=0.08ms elapsed=488.60s tid=0x0000000002380000 nid=0x37a6e4 runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=449.46ms elapsed=489.69s tid=0x0000000002190000 nid=0x37a6d6 runnable  
"VM Periodic Task Thread" os_prio=0 cpu=1743.61ms elapsed=489.41s tid=0x00007f62940b8000 nid=0x37a6e0 waiting on condition  

JNI global refs: 19, weak refs: 0

Heap
 garbage-first heap   total 107520K, used 22234K [0x00000000b2400000, 0x0000000100000000)
  region size 1024K, 15 young (15360K), 3 survivors (3072K)
 Metaspace       used 22200K, capacity 22728K, committed 22960K, reserved 1069056K
  class space    used 2286K, capacity 2520K, committed 2560K, reserved 1048576K

22-Jan-2019 08:46:03.627 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
22-Jan-2019 08:46:03.629 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8082"]
22-Jan-2019 08:46:03.657 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]
22-Jan-2019 08:46:03.667 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
22-Jan-2019 08:46:03.670 SEVERE [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWARs Error waiting for multi-thread deployment of WAR files to complete
 java.lang.InterruptedException
	at java.base/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:190)
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
	at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1621)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1149)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1351)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1355)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1333)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)

22-Jan-2019 08:46:03.678 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/srv/t/apache-tomcat-9.0.14/webapps/Echo.war]
22-Jan-2019 08:46:03.678 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/srv/t/apache-tomcat-9.0.14/webapps/Echo.war] has finished in [1] ms
22-Jan-2019 08:46:03.693 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8082"]
22-Jan-2019 08:46:03.718 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
22-Jan-2019 08:46:03.726 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8082"]
22-Jan-2019 08:46:03.730 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
Comment 21 Remy Maucherat 2019-01-22 08:53:10 UTC
Ok, Mark got it right: only one utility thread (thanks to my bad new default ... now fixed), blocked during deployment. No idea why this happens at this point.

The log entry here also gave away the problem:
22-Jan-2019 08:46:03.670 SEVERE [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWARs Error waiting for multi-thread deployment of WAR files to complete
 java.lang.InterruptedException
	at java.base/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:190)
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)
Comment 22 Mark Thomas 2019-01-22 13:31:06 UTC
Got it.

The one utility thread is being used for background processing. There are then no threads left for multi-threaded deployment so the whole thing blocks.

This might only have been possible when the  "/ 2" was present. I'll double check the logic.
Comment 23 Remy Maucherat 2019-01-22 13:44:39 UTC
Ok, I understand it too now, the background processor uses the first thread and then the deployment task waits in the queue forever. I'll try to add code that makes parallel deployment use a minimum of two threads.
Comment 24 Mark Thomas 2019-01-22 13:49:25 UTC
First of all I forgot to say thank you to Peter for the thread dump. It was extremely helpful.

I'd lean towards increasing the default and minimum utility thread count from one to two. That should fix this issue and address folks using silly values like -100 as well.
Comment 25 Rainer Jung 2019-01-22 13:57:21 UTC
Small addition: I think Rémy mentioned that we are currently using an unlimited queue for the executor. It is one of the patterns I typically do not like much, because then problems will go unnoticed for a long time and the queue can consume unbounded memory.

Wouldn't it be better to limit the queue and log any exception resulting from the queue being full? I haven't checked though, whether there's a reasonable default queue size (it might depend on the number of context to deploy and the number of threads doing the work). But except for unusual cases a queue of length 20 might suffice (plus configurability and docs).

WDYT?

Regards,

Rainer
Comment 26 Remy Maucherat 2019-01-22 14:05:16 UTC
(In reply to Mark Thomas from comment #24)
> First of all I forgot to say thank you to Peter for the thread dump. It was
> extremely helpful.
> 
> I'd lean towards increasing the default and minimum utility thread count
> from one to two. That should fix this issue and address folks using silly
> values like -100 as well.

Maybe yes, otherwise and blocking pattern like the one used by the deployer is not allowed. Although in theory it's not a very good pattern.

(In reply to Rainer Jung from comment #25)
> Wouldn't it be better to limit the queue and log any exception resulting
> from the queue being full?

Sure, try it ;)
(read the scheduled executor javadoc first though)

Personally, I like to focus on real problems, and that's not one, so we're fine.
Comment 27 Remy Maucherat 2019-01-22 14:29:14 UTC
Thanks for all the debugging and info.
The minimum thread count value will now be two in Tomcat 9.0.15.
Comment 28 Rainer Jung 2019-01-22 15:11:37 UTC
> (In reply to Rainer Jung from comment #25)
> > Wouldn't it be better to limit the queue and log any exception resulting
> > from the queue being full?
> 
> Sure, try it ;)
> (read the scheduled executor javadoc first though)

Thanks for the free lesson Rémy, so once we use ScheduledThreadPoolExecutor the queue is fixed and can not be limited.

> Personally, I like to focus on real problems, and that's not one, so we're
> fine.

Although I don't quite understand why this is not a "real problem", I can see the language drift between "I like" and "we're fine". Of course I am OK with you not pursuing this any further ;)

Regards,

Rainer
Comment 29 Remy Maucherat 2019-01-22 15:27:52 UTC
(In reply to Rainer Jung from comment #28)
> > (In reply to Rainer Jung from comment #25)
> > > Wouldn't it be better to limit the queue and log any exception resulting
> > > from the queue being full?
> > 
> > Sure, try it ;)
> > (read the scheduled executor javadoc first though)
> 
> Thanks for the free lesson Rémy, so once we use ScheduledThreadPoolExecutor
> the queue is fixed and can not be limited.

I am frustrated by this :( It is also not possible to set a max thread count (it is always the core count), and expiring idle threads is bad too (it will periodically respawn the core threads instead of reducing live thread count). On the plus side, it is actually more robust in the real world than what we had before, since a task monitor is very easy to implement (there is one for all periodic tasks).

The EE specification has a (more complete) scheduled executor API. (but it's not really worth doing it)
https://docs.oracle.com/javaee/7/api/javax/enterprise/concurrent/ManagedScheduledExecutorService.html
Comment 30 Rainer Jung 2019-01-22 15:57:38 UTC
I agree.