Hi Tomcat, During our performance testing, we found out another Tomcat 7 Acceptor bottleneck, which is the Tomcat's fix to enforce busy-thread-ratio below threshold (default appears to 75%) else disable-keep-alive. E.g. org.apache.coyote.http11.Http11Processor#disableKeepAlive This change was added in 2009 to lock-read worker size (in JIoEndpoint#getCurrentThreadsBusy and later refactored to AbstractEndpoint#getCurrentThreadsBusy). https://github.com/apache/tomcat/commit/ecc2fb6f2ecb7f8cb51559adcfca82fa70b2bebf#diff-3253c70c8b90ec7cd422faf801dfa4f8 In Tomcat 6, there's no enforcement to disable keep alive when busy threads go above disable-keep-alive-percentage. In Tomcat 7, we're seeing added contention for the lock in java.util.concurrent.ThreadPoolExecutor#mainLock. This cause more pressure on get-pool-size which is also waiting for the same lock. example stacktrace: "tomcat-exec-executor-2@16852" daemon prio=5 tid=0x66 nid=NA runnable java.lang.Thread.State: RUNNABLE at java.util.concurrent.ThreadPoolExecutor.getActiveCount(ThreadPoolExecutor.java:1830) at org.apache.catalina.core.StandardThreadExecutor.getActiveCount(StandardThreadExecutor.java:281) at org.apache.tomcat.util.net.AbstractEndpoint.getCurrentThreadsBusy(AbstractEndpoint.java:547) at org.apache.coyote.http11.Http11Processor.disableKeepAlive(Http11Processor.java:130) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) - locked <merged>(a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
Created attachment 37025 [details] Remove enforcement that disable keep-alive when busy threads go above disable-keep-alive-percentage
The patch is not a good move. If you have a problem with this, you can disable it and recompile as you did, but in practice the solution is to not use java.io. As there is no good solution here, I'm closing the issue.