Bug 64158

Hi Tomcat,

During our performance testing, we found out another Tomcat 7 Acceptor bottleneck, which is the Tomcat's fix to enforce busy-thread-ratio below threshold (default appears to 75%) else disable-keep-alive. E.g.  org.apache.coyote.http11.Http11Processor#disableKeepAlive

This change was added in 2009 to lock-read worker size (in JIoEndpoint#getCurrentThreadsBusy and later refactored to AbstractEndpoint#getCurrentThreadsBusy). https://github.com/apache/tomcat/commit/ecc2fb6f2ecb7f8cb51559adcfca82fa70b2bebf#diff-3253c70c8b90ec7cd422faf801dfa4f8


In Tomcat 6, there's no enforcement to disable keep alive when busy threads go above disable-keep-alive-percentage.


In Tomcat 7, we're seeing added contention for the lock in java.util.concurrent.ThreadPoolExecutor#mainLock. This cause more pressure on get-pool-size which is also waiting for the same lock.




example stacktrace:


"tomcat-exec-executor-2@16852" daemon prio=5 tid=0x66 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
      at java.util.concurrent.ThreadPoolExecutor.getActiveCount(ThreadPoolExecutor.java:1830)
      at org.apache.catalina.core.StandardThreadExecutor.getActiveCount(StandardThreadExecutor.java:281)
      at org.apache.tomcat.util.net.AbstractEndpoint.getCurrentThreadsBusy(AbstractEndpoint.java:547)
      at org.apache.coyote.http11.Http11Processor.disableKeepAlive(Http11Processor.java:130)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
      - locked <merged>(a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:748)