Bug 51240 - maxConnections not honors config when acceptorThreadCount > 1
Summary: maxConnections not honors config when acceptorThreadCount > 1
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Connectors (show other bugs)
Version: 7.0.14
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-05-22 19:22 UTC by Guillermo Grandes
Modified: 2011-05-28 19:25 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Guillermo Grandes 2011-05-22 19:22:26 UTC
When acceptorThreadCount > 1, maxConnections not honors config, this affects BIO & NIO connector (others don't know).

---------- test config begin --------

    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="300" minSpareThreads="4"/>
    
    <Connector port="9993"
               protocol="org.apache.coyote.http11.Http11Protocol"
               URIEncoding="ISO-8859-1"
               enableLookups="false"

               acceptorThreadCount="2"
               executor="tomcatThreadPool"
               acceptCount="1"
               maxConnections="1"
               />

    <Connector port="9994"
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               URIEncoding="ISO-8859-1"
               enableLookups="false"

               acceptorThreadCount="2"
               executor="tomcatThreadPool"
               acceptCount="1"
               maxConnections="1"
               />

---------- test config end ----------

---- Test-1 (acceptorThreadCount="1") --- OK ---
  $ ab2 -n 20000 -c 1000 http://localhost:999x/
  $ netstat -atn | grep :999x | grep ESTABLISHED

  This show range 3-5 connections (acceptable number)

---- Test-2 (acceptorThreadCount="2") --- KO ---
  $ ab2 -n 20000 -c 1000 http://localhost:999x/
  $ netstat -atn | grep :999x | grep ESTABLISHED

  This show +100 connections > "OOPS", too much far

----

Logs show traces like this when socket closes:
----- catalina.out begin -----
May 22, 2011 9:10:51 PM org.apache.tomcat.util.net.AbstractEndpoint countDownConnection
WARNING: Incorrect connection count, multiple socket.close called on the same socket.
----- catalina.out end -------
Comment 1 Filip Hanik 2011-05-23 19:09:58 UTC
I'd strongly recommend to default acceptorThreadCount=1 and possible deprecating the attribute all together.
Acceptor thread count larger than 1 has really zero impact on performance, while degrading system resources.
The operating system holds a shared lock underneath to accept new connections, so having multiple threads calling ServerSocket.accept is doing nothing except queuing up for that lock.
Comment 2 Mark Thomas 2011-05-25 07:30:50 UTC
I agree with Filip's comment re forcing acceptorThreadCount to 1 but I'd still like to get to the bottom of why the maxConnections limit isn't being enforced. I'll try and look at this today.
Comment 3 Tim Whittington 2011-05-25 10:37:48 UTC
It looks like there's a race condition between the acceptor thread being permitted to accept a connection and updating the connection counter.

Two acceptor threads can pass the awaitConnection() condition, accept a connection, and then both call countUp(). The connection count then goes above the signal level and awaitConnection() blocking condition will never be met as long as the connections stays above max (since CounterLatch appears to be designed to count both up and down, it compares the signal level exactly).

The fix could be to remove countUp()/countDown(), change CounterLatch.await() to an CounterLatch.awaitAndIncrement()/CounterLatch.awaitAndDecrement() pair and have the connection count atomically updated in Sync.tryAcquireShared() using AtomicLong.compareAndSet() with the +1/-1 delta passed in as the argument.

e.g:

        protected int tryAcquireShared(int delta) {
            while (true) {
                final long current = count.get();
                if (!released && (current == signal)) {
                    return -1;
                }
                if (count.compareAndSet(current, current + delta)) {
                    return 1;
                }
            }
        }
Comment 4 Mark Thomas 2011-05-26 15:25:07 UTC
I ended up replacing CounterLatch with LimitLatch that has reduced functionality that is aligned more closely with what the connectors need.

I also made maxConnections dynamically configurable.

I'll look into removing / deprecating the acceptorThreadCount attribute next.

This is fixed in 7.0.x and will be included in 7.0.15.

Note: If testing with ab be aware that the TCP backlog will cause more established connections to be observed than Tomcat is currently handling.
Comment 5 Mark Thomas 2011-05-28 19:25:54 UTC
On reflection I decided to leave the acceptorThreadCount configuration option. It already defaults to one and since the acceptor thread does more the just call socket.accept() if there is a spike in new connections, it is possible that multiple acceptor threads may offer some limited benefit.