Bug 62185

Summary: Acceptor Thread Terminates
Product: Tomcat 7 Reporter: BogdanD <bogdan.durla>
Component: ConnectorsAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED INVALID    
Severity: normal    
Priority: P2    
Version: 7.0.85   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description BogdanD 2018-03-16 15:03:10 UTC
Acceptor Thread terminates and no further requests are being accepted by Apache Tomcat.

Steps to reproduce: 

Set max user processes to a low number(to the user that starts the server): 

e.g.: ulimit -u 40

run a few thousand requests or more to an Apache endpoint. 

If the ulimit for max processes is reached, java/org/apache/tomcat/util/net/JIoEndpoint.java:255 will throw a VirtualMachineError and it will terminate the AcceptorThread.

Exception in thread "http-bio-8080-Acceptor-0" java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:166)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:146)
        at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:533)
        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:236)
        at java.lang.Thread.run(Thread.java:745)
Comment 1 Mark Thomas 2018-03-16 17:06:33 UTC
That is an (OS) configuration error, not a Tomcat bug.

Generally, OOME are not expected to be recoverable. If Tomcat experiences one it needs to be shutdown (if it isn't already) and restarted.
Comment 2 Christopher Schultz 2018-03-16 17:47:01 UTC
(In reply to Mark Thomas from comment #1)
> That is an (OS) configuration error, not a Tomcat bug.

For OP: The correct solution to this is to make sure that the (total) <Connector>'s maxThreads is low enough that this should not happen. That's why Mark is asserting that this is a configuration error.
Comment 3 Remy Maucherat 2018-03-16 18:20:03 UTC
Christopher, we're not nice and not giving straight answers in BZ to user questions, otherwise it would be a super easy and convenient way to get free support from the core Tomcat developers.
Comment 4 BogdanD 2018-03-19 11:25:18 UTC
(In reply to Mark Thomas from comment #1)
> That is an (OS) configuration error, not a Tomcat bug.
> 
> Generally, OOME are not expected to be recoverable. If Tomcat experiences
> one it needs to be shutdown (if it isn't already) and restarted.

Hi guys, thanks for the replies. I understand the root cause is the "misconfiguration" but do you consider it as normal behavior to have the Apache process(after the exception) up and running but not accepting any request since it's Acceptor thread is gone forever?

We consider this an improper handling of the case when Tomcat cannot create new worker threads(The OOME thrown by the JVM is also a bit misleading in this case, because is not a memory issue).

We encountered this in a production environment, during a spike and also because there were other processes running and forking in the same UID as Tomcat. The Tomcat process was still up but it couldn't accept new connections.
Comment 5 Christopher Schultz 2018-03-19 13:45:44 UTC
           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|REOPENED                    |RESOLVED

That's Rémy's way of encouraging you to join the user or dev lists. Bugzilla isn't a great place to have a conversation. Please post there and reference this BZ issue.