Bug 62185 - Acceptor Thread Terminates
Summary: Acceptor Thread Terminates
Status: RESOLVED INVALID
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Connectors (show other bugs)
Version: 7.0.85
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-03-16 15:03 UTC by BogdanD
Modified: 2018-03-19 13:45 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description BogdanD 2018-03-16 15:03:10 UTC
Acceptor Thread terminates and no further requests are being accepted by Apache Tomcat.

Steps to reproduce: 

Set max user processes to a low number(to the user that starts the server): 

e.g.: ulimit -u 40

run a few thousand requests or more to an Apache endpoint. 

If the ulimit for max processes is reached, java/org/apache/tomcat/util/net/JIoEndpoint.java:255 will throw a VirtualMachineError and it will terminate the AcceptorThread.

Exception in thread "http-bio-8080-Acceptor-0" java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:166)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:146)
        at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:533)
        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:236)
        at java.lang.Thread.run(Thread.java:745)
Comment 1 Mark Thomas 2018-03-16 17:06:33 UTC
That is an (OS) configuration error, not a Tomcat bug.

Generally, OOME are not expected to be recoverable. If Tomcat experiences one it needs to be shutdown (if it isn't already) and restarted.
Comment 2 Christopher Schultz 2018-03-16 17:47:01 UTC
(In reply to Mark Thomas from comment #1)
> That is an (OS) configuration error, not a Tomcat bug.

For OP: The correct solution to this is to make sure that the (total) <Connector>'s maxThreads is low enough that this should not happen. That's why Mark is asserting that this is a configuration error.
Comment 3 Remy Maucherat 2018-03-16 18:20:03 UTC
Christopher, we're not nice and not giving straight answers in BZ to user questions, otherwise it would be a super easy and convenient way to get free support from the core Tomcat developers.
Comment 4 BogdanD 2018-03-19 11:25:18 UTC
(In reply to Mark Thomas from comment #1)
> That is an (OS) configuration error, not a Tomcat bug.
> 
> Generally, OOME are not expected to be recoverable. If Tomcat experiences
> one it needs to be shutdown (if it isn't already) and restarted.

Hi guys, thanks for the replies. I understand the root cause is the "misconfiguration" but do you consider it as normal behavior to have the Apache process(after the exception) up and running but not accepting any request since it's Acceptor thread is gone forever?

We consider this an improper handling of the case when Tomcat cannot create new worker threads(The OOME thrown by the JVM is also a bit misleading in this case, because is not a memory issue).

We encountered this in a production environment, during a spike and also because there were other processes running and forking in the same UID as Tomcat. The Tomcat process was still up but it couldn't accept new connections.
Comment 5 Christopher Schultz 2018-03-19 13:45:44 UTC
           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|REOPENED                    |RESOLVED

That's Rémy's way of encouraging you to join the user or dev lists. Bugzilla isn't a great place to have a conversation. Please post there and reference this BZ issue.