Bug 58313 - Data race inside the non-thread-safe HashMap org.apache.catalina.connector.OutputBuffer.encoders
Summary: Data race inside the non-thread-safe HashMap org.apache.catalina.connector.Ou...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 6
Classification: Unclassified
Component: Catalina (show other bugs)
Version: unspecified
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-09-01 13:34 UTC by Yilong Li
Modified: 2015-10-25 21:58 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Yilong Li 2015-09-01 13:34:55 UTC
I am running the test suite against a dynamic race detector called RV-Predict. I get a few race reports on the HashMap org.apache.catalina.connector.OutputBuffer.encoders:

Data race on field java.util.HashMap.$state: {{{
Concurrent write in thread T83 (locks held: {Monitor@67298f15})
 ---->  at org.apache.catalina.connector.OutputBuffer.clearEncoders(OutputBuffer.java:255)
        at org.apache.catalina.connector.Response.clearEncoders(Response.java:295)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:587)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
        at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1073)
        at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.run(Nio2Endpoint.java:1032)
        - locked Monitor@67298f15 at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.run(Nio2Endpoint.java:1031)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    T83 is created by T82
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:1010)

Concurrent read in thread T84 (locks held: {})
 ---->  at org.apache.catalina.connector.OutputBuffer.setConverter(OutputBuffer.java:580)
        at org.apache.catalina.connector.OutputBuffer.checkConverter(OutputBuffer.java:563)
        at org.apache.catalina.connector.Response.getWriter(Response.java:599)
        at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:212)
        at org.apache.coyote.http11.TestAbstractHttp11Processor$Bug57621Servlet$1.run(TestAbstractHttp11Processor.java:757)
        at org.apache.catalina.core.AsyncContextImpl$RunnableWrapper.run(AsyncContextImpl.java:549)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    T84 is created by T83
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:1010)
}}} 

Data race on field java.util.HashMap.$state: {{{
Concurrent write in thread T60 (locks held: {Monitor@93a8632})
 ---->  at org.apache.catalina.connector.OutputBuffer.clearEncoders(OutputBuffer.java:255)
        at org.apache.catalina.connector.Response.clearEncoders(Response.java:295)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:587)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:277)
        - locked Monitor@93a8632 at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:259)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    T60 is created by T58
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:1010)

Concurrent read in thread T61 (locks held: {})
 ---->  at org.apache.catalina.connector.OutputBuffer.setConverter(OutputBuffer.java:604)
        at org.apache.catalina.connector.OutputBuffer.checkConverter(OutputBuffer.java:563)
        at org.apache.catalina.connector.Response.getWriter(Response.java:599)
        at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:212)
        at org.apache.coyote.http11.TestAbstractHttp11Processor$Bug57621Servlet$1.run(TestAbstractHttp11Processor.java:757)
        at org.apache.catalina.core.AsyncContextImpl$RunnableWrapper.run(AsyncContextImpl.java:549)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    T61 is created by T60
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:1010)
}}} 

Looks like OutputBuffer.encoders is accessed from multiple threads without proper synchronization.
Comment 1 Mark Thomas 2015-09-02 15:19:24 UTC
I agree that this is an issue. A switch to ConcurrentHashMap should address the immediate problem.

The idea behind clearEncoders() is a good one. The encoders are expensive objects and don't scale well. We don't really want to keep the Map around during an async request. On the otherhand, we don't really want to have to keep re-creating it.

I'm wondering about ways to improve the currenty approach. I'm going to do some simple testing to see how an approach based on pools of encoders would perform.
Comment 2 Mark Thomas 2015-09-02 19:08:44 UTC
Fixed in trunk, 8.0.x (for 8.0.27 onwards) and 7.0.x (for 7.0.65 onwards). Back-port proposed for 6.0.x.
Comment 3 Konstantin Kolinko 2015-10-25 21:58:01 UTC
Fixed in Tomcat 6 by r1710487 and will be in 6.0.45 onwards.