Bug 61673 - java.util.ConcurrentModificationException thrown when trying to push around 1000 resources using PushBuilder
Summary: java.util.ConcurrentModificationException thrown when trying to push around 1...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 9.0.1
Hardware: PC All
: P2 normal (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-10-27 05:13 UTC by Thirunavukarasu
Modified: 2017-11-15 21:44 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thirunavukarasu 2017-10-27 05:13:53 UTC
Exception in thread "https-openssl-apr-8443-exec-178" java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
	at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
	at org.apache.coyote.http2.Http2UpgradeHandler.close(Http2UpgradeHandler.java:943)
	at org.apache.coyote.http2.Http2UpgradeHandler.closeConnection(Http2UpgradeHandler.java:483)
	at org.apache.coyote.http2.Stream.close(Stream.java:600)
	at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:85)
	at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)


ConcurrentModificationException is thrown when trying to push around 1000 resources from a servlet. Note, this issue is not consistently reproducible.

Steps to reproduce:
1. Access the demo.html (details provided below) over http and it works fine 
2. Keep the browser open and idle for few mins (i tried the next request after 5 mins)
3. Then try to invoke the demo servlet (details provided below) over https, you can now see the concurrent-modification-exception in tomcat log
  

Environment:
    Tomcat 9.0.1
    JDK 8 update 131
    tomcat-native-1.2.14-win32-bin (64bit)
    Firefox 56 (64bit)

Enabled HTTP/2 using the below configuration

 <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
            <Certificate certificateKeyFile="certs/ca.key"
                         certificateFile="certs/ca.crt"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>


Code snippet to reproduce the issue (DemoServlet doGet method)

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
		PushBuilder pb = request.newPushBuilder();
		for(int i=0;i<1000;i++) {
			String image = "image-"+i+".png";
			pb.path("/images/"+image);
			pb.push();
		}
		request.getRequestDispatcher("/demo.html").include(request, response);
	}

demo.html is a simple html file with a list of 1000 image tags.
Comment 1 Remy Maucherat 2017-11-03 14:48:51 UTC
Does making Http2UpgradeHandler.streams a ConcurrentHashMap help ?
Comment 2 Mark Thomas 2017-11-15 21:39:02 UTC
What appears to be happening is that while the resources are being pushed, something goes wrong and the connection is closed. The connection close process iterates over the existing streams and resets them. That is going to result in concurrent access to the streams HashMap. Switching to the concurrent implementation will fix this issue. It is possible that fixing this may expose other concurrency issues. If it does, please open new bugs for them.
Comment 3 Mark Thomas 2017-11-15 21:44:29 UTC
Fixed in:
- trunk for 9.0.2 onwards
- 8.5.x for 8.5.24 onwards