Exception in thread "https-openssl-apr-8443-exec-178" java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) at java.util.HashMap$ValueIterator.next(HashMap.java:1466) at org.apache.coyote.http2.Http2UpgradeHandler.close(Http2UpgradeHandler.java:943) at org.apache.coyote.http2.Http2UpgradeHandler.closeConnection(Http2UpgradeHandler.java:483) at org.apache.coyote.http2.Stream.close(Stream.java:600) at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:85) at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) ConcurrentModificationException is thrown when trying to push around 1000 resources from a servlet. Note, this issue is not consistently reproducible. Steps to reproduce: 1. Access the demo.html (details provided below) over http and it works fine 2. Keep the browser open and idle for few mins (i tried the next request after 5 mins) 3. Then try to invoke the demo servlet (details provided below) over https, you can now see the concurrent-modification-exception in tomcat log Environment: Tomcat 9.0.1 JDK 8 update 131 tomcat-native-1.2.14-win32-bin (64bit) Firefox 56 (64bit) Enabled HTTP/2 using the below configuration <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" maxThreads="150" SSLEnabled="true" > <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" /> <SSLHostConfig> <Certificate certificateKeyFile="certs/ca.key" certificateFile="certs/ca.crt" type="RSA" /> </SSLHostConfig> </Connector> Code snippet to reproduce the issue (DemoServlet doGet method) protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { PushBuilder pb = request.newPushBuilder(); for(int i=0;i<1000;i++) { String image = "image-"+i+".png"; pb.path("/images/"+image); pb.push(); } request.getRequestDispatcher("/demo.html").include(request, response); } demo.html is a simple html file with a list of 1000 image tags.
Does making Http2UpgradeHandler.streams a ConcurrentHashMap help ?
What appears to be happening is that while the resources are being pushed, something goes wrong and the connection is closed. The connection close process iterates over the existing streams and resets them. That is going to result in concurrent access to the streams HashMap. Switching to the concurrent implementation will fix this issue. It is possible that fixing this may expose other concurrency issues. If it does, please open new bugs for them.
Fixed in: - trunk for 9.0.2 onwards - 8.5.x for 8.5.24 onwards