The issue looks like concurrent modification of Http2UpgradeHandler.streams to me. I think pruning needs to be performed while holding the priorityTreeLock but I want to try and recreate this to confirm that theory before looking at a fix.

Fixed in:
- 10.0.x for 10.0.3 onwards
- 9.0.x for 9.0.44 onwards
- 8.5.x for 8.5.63 onwards

7.0.x is not affected.

I have now tested the fix for this issue using SNAPSHOT tomcat-9.0-20210208.120325-3060.

I am afraid that the fix does not solve the issue. I still get a null pointer exception at the same place in the code (however other row number 1252).

>08-Feb-2021 14:07:10.564 FINE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams Connection [1] Starting pruning of old streams. Limit is [500] and there are currently [509] streams.
>08-Feb-2021 14:07:10.564 FINE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams Connection [1] Pruned completed stream [1]
>08-Feb-2021 14:07:10.564 FINE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams Connection [1] Pruned completed stream [3]
>08-Feb-2021 14:07:10.564 FINE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams Connection [1] Pruned completed stream [5]
>08-Feb-2021 14:07:10.564 FINE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams Connection [1] Pruned completed stream [7]
>08-Feb-2021 14:07:10.564 FINE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams Connection [1] Pruned completed stream [5]
>08-Feb-2021 14:07:10.565 SEVERE [https-openssl-apr-0.0.0.0-8443-exec-34] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error reading request, ignored
>	java.lang.NullPointerException
>		at org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams(Http2UpgradeHandler.java:1252)
>		at org.apache.coyote.http2.Http2UpgradeHandler.createRemoteStream(Http2UpgradeHandler.java:1129)
>		at org.apache.coyote.http2.Http2UpgradeHandler.headersStart(Http2UpgradeHandler.java:1512)
>		at org.apache.coyote.http2.Http2Parser.readHeadersFrame(Http2Parser.java:225)
>		at org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:99)
>		at org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:71)
>		at org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:339)
>		at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:60)
>		at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
>		at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
>		at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:1991)
>		at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
>		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>		at java.lang.Thread.run(Thread.java:748)


For testing purposes I changed the source code in Http2UpgradeHandler line 1252. 

From 
> while (toClose > 0 && parent.getIdAsInt() > 0 && parent.getIdAsInt() < stream.getIdAsInt() && parent.getChildStreams().isEmpty()) {
To
>  while (toClose > 0 && parent != null && parent.getIdAsInt() > 0 && parent.getIdAsInt() < stream.getIdAsInt() && parent.getChildStreams().isEmpty()) {
In order to avoid the NPE. This fix solved the issue. At least no NPE was thrown and no problem with loading resources could be seen in the web browsers. 
I hope this can help you pin point the root cause.

That patch looks to be tackling the symptom rather than the root cause. My concern is that the root cause may trigger other errors.

What would really help here is a test case (ideally a simple WAR) that demonstrates the issue.

(In reply to Mark Thomas from comment #4)
> That patch looks to be tackling the symptom rather than the root cause. My
> concern is that the root cause may trigger other errors.
> 
> What would really help here is a test case (ideally a simple WAR) that
> demonstrates the issue.

I understand. I will try to create a test case that you can use.

Moving to NEEDINFO as we need a test case for this.

Woot! I managed to trigger this locally just with Tomcat's default home page. Investigating...

Found it. The priority hierarchy was being corrupted when a closed stream was replaced with a RecycledStream instance.

Fixed in:
- 10.0.x for 10.0.3 onwards
- 9.0.x for 9.0.44 onwards
- 8.5.x for 8.5.63 onwards

7.0.x is not affected.

Still seen in 9.0.55:

 Exception in thread "https-jsse-nio-8443-exec-6" java.lang.NullPointerException
[INFO] [talledLocalContainer] 	at org.apache.coyote.http2.Http2UpgradeHandler.pruneClosedStreams(Http2UpgradeHandler.java:1293)
[INFO] [talledLocalContainer] 	at org.apache.coyote.http2.Http2UpgradeHandler.createRemoteStream(Http2UpgradeHandler.java:1170)
[INFO] [talledLocalContainer] 	at org.apache.coyote.http2.Http2UpgradeHandler.headersStart(Http2UpgradeHandler.java:1596)
[INFO] [talledLocalContainer] 	at org.apache.coyote.http2.Http2AsyncUpgradeHandler.headersStart(Http2AsyncUpgradeHandler.java:41)
[INFO] [talledLocalContainer] 	at org.apache.coyote.http2.Http2Parser.readHeadersFrame(Http2Parser.java:226)

Thanks for the report.

I've taken another look at the code and I have found what I believe is the remaining root cause of the NPE.

Fixed in:
- 10.1.x for 10.1.0-M16 onwards
- 10.0.x for 10.0.22 onwards
- 9.0.x for 9.0.64 onwards
- 8.5.x for 8.5.80 onwards