Bug 64771 - Windows CPU processor always running by a thread reading request body from https connection
Summary: Windows CPU processor always running by a thread reading request body from ht...
Status: NEEDINFO
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Util (show other bugs)
Version: 9.0.37
Hardware: PC All
: P2 major (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-26 17:19 UTC by Jerry Hou
Modified: 2021-02-23 13:13 UTC (History)
0 users



Attachments
Java Thread Dump (74.51 KB, text/plain)
2020-09-26 17:19 UTC, Jerry Hou
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jerry Hou 2020-09-26 17:19:24 UTC
Created attachment 37472 [details]
Java Thread Dump

The problem started with windows CPU always keep at 25%, but it should be around 2-7% is there is no request.

I use windows ProcessExplorer identified that one thread is reading request body from an https connection but there is no request at that time based on the audit logs, and the thread has been running more than 2 hours.

I use windows ProcessExplorer and jdk jstack identified the thread "0xd14" is always keep running even there is no request to Tomcat.


"boundedElastic-90" #434 daemon prio=5 os_prio=0 tid=0x0000000021055000 nid=0xd14 runnable [0x000000004383d000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:603)
	- locked <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)


"I/O dispatcher 3" #30 prio=5 os_prio=0 tid=0x00000000208d2800 nid=0x17e8 waiting for monitor entry [0x0000000024ace000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.tomcat.util.net.openssl.OpenSSLEngine.wrap(OpenSSLEngine.java:391)
	- waiting to lock <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine)
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
	at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:802)



I try to understand the source "OpenSSLEngine" based on the thread logs, it seems that the reading request body is not yet fully completed but the request connection is closed by the caller.

I need help to have a fix to prevent this kind of problem happen in Tomcat to prevent such problem happen in production.
Comment 1 Jerry Hou 2020-09-26 17:23:27 UTC
It is starting from 	at com.ncs.secureconnect.webflux.common.gateway.context.DefaultContext.bufferBody(DefaultContext.java:173)
Comment 2 mgrigorov 2020-09-28 05:25:11 UTC
Could you please provide the full stack traces for these two threads ?
Comment 3 Jerry Hou 2020-09-28 06:31:54 UTC
"boundedElastic-90" #434 daemon prio=5 os_prio=0 tid=0x0000000021055000 nid=0xd14 runnable [0x000000004383d000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:603)
	- locked <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
	at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:617)
	at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1228)
	at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1140)
	at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:780)
	at org.apache.coyote.http11.Http11InputBuffer.available(Http11InputBuffer.java:668)
	at org.apache.coyote.http11.Http11Processor.available(Http11Processor.java:1185)
	at org.apache.coyote.AbstractProcessor.isReadyForRead(AbstractProcessor.java:825)
	at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:574)
	at org.apache.coyote.Request.action(Request.java:432)
	at org.apache.catalina.connector.InputBuffer.isReady(InputBuffer.java:305)
	at org.apache.catalina.connector.CoyoteInputStream.isReady(CoyoteInputStream.java:201)
	at org.springframework.http.server.reactive.ServletServerHttpRequest$RequestBodyPublisher.checkOnDataAvailable(ServletServerHttpRequest.java:290)
	at org.springframework.http.server.reactive.AbstractListenerReadPublisher.changeToDemandState(AbstractListenerReadPublisher.java:222)
	at org.springframework.http.server.reactive.AbstractListenerReadPublisher.access$1000(AbstractListenerReadPublisher.java:48)
	at org.springframework.http.server.reactive.AbstractListenerReadPublisher$State$2.request(AbstractListenerReadPublisher.java:333)
	at org.springframework.http.server.reactive.AbstractListenerReadPublisher$ReadSubscription.request(AbstractListenerReadPublisher.java:260)
	at reactor.core.publisher.MonoReduce$ReduceSubscriber.onSubscribe(MonoReduce.java:78)
	at org.springframework.http.server.reactive.AbstractListenerReadPublisher$State$1.subscribe(AbstractListenerReadPublisher.java:301)
	at org.springframework.http.server.reactive.AbstractListenerReadPublisher.subscribe(AbstractListenerReadPublisher.java:105)
	at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:65)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4324)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4184)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4120)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4067)
	at com.bac.bacconnect.webflux.common.gateway.context.DefaultContext.bufferBody(DefaultContext.java:173)
	at com.bac.bacconnect.commons.filters.SecurityFilter.e(SecurityFilter.java:205)
	at com.bac.bacconnect.commons.filters.SecurityFilter.doFilterInternally(SecurityFilter.java:132)
	at com.bac.bacconnect.commons.filters.SecurityFilter.doFilter(SecurityFilter.java:106)
	at com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:42)
	at com.bac.bacconnect.igw.webflux.controller.Api.a(Api.java:102)
	at com.bac.bacconnect.igw.webflux.controller.Api$$Lambda$417/1031507663.accept(Unknown Source)
	at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247)
	at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132)
	at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153)
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
	at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:141)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
	at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92)
	at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
	at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:845)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
	at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmitScalar(FluxFlatMap.java:480)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:413)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
	at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:267)
	at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:137)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:171)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onComplete(FluxPeekFuseable.java:931)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:78)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2346)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
	at reactor.core.publisher.MonoNext$NextSubscriber.onComplete(MonoNext.java:96)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:838)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:600)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:580)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:457)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270)
	at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:289)
	at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:137)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:171)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onComplete(FluxDefaultIfEmpty.java:100)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:262)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1783)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:320)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:337)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
	at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:249)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	
	
	

"I/O dispatcher 3" #30 prio=5 os_prio=0 tid=0x00000000208d2800 nid=0x17e8 waiting for monitor entry [0x0000000024ace000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.tomcat.util.net.openssl.OpenSSLEngine.wrap(OpenSSLEngine.java:391)
	- waiting to lock <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine)
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
	at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:802)
	at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1268)
	at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764)
	at org.apache.tomcat.util.net.SocketWrapperBase.flushNonBlocking(SocketWrapperBase.java:735)
	at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:709)
	at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:564)
	at org.apache.coyote.http11.filters.IdentityOutputFilter.flush(IdentityOutputFilter.java:117)
	at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:217)
	at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1179)
	at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:399)
	at org.apache.coyote.Response.action(Response.java:209)
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)
	at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272)
	at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
	at org.springframework.http.server.reactive.ServletServerHttpResponse.flush(ServletServerHttpResponse.java:198)
	at org.springframework.http.server.reactive.ServletServerHttpResponse.access$500(ServletServerHttpResponse.java:50)
	at org.springframework.http.server.reactive.ServletServerHttpResponse$ResponseBodyFlushProcessor.flush(ServletServerHttpResponse.java:316)
	at org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$3.writeComplete(AbstractListenerWriteFlushProcessor.java:288)
	at org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$WriteResultSubscriber.onComplete(AbstractListenerWriteFlushProcessor.java:434)
	at org.springframework.http.server.reactive.WriteResultPublisher$State.publishComplete(WriteResultPublisher.java:256)
	at org.springframework.http.server.reactive.WriteResultPublisher.publishComplete(WriteResultPublisher.java:84)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.changeStateToComplete(AbstractListenerWriteProcessor.java:280)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.access$300(AbstractListenerWriteProcessor.java:46)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor$State$3.onWritePossible(AbstractListenerWriteProcessor.java:368)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.onWritePossible(AbstractListenerWriteProcessor.java:153)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.writeIfPossible(AbstractListenerWriteProcessor.java:293)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.changeStateToReceived(AbstractListenerWriteProcessor.java:272)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.access$400(AbstractListenerWriteProcessor.java:46)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor$State$2.onComplete(AbstractListenerWriteProcessor.java:359)
	at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.onComplete(AbstractListenerWriteProcessor.java:141)
	at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onComplete(ChannelSendOperator.java:231)
	at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onComplete(FluxContextStart.java:115)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:359)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211)
	at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onSubscribe(FluxTake.java:370)
	at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
	at org.springframework.http.server.reactive.ChannelSendOperator.subscribe(ChannelSendOperator.java:77)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:199)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
	at com.bac.bacconnect.igw.webflux.controller.Api.a(Api.java:113)
	at com.bac.bacconnect.igw.webflux.controller.Api$$Lambda$424/723410695.handle(Unknown Source)
	at com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:36)
	at com.bac.bacconnect.commons.common.filter.FilterChain.a(FilterChain.java:48)
	at com.bac.bacconnect.commons.common.filter.FilterChain$$Lambda$429/662427353.handle(Unknown Source)
	at com.bac.bacconnect.igw.services.proxy.DefaultProxyService.decrypt(DefaultProxyService.java:48)
	at com.bac.bacconnect.commons.filters.DecryptionFilter.doFilter(DecryptionFilter.java:31)
	at com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:42)
	at com.bac.bacconnect.commons.common.filter.FilterChain.a(FilterChain.java:48)
	at com.bac.bacconnect.commons.common.filter.FilterChain$$Lambda$429/662427353.handle(Unknown Source)
	at com.bac.bacconnect.commons.services.common.agent.async.AsyncClientAgent.a(AsyncClientAgent.java:220)
	at com.bac.bacconnect.commons.services.common.agent.async.AsyncClientAgent$$Lambda$441/412883409.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	at com.bac.bacconnect.commons.services.common.agent.async.d.a(AsyncResponseCallback.java:118)
	at com.bac.bacconnect.commons.services.common.agent.async.d.completed(AsyncResponseCallback.java:24)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Comment 4 Mark Thomas 2020-09-28 10:17:42 UTC
There is insufficient information in the stack trace to identify a bug in Tomcat.

I suspect that the application is retaining references to request/response objects beyond the completion of the response.

I'd recommend re-testing without the Tomcat Native library. If you can reproduce the issue with a pure Java stack it will be easier to debug.

What we really need is a *minimal* test case that reproduces this issue on a clean install of the latest 9.0.x release.
Comment 5 Mark Thomas 2020-12-01 09:34:48 UTC
Two months have passed and we still don't have a minimal WAR that reproduces this issue. I'll leave this open another month but without a test case to investigate this will get closed.
Comment 6 Mark Thomas 2021-01-04 16:30:01 UTC
Resolving as per previous comment.
Comment 7 Phil Clay 2021-01-20 19:57:51 UTC
I believe I have hit this same issue.

I reported a bug against Spring as https://github.com/spring-projects/spring-framework/issues/26407

I have created a standalone application that can be used to reproduce the problem.  The app is linked from the description of the spring issue.

There is definitely a case where OpenSSLEngine.unwrap gets into an infinite loop.  It would be great if someone familiar with OpenSSLEngine can take a look.  It is not clear if the problem is in tomcat or spring.

Can this tomcat issue be reopened?  Or should I file a new issue?
Comment 8 Remy Maucherat 2021-01-21 07:49:39 UTC
That doesn't look good, it seems the state of the destination buffer is changing concurrently (suddenly, no remaining bytes although the capacity is still there), which then breaks the algorithm. Maybe this loop should be safer, but it probably means there's missing sync somewhere else [given the report the threading could be funky, async does not allow to do *anything*]. It could be a good idea to add a safety check in unwrap, but then the data read is already corrupted and that request is not going to end well.

--- a/java/org/apache/tomcat/util/net/openssl/OpenSSLEngine.java
+++ b/java/org/apache/tomcat/util/net/openssl/OpenSSLEngine.java
@@ -600,6 +600,10 @@
             }
             if (capacity == 0) {
                 break;
+            } else if (idx == endOffset) {
+                // Avoid loop which could be caused by an unexpected state
+                // change of the destination buffers and report an error
+                throw new IllegalArgumentException("BAAAAAAAD");
             } else if (pendingApp == 0) {
                 pendingApp = pendingReadableBytesInSSL();
             }
Comment 9 Mark Thomas 2021-01-21 08:11:31 UTC
RE-opening this issue while I look at the test case...
Comment 10 Remy Maucherat 2021-01-21 09:21:28 UTC
(In reply to Mark Thomas from comment #9)
> RE-opening this issue while I look at the test case...

The use of non container threads doing ??? seems to be the root cause of the problems here. Also the behavior of available() could be a contributor, and you did change that for the better very recently. So you might not be able to reproduce.

OTOH, I would still like to add a ISE exit for the loop in OpenSSLEngine.unwrap, in case the state becomes inconsistent.
Comment 11 Mark Thomas 2021-01-21 17:02:28 UTC
The reproducer provided in the Spring issue uses Tomcat 9.0.41 and that has the available() fixes.

Bug 65001 might be a factor and that is a post 9.0.41 fix but since I can't reproduce the issue, I can't test that. I'll point the OP of the Spring issue to the snapshots to see if that helps.

My current thoughts are this is not a Tomcat issue as per my comment on the Spring issue.

I agree with your ISE suggestion to catch his state. Async code is easy to get wrong and we should be defensive where practical.
Comment 12 Mark Thomas 2021-01-28 14:10:53 UTC
Rémy has added the defensive code.

I'm moving this to NEEDINFO while the Spring team continue to investigate.
Comment 13 Mark Thomas 2021-02-23 13:13:13 UTC
As per the update to the spring issue (https://github.com/spring-projects/spring-framework/issues/26407) 9.0.x now contains a fix that may help with this. Currently waiting for user feedback.