Created attachment 37472 [details] Java Thread Dump The problem started with windows CPU always keep at 25%, but it should be around 2-7% is there is no request. I use windows ProcessExplorer identified that one thread is reading request body from an https connection but there is no request at that time based on the audit logs, and the thread has been running more than 2 hours. I use windows ProcessExplorer and jdk jstack identified the thread "0xd14" is always keep running even there is no request to Tomcat. "boundedElastic-90" #434 daemon prio=5 os_prio=0 tid=0x0000000021055000 nid=0xd14 runnable [0x000000004383d000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:603) - locked <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) "I/O dispatcher 3" #30 prio=5 os_prio=0 tid=0x00000000208d2800 nid=0x17e8 waiting for monitor entry [0x0000000024ace000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tomcat.util.net.openssl.OpenSSLEngine.wrap(OpenSSLEngine.java:391) - waiting to lock <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:802) I try to understand the source "OpenSSLEngine" based on the thread logs, it seems that the reading request body is not yet fully completed but the request connection is closed by the caller. I need help to have a fix to prevent this kind of problem happen in Tomcat to prevent such problem happen in production.
It is starting from at com.ncs.secureconnect.webflux.common.gateway.context.DefaultContext.bufferBody(DefaultContext.java:173)
Could you please provide the full stack traces for these two threads ?
"boundedElastic-90" #434 daemon prio=5 os_prio=0 tid=0x0000000021055000 nid=0xd14 runnable [0x000000004383d000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:603) - locked <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:617) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1228) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1140) at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:780) at org.apache.coyote.http11.Http11InputBuffer.available(Http11InputBuffer.java:668) at org.apache.coyote.http11.Http11Processor.available(Http11Processor.java:1185) at org.apache.coyote.AbstractProcessor.isReadyForRead(AbstractProcessor.java:825) at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:574) at org.apache.coyote.Request.action(Request.java:432) at org.apache.catalina.connector.InputBuffer.isReady(InputBuffer.java:305) at org.apache.catalina.connector.CoyoteInputStream.isReady(CoyoteInputStream.java:201) at org.springframework.http.server.reactive.ServletServerHttpRequest$RequestBodyPublisher.checkOnDataAvailable(ServletServerHttpRequest.java:290) at org.springframework.http.server.reactive.AbstractListenerReadPublisher.changeToDemandState(AbstractListenerReadPublisher.java:222) at org.springframework.http.server.reactive.AbstractListenerReadPublisher.access$1000(AbstractListenerReadPublisher.java:48) at org.springframework.http.server.reactive.AbstractListenerReadPublisher$State$2.request(AbstractListenerReadPublisher.java:333) at org.springframework.http.server.reactive.AbstractListenerReadPublisher$ReadSubscription.request(AbstractListenerReadPublisher.java:260) at reactor.core.publisher.MonoReduce$ReduceSubscriber.onSubscribe(MonoReduce.java:78) at org.springframework.http.server.reactive.AbstractListenerReadPublisher$State$1.subscribe(AbstractListenerReadPublisher.java:301) at org.springframework.http.server.reactive.AbstractListenerReadPublisher.subscribe(AbstractListenerReadPublisher.java:105) at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:65) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.Mono.subscribeWith(Mono.java:4324) at reactor.core.publisher.Mono.subscribe(Mono.java:4184) at reactor.core.publisher.Mono.subscribe(Mono.java:4120) at reactor.core.publisher.Mono.subscribe(Mono.java:4067) at com.bac.bacconnect.webflux.common.gateway.context.DefaultContext.bufferBody(DefaultContext.java:173) at com.bac.bacconnect.commons.filters.SecurityFilter.e(SecurityFilter.java:205) at com.bac.bacconnect.commons.filters.SecurityFilter.doFilterInternally(SecurityFilter.java:132) at com.bac.bacconnect.commons.filters.SecurityFilter.doFilter(SecurityFilter.java:106) at com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:42) at com.bac.bacconnect.igw.webflux.controller.Api.a(Api.java:102) at com.bac.bacconnect.igw.webflux.controller.Api$$Lambda$417/1031507663.accept(Unknown Source) at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132) at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145) at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274) at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145) at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:141) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107) at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92) at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274) at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107) at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:845) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107) at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmitScalar(FluxFlatMap.java:480) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:413) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:267) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:137) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:171) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75) at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160) at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onComplete(FluxPeekFuseable.java:931) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:78) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2346) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026) at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75) at reactor.core.publisher.MonoNext$NextSubscriber.onComplete(MonoNext.java:96) at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160) at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:838) at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:600) at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:580) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:457) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270) at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:289) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:137) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:171) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onComplete(FluxDefaultIfEmpty.java:100) at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160) at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:262) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1783) at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:320) at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:337) at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192) at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:249) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "I/O dispatcher 3" #30 prio=5 os_prio=0 tid=0x00000000208d2800 nid=0x17e8 waiting for monitor entry [0x0000000024ace000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tomcat.util.net.openssl.OpenSSLEngine.wrap(OpenSSLEngine.java:391) - waiting to lock <0x0000000608ad2bf0> (a org.apache.tomcat.util.net.openssl.OpenSSLEngine) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:802) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1268) at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764) at org.apache.tomcat.util.net.SocketWrapperBase.flushNonBlocking(SocketWrapperBase.java:735) at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:709) at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:564) at org.apache.coyote.http11.filters.IdentityOutputFilter.flush(IdentityOutputFilter.java:117) at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:217) at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1179) at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:399) at org.apache.coyote.Response.action(Response.java:209) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) at org.springframework.http.server.reactive.ServletServerHttpResponse.flush(ServletServerHttpResponse.java:198) at org.springframework.http.server.reactive.ServletServerHttpResponse.access$500(ServletServerHttpResponse.java:50) at org.springframework.http.server.reactive.ServletServerHttpResponse$ResponseBodyFlushProcessor.flush(ServletServerHttpResponse.java:316) at org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$3.writeComplete(AbstractListenerWriteFlushProcessor.java:288) at org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$WriteResultSubscriber.onComplete(AbstractListenerWriteFlushProcessor.java:434) at org.springframework.http.server.reactive.WriteResultPublisher$State.publishComplete(WriteResultPublisher.java:256) at org.springframework.http.server.reactive.WriteResultPublisher.publishComplete(WriteResultPublisher.java:84) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.changeStateToComplete(AbstractListenerWriteProcessor.java:280) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.access$300(AbstractListenerWriteProcessor.java:46) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor$State$3.onWritePossible(AbstractListenerWriteProcessor.java:368) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.onWritePossible(AbstractListenerWriteProcessor.java:153) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.writeIfPossible(AbstractListenerWriteProcessor.java:293) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.changeStateToReceived(AbstractListenerWriteProcessor.java:272) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.access$400(AbstractListenerWriteProcessor.java:46) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor$State$2.onComplete(AbstractListenerWriteProcessor.java:359) at org.springframework.http.server.reactive.AbstractListenerWriteProcessor.onComplete(AbstractListenerWriteProcessor.java:141) at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onComplete(ChannelSendOperator.java:231) at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onComplete(FluxContextStart.java:115) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:359) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onSubscribe(FluxTake.java:370) at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70) at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) at org.springframework.http.server.reactive.ChannelSendOperator.subscribe(ChannelSendOperator.java:77) at reactor.core.publisher.Mono.subscribe(Mono.java:4213) at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:199) at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at com.bac.bacconnect.igw.webflux.controller.Api.a(Api.java:113) at com.bac.bacconnect.igw.webflux.controller.Api$$Lambda$424/723410695.handle(Unknown Source) at com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:36) at com.bac.bacconnect.commons.common.filter.FilterChain.a(FilterChain.java:48) at com.bac.bacconnect.commons.common.filter.FilterChain$$Lambda$429/662427353.handle(Unknown Source) at com.bac.bacconnect.igw.services.proxy.DefaultProxyService.decrypt(DefaultProxyService.java:48) at com.bac.bacconnect.commons.filters.DecryptionFilter.doFilter(DecryptionFilter.java:31) at com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:42) at com.bac.bacconnect.commons.common.filter.FilterChain.a(FilterChain.java:48) at com.bac.bacconnect.commons.common.filter.FilterChain$$Lambda$429/662427353.handle(Unknown Source) at com.bac.bacconnect.commons.services.common.agent.async.AsyncClientAgent.a(AsyncClientAgent.java:220) at com.bac.bacconnect.commons.services.common.agent.async.AsyncClientAgent$$Lambda$441/412883409.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at com.bac.bacconnect.commons.services.common.agent.async.d.a(AsyncResponseCallback.java:118) at com.bac.bacconnect.commons.services.common.agent.async.d.completed(AsyncResponseCallback.java:24) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.lang.Thread.run(Thread.java:748)
There is insufficient information in the stack trace to identify a bug in Tomcat. I suspect that the application is retaining references to request/response objects beyond the completion of the response. I'd recommend re-testing without the Tomcat Native library. If you can reproduce the issue with a pure Java stack it will be easier to debug. What we really need is a *minimal* test case that reproduces this issue on a clean install of the latest 9.0.x release.
Two months have passed and we still don't have a minimal WAR that reproduces this issue. I'll leave this open another month but without a test case to investigate this will get closed.
Resolving as per previous comment.
I believe I have hit this same issue. I reported a bug against Spring as https://github.com/spring-projects/spring-framework/issues/26407 I have created a standalone application that can be used to reproduce the problem. The app is linked from the description of the spring issue. There is definitely a case where OpenSSLEngine.unwrap gets into an infinite loop. It would be great if someone familiar with OpenSSLEngine can take a look. It is not clear if the problem is in tomcat or spring. Can this tomcat issue be reopened? Or should I file a new issue?
That doesn't look good, it seems the state of the destination buffer is changing concurrently (suddenly, no remaining bytes although the capacity is still there), which then breaks the algorithm. Maybe this loop should be safer, but it probably means there's missing sync somewhere else [given the report the threading could be funky, async does not allow to do *anything*]. It could be a good idea to add a safety check in unwrap, but then the data read is already corrupted and that request is not going to end well. --- a/java/org/apache/tomcat/util/net/openssl/OpenSSLEngine.java +++ b/java/org/apache/tomcat/util/net/openssl/OpenSSLEngine.java @@ -600,6 +600,10 @@ } if (capacity == 0) { break; + } else if (idx == endOffset) { + // Avoid loop which could be caused by an unexpected state + // change of the destination buffers and report an error + throw new IllegalArgumentException("BAAAAAAAD"); } else if (pendingApp == 0) { pendingApp = pendingReadableBytesInSSL(); }
RE-opening this issue while I look at the test case...
(In reply to Mark Thomas from comment #9) > RE-opening this issue while I look at the test case... The use of non container threads doing ??? seems to be the root cause of the problems here. Also the behavior of available() could be a contributor, and you did change that for the better very recently. So you might not be able to reproduce. OTOH, I would still like to add a ISE exit for the loop in OpenSSLEngine.unwrap, in case the state becomes inconsistent.
The reproducer provided in the Spring issue uses Tomcat 9.0.41 and that has the available() fixes. Bug 65001 might be a factor and that is a post 9.0.41 fix but since I can't reproduce the issue, I can't test that. I'll point the OP of the Spring issue to the snapshots to see if that helps. My current thoughts are this is not a Tomcat issue as per my comment on the Spring issue. I agree with your ISE suggestion to catch his state. Async code is easy to get wrong and we should be defensive where practical.
Rémy has added the defensive code. I'm moving this to NEEDINFO while the Spring team continue to investigate.
As per the update to the spring issue (https://github.com/spring-projects/spring-framework/issues/26407) 9.0.x now contains a fix that may help with this. Currently waiting for user feedback.
A further issue was identified and fixed. Awaiting user feedback.
OP confirms that the issue is fixed.