Hi. When I upload a big file per http2 using the http upload methods. A stream error occurs and the connection is severed from sending the file. however if i turn off http2 the file is sent normally. The Error occurs when I upload files larger than 1mb (But its not acurrate) using angular 8 by JSON REST post (observable), if i use Postman and send the same file, the error not happen. As a workaround, I changed the application.properties setting to: server.http2.enabled=false But I would like to use http2. This is my environment: Spring Boot 2.2.1.RELEASE, Tomcat native version of Spring Boot (Apache Tomcat/9.0.27) And Java 11 Oracle: java 11.0.5 2019-10-15 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.5+10-LTS) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode) Here is a sample code of use: https://github.com/darckyn/test-http2 And here is the discution about this in Spring Boot GitHub: https://github.com/spring-projects/spring-boot/issues/18806 Thx StackTrace: org.apache.catalina.connector.ClientAbortException: org.apache.coyote.CloseNowException: Connection [3], Stream [1], This stream is not writable at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1153) ~[jackson-core-2.10.0.jar:2.10.0] at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:923) ~[jackson-databind-2.10.0.jar:2.10.0] at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287) ~[spring-web-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) ~[spring-web-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:295) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:226) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82) ~[spring-web-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:124) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.1.RELEASE.jar:5.2.1.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:362) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:72) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] Caused by: org.apache.coyote.CloseNowException: Connection [3], Stream [1], This stream is not writable at org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:843) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:886) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1009) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.Http2OutputBuffer.flush(Http2OutputBuffer.java:77) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.http2.StreamProcessor.flush(StreamProcessor.java:212) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:395) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.coyote.Response.action(Response.java:209) ~[tomcat-embed-core-9.0.27.jar:9.0.27] at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305) ~[tomcat-embed-core-9.0.27.jar:9.0.27] ... 41 common frames omitted
That this is client dependent suggests the client might be tripping over the abusive behaviour detection added in 9.0.23. You can try the following settings in the UpgradeProtocol in server.xml: overheadContinuationThreshold="0" overheadDataThreshold="0" overheadWindowUpdateThreshold="0" If that fixes the issue then you probably want to look at what the client is doing. http2 debug logging in Tomcat is one of the simplest ways. Alternatively, provide a simple test case that demonstrates the issue that we can use to reproduce it.
Hello. Here is a sample code of use: https://github.com/darckyn/test-http2 Just send a file (the file is on this sample) to this spring boot code on. (build and start) This "abusive behaviour detection" is only enable in http-2 and when I upload large files by http code. Please, can you help me? How can I enable these settings in spring boot? overheadContinuationThreshold="0" overheadDataThreshold="0" overheadWindowUpdateThreshold="0" thx.
Your response does not give any additional info, so please do not change the bug status.
Rodrigo, you can customise the Http2Protocol by adding the following bean to your Spring Boot application: @Bean public TomcatConnectorCustomizer http2ProtocolCustomizer() { return (connector) -> { for (UpgradeProtocol upgradeProtocol: connector.findUpgradeProtocols()) { if (upgradeProtocol instanceof Http2Protocol) { Http2Protocol http2Protocol = (Http2Protocol)upgradeProtocol; http2Protocol.setOverheadContinuationThreshold(0); http2Protocol.setOverheadDataThreshold(0); http2Protocol.setOverheadWindowUpdateThreshold(0); } } }; } With this bean in place in your sample application, I'm no longer able to reproduce the problem with curl 7.54.0 on macOS. With the hint from Mark that the problem may be client-specific, I also tried with curl 7.67.0 and the problem does not occur, even without the customization of the Http2Protocol.
Thanks Andy. I appreciate the help with this. All the evidence points towards older versions of curl using small data and/or continuation and/or window update frames which Tomcat detects as abusive and therefore closes the connection. The solution is to use a newer client or disable / reduce the overhead protection if you need to use an older client. Marking as INVALID since Tomcat is behaving as intended.
(In reply to Andy Wilkinson from comment #4) > Rodrigo, you can customise the Http2Protocol by adding the following bean to > your Spring Boot application: > > @Bean > public TomcatConnectorCustomizer http2ProtocolCustomizer() { > return (connector) -> { > for (UpgradeProtocol upgradeProtocol: connector.findUpgradeProtocols()) { > if (upgradeProtocol instanceof Http2Protocol) { > Http2Protocol http2Protocol = (Http2Protocol)upgradeProtocol; > http2Protocol.setOverheadContinuationThreshold(0); > http2Protocol.setOverheadDataThreshold(0); > http2Protocol.setOverheadWindowUpdateThreshold(0); > } > } > }; > } > > With this bean in place in your sample application, I'm no longer able to > reproduce the problem with curl 7.54.0 on macOS. With the hint from Mark > that the problem may be client-specific, I also tried with curl 7.67.0 and > the problem does not occur, even without the customization of the > Http2Protocol. Thx for all the help. The error no longer occurs after deploying bean. I am using the observable (subscribe) of angular to send the file, here is the frameworks: https://www.primefaces.org/primeng/#/fileupload https://angular.io/guide/http
Disabling overhead protection works, thank you! But, are there downsides to doing so? You recommend using a newer client, but the clients I'm using that run into this problem are the current versions of Firefox and Chrome.