Created attachment 37742 [details]
java.io.IOException stack example

Hello,

Investigating a download issue on several copies of our application (the application is deployed on hundreds of tomcats, each application is independent, no clustering, just different client), we found out that we have thousands of errors such as the one attached.

We are using tomcat 9.0.41 and 9.0.43 but I think this issue is much older than that (we only keep 3 months of logs). The JDK version used is either OpenJDK 8 or 11 from AdoptOpenJDK (the JDK version doesn't change anything).

The installation of tomcat is from the official website and we are compiling APR automatically following the documentation on each tomcat release.

The connector configuration is:
====================================================================
    <Connector
        protocol="org.apache.coyote.http11.Http11AprProtocol"
        port="8080"
        maxThreads="500"
        acceptCount="2500"
        maxPostSize="10485760"
        maxKeepAliveRequests="-1"
        useSendfile="false"
        compression="on"
        compressibleMimeType="text/html,text/plain,text/xml,text/css,text/csv,text/rtf,text/richtext,text/javascript,application/x-javascript,application/javascript,application/ecmascript,application/rss+xml,application/xml,application/json,application/wasm,application/prs.hal-forms+json"
        compressionMinSize="250"
        connectionTimeout="20000" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
                useSendfile="false"
                compression="on"
                compressibleMimeType="text/html,text/plain,text/xml,text/css,text/csv,text/rtf,text/richtext,text/javascript,application/x-javascript,application/javascript,application/ecmascript,application/rss+xml,application/xml,application/json,application/wasm,application/prs.hal-forms+json"
                compressionMinSize="250" />
    </Connector>
====================================================================
The service is behind an haproxy which handle TLS.

Java parameters varies for memory but other than that, this is a typical setenv.sh:
====================================================================
#!/bin/sh
# Ansible managed
export JAVA_HOME="/srv/resources/jvm/jdk-11.0.10+9"
export CATALINA_HOME="/applications/tomcat/engine"

export compilation="-XX:-TieredCompilation"
export logs="-Xlog:gc*:file=${CATALINA_HOME}/logs/gc.log::filecount=10,filesize=20m:time,tid,level,tags"
export timeouts="-Dsun.net.client.defaultConnectTimeout=300000 -Dsun.net.client.defaultReadTimeout=7200000"
export FlightRecording="-XX:StartFlightRecording=disk=true,dumponexit=true,filename=${CATALINA_HOME}/logs"

export JAVA_OPTS="-Djava.awt.headless=true -Dcom.sun.security.enableAIAcaIssuers=true -XX:+UseG1GC -XX:MaxGCPauseMillis=800 -XX:+UseStringDeduplication -Djava.net.preferIPv4Stack=true -Xmx64g -XX:ReservedCodeCacheSize=2g -XX:MetaspaceSize=10g -XX:G1HeapRegionSize=32M ${compilation} ${logs} ${timeouts} ${FlightRecording}"

# JMX monitoring
export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/applications/tomcat/settings/jmxremote/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/applications/tomcat/settings/jmxremote/jmxremote.access -Djava.rmi.server.hostname=REDACTED"

# Activate Apache Portable Runtime (APR) based Native library for Tomcat
export LD_LIBRARY_PATH=/usr/local/apr/lib
====================================================================

Changing the connector from "org.apache.coyote.http11.Http11AprProtocol" to "org.apache.coyote.http11.Http11Nio2Protocol" completely eliminate the issue, we see 0 error. Reproducing the issue is complicated and we have no automatic way to trigger this yet.

The issue https://bz.apache.org/bugzilla/show_bug.cgi?id=62626 looks similar and we are indeed using imageio (when we use JDK 8, we are using jai-imageio-core-1.4.0, when we use JDK 11, we use the included imageio).

I'm sorry but I may not be able to help you further since were are switching everthing to nio2 now, but I still wanted to raise the issue so that you know that such problem still exist.