> -Djava.security.policy=${CATALINA_BASE}/conf/catalina.policy

The above value is just wrong. The value has to start with '=' (see Java documentation for "java.security.policy" for the meaning).

See how catalina.sh sets it, e.g.:

-Djava.security.policy=="$CATALINA_BASE"/conf/catalina.policy

Note how '=' character is doubled.

Hi Konstantin,
I have tried your recommendation but it didn't resolve the issue.

double equal signs
------------------
export JAVA_OPTS="-Djava.awt.headless=true -Djava.security.manager -Djava.security.policy==${CATALINA_BASE}/conf/catalina.policy -XX:+HeapDumpOnOutOfMemoryError -Xms512M -Xmx1024M -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=2048m -Djava.security.egd=file:/dev/./urandom"

double equal signs and double quotes around ${CATALINA_BASE}
------------------------------------------------------------
export JAVA_OPTS="-Djava.awt.headless=true -Djava.security.manager -Djava.security.policy==\"${CATALINA_BASE}\"/conf/catalina.policy -XX:+HeapDumpOnOutOfMemoryError -Xms512M -Xmx1024M -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=2048m -Djava.security.egd=file:/dev/./urandom"

I don't think there is a problem with this setting because this setting worked with Nio and Nio2 before 8.5.70.

The difference between '=' and '==' is that '=' means add this policy to the ones defined in the security properties files whereas '==' means only use this file.

The issue is valid. I can reproduce it. Working on a fix now.

(In reply to Mark Thomas from comment #3)
> The issue is valid. I can reproduce it. Working on a fix now.

OK.

Noting that there is nothing but Tomcat and Java code in stacktraces. I expect those to have full permissions, like Tomcat own code does. Is something unexpected with those threads?


> java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.org.apache.tomcat.util.net")

Such class loading errors can be omitted by preloading the affected classes. For known issues it is done by org.apache.catalina.security.SecurityClassLoad class.

This can also be done with "classesToInitialize" attribute of a org.apache.catalina.core.JreMemoryLeakPreventionListener configured in server.xml file.

> java.security.AccessControlException: access denied ("java.net.SocketPermission" "10.13.149.73:54048" "accept,resolve")

No easy solution here. BTW this thread has only JRE code in stacktrace.

This looks to be related to the NIO2 completion handlers. Secure connections do a handshake first so the main request processing is on a completion handler thread. These don't appear to have any security context associated with them although I need to dig into this some more.

The non-secure threads start processing on a standard executor thread - hence why they don't see this issue.

My concern at this point is that we could end up in a position of having to pre-load a much larger set of classes.

It is worth noting that the SecurityManager is deprecated in newer versions of Java and that support for running Tomcat under a SecurityManager is likely to be removed in the (distant) future.

(In reply to Mark Thomas from comment #5)
> This looks to be related to the NIO2 completion handlers. Secure connections
> do a handshake first so the main request processing is on a completion
> handler thread. These don't appear to have any security context associated
> with them although I need to dig into this some more.
> 
> The non-secure threads start processing on a standard executor thread -
> hence why they don't see this issue.
> 
> My concern at this point is that we could end up in a position of having to
> pre-load a much larger set of classes.
> 
> It is worth noting that the SecurityManager is deprecated in newer versions
> of Java and that support for running Tomcat under a SecurityManager is
> likely to be removed in the (distant) future.

I couldn't immediately get it working as well. Instead, we could document that the security manager is not supported with NIO2, for now at least.

I've found a workaround. The short version is when running under a security manager, have the handshake completion handlers for NIO2 always dispatch to a container thread. Slower, but it works. Commit will follow shortly (I just want to run the tests first).

Fixed in:
- 10.1.x for 10.1.0-M8 onwards
- 10.0.x for 10.0.14 onwards
- 9.0.x for 9.0.56 onwards
- 8.5.x for 8.5.74 onwards

Sounds great. Appreciate the quick turn around.  Don't mind a slower option for now. Looking forward to test this in the next release

Please reopen this ticket. We installed latest Tomcat 9.0.56 still NIO2 issue isn't fix yet. 

NIO2 issues that go away when you specify NIO as the connector type. Or disable server manager

2021-12-08T15:55:29.610Z SEVERE org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed Socket accept failed
java.security.AccessControlException: access denied ("java.net.SocketPermission" "10.81.154.81:58774" "accept,resolve") 
  
2021-12-08T15:55:30.428Z SEVERE org.apache.tomcat.util.net.Nio2Endpoint setSocketOptions Error setting socket options
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.org.apache.tomcat.util.net") 
  
Please look into it

It appears that there is an OS component to this.

This works on MacOS but not on Linux (same Java vendor and version).

We need NIO 2 issue to be fixed for both RHEL and Windows version

Can we expedite this and release Tomcat 9.0.56 and 8.5.74 binaries for RHEL version As soon as possible ?

This can't be fixed in Tomcat. Even when the security policy has an entry that is "Allow everything to do everything" the NIO2 code still triggers security exceptions when running under a SecurityManager. You could try raising a bug with your JRE vendor but, give the deprecation of the SecurityManager, I suspect that fixing this will not be a priority.

You have two options to solve this with Tomcat:
- turn off the security manager
- switch to NIO

The security manager has been deprecated in Java 17 and will eventually be removed. Now might be a good time to start to think about migrating away from using the SecurityManager.

There is very little between NIO and NIO2 in terms of performance. While individual benchmarks will demonstrate one is better than the other, the difference is typically small and in real world usage the different is smaller still.

Do you want to keep the previous patch just in case ? It seems like it did something on your platform, then.

(In reply to Remy Maucherat from comment #14)
> Do you want to keep the previous patch just in case ? It seems like it did
> something on your platform, then.

I don't think so. I think there was something wrong with my testing of that patch.

Hi,
Why woould NIO2 and SecurityManager works with 8.5.69 and earlier 8.5.x versions and 9.0.50 and earlier 9.5.x versions using the same platform RHEL 7 or Windows and the same Java version?
And when upgrade to 8.5.70/9.0.52 or later version on the same platform this wouldn't work?

If this is not fixable on RHEL7 and Windows will there be a statement that NIO2 and Security Manager is not supported from later versions on these platforms?

Good question. The SecurityManager is so rarely used I had assumed you found this issue switching to NIO2 rather than upgrading between versions.

Time for a binary search to find the commit where this started...

Found it:
https://github.com/apache/tomcat/commit/dae37f4421

That makes sense since it changes the ThreadPoolExecutor implementation. See also bug 65454

I don't know what the root cause is yet so I am re-opening this while I investigate further as there is a possibility that this is fixable.

I now have a clearer picture of what is going on.

The fix for bug 65454 included an unintended change. The pre-starting of the core thread pool was removed. I'll restore that shortly. It also made a bug that was already present more obvious.

When running under a security manager, the NIO2 acceptor uses an InnocuousThread to execute the completion handler for the acceptor. By design, the access control context associated with InnocuousThread instances is empty. When the accepted connection is passed to the executor, if a worker thread needs to be created, that thread inherits the access control context of the calling thread. When the calling thread is an InnocuousThread that access control context is empty so the work thread does not have the permissions it needs. That leads to security exceptions and failed requests.

If the core threads are pre-started they have the correct permissions and depending on timing new threads that are started will either be started from those threads and have the correct permissions or from the Acceptor and have the wrong permissions.

The next step is to see if I can find a way to avoid this issue.

I have a solution and it has been committed for all current branches.

Fixed in:
- 10.1.x for 10.1.0-M9 onwards
- 10.0.x for 10.0.15 onwards
- 9.0.x for 9.0.57 onwards
- 8.5.x for 8.5.74 onwards

Any testing in advance of those releases would be helpful. I can provide test builds if that helps.

Thanks Mark. This sounds promising. For test in advance, where can we pull this image?

Which version do you need and I'll create a test build for you.

How about 8.5.74 and 9.0.57 on RHEL 7 and Windows?

8.5.x:
https://people.apache.org/~markt/dev/v8.5.72-4795df9/

9.0.x:
https://people.apache.org/~markt/dev/v9.0.57-115334b/

These are not official releases. They are development builds solely to test whether this bug has been fully fixed.

Thanks. Will work on it.

Hi Mark, I have carried out a number of connectivity tests.  Result as follow.

Tomcat 8.5.72-4795df9 and 9.0.57-115334b with Oracle Java JDK 9,11,13 (specifically 9.0.4, 11.0.2, 13.0.2) on RHEL 7 and Windows
HTTP - pass
HTTPS NIO - pass
HTTPS NIO2 - pass.

Tomcat 8.5.72-4795df9 and 9.0.57-115334b with Oracle Java JRE 8 (1.8.0.212b31 or jdk_1.8.0.251) on RHEL 7 and Windows
HTTP - failed
HTTPS NIO - failed
HTTPS NIO2 - failed
See errors from logs below.

Test configuration is the one from the image.
I only added this section in server.xml
   <!-- HTTP Connector -->
    <Connector
      port="8081"
      connectionTimeout="20000"
      redirectPort="8082"
      protocol="HTTP/1.1"
    />
 <!-- HTTPS Connector -->
    <Connector
      port="8082"
      protocol="org.apache.coyote.http11.Http11Nio2Protocol"
      maxThreads="150"
      SSLEnabled="true"
      >
      <SSLHostConfig>
        <Certificate
          certificateKeystoreFile="tomcat.jks"
          certificateKeystoreType="JKS"
          certificateKeystorePassword="XXXXXXXX"
          type="RSA"
        />
      </SSLHostConfig>
    </Connector>

  <!-- HTTPS Connector -->
    <Connector
      port="8083"
      protocol="org.apache.coyote.http11.Http11NioProtocol"
      maxThreads="150"
      SSLEnabled="true"
      >
      <SSLHostConfig>
        <Certificate
          certificateKeystoreFile="tomcat.jks"
          certificateKeystoreType="JKS"
          certificateKeystorePassword="XXXXXXXX"
          type="RSA"
        />
      </SSLHostConfig>
    </Connector>


Connection test was done using curl.
For example
curl -k -i https://<fqdn>:8081 (for HTTP)
curl -k -i https://<fqdn>:8082 (for HTTPS NIO2)
curl -k -i https://<fqdn>:8083 (for HTTPS NIO)

All errors are related to java.lang.NoSuchMethodError .
Was wonder if the test image is not compiled to support Java 8?
Going forward we will need Java 8 support.


Error logs
-----------
Tomcat 8.5.72-4795df9

HTTP

11-Dec-2021 21:33:11.804 SEVERE [http-nio-8081-exec-1] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error reading request, ignored
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
                at org.apache.coyote.http11.Http11InputBuffer.init(Http11InputBuffer.java:763)
                at org.apache.coyote.http11.Http11Processor.setSocketWrapper(Http11Processor.java:740)
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:499)
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:849)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1677)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)
11-Dec-2021 21:33:11.805 SEVERE [http-nio-8081-exec-1] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
                at org.apache.coyote.http11.Http11InputBuffer.recycle(Http11InputBuffer.java:284)
                at org.apache.coyote.http11.Http11Processor.recycle(Http11Processor.java:1637)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:1039)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:1007)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1677)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)

HTTPS NIO
11-Dec-2021 21:34:22.782 SEVERE [https-jsse-nio-8083-exec-1] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
                at org.apache.tomcat.util.net.TLSClientHelloExtractor.<init>(TLSClientHelloExtractor.java:189)
                at org.apache.tomcat.util.net.SecureNioChannel.processSNI(SecureNioChannel.java:289)
                at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:179)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1652)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)

HTTPS NIO2
11-Dec-2021 21:34:50.917 SEVERE [https-jsse-nio2-8082-Acceptor-0] org.apache.tomcat.util.net.Nio2Endpoint.setSocketOptions
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
                at org.apache.tomcat.util.net.SocketBufferHandler.reset(SocketBufferHandler.java:146)
                at org.apache.tomcat.util.net.Nio2Channel.reset(Nio2Channel.java:58)
                at org.apache.tomcat.util.net.SecureNio2Channel.reset(SecureNio2Channel.java:134)
                at org.apache.tomcat.util.net.Nio2Endpoint.setSocketOptions(Nio2Endpoint.java:315)
                at org.apache.tomcat.util.net.Nio2Endpoint$Acceptor.run(Nio2Endpoint.java:439)
                at java.lang.Thread.run(Thread.java:748)


Tomcat 9.0.57-115334b
HTTP
11-Dec-2021 21:40:59.233 SEVERE [http-nio-8081-Acceptor] org.apache.tomcat.util.net.NioEndpoint.setSocketOptions Error setting socket options
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
                at org.apache.tomcat.util.net.SocketBufferHandler.reset(SocketBufferHandler.java:213)
                at org.apache.tomcat.util.net.NioChannel.reset(NioChannel.java:59)
                at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:489)
                at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:78)
                at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:149)
                at java.lang.Thread.run(Thread.java:748)


HTTPS NIO

11-Dec-2021 21:41:20.781 SEVERE [https-jsse-nio-8083-Acceptor] org.apache.tomcat.util.net.NioEndpoint.setSocketOptions Error setting socket options
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
                at org.apache.tomcat.util.net.SocketBufferHandler.reset(SocketBufferHandler.java:213)
                at org.apache.tomcat.util.net.NioChannel.reset(NioChannel.java:59)
                at org.apache.tomcat.util.net.SecureNioChannel.reset(SecureNioChannel.java:93)
                at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:489)
                at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:78)
                at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:149)
                at java.lang.Thread.run(Thread.java:748)

HTTPS NIO2
11-Dec-2021 21:41:42.097 SEVERE [https-jsse-nio2-8082-exec-2] org.apache.tomcat.util.net.Nio2Endpoint.setSocketOptions Error setting socket options
        java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
                at org.apache.tomcat.util.net.SocketBufferHandler.reset(SocketBufferHandler.java:213)
                at org.apache.tomcat.util.net.Nio2Channel.reset(Nio2Channel.java:59)
                at org.apache.tomcat.util.net.SecureNio2Channel.reset(SecureNio2Channel.java:134)
                at org.apache.tomcat.util.net.Nio2Endpoint.setSocketOptions(Nio2Endpoint.java:321)
                at org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor.completed(Nio2Endpoint.java:460)
                at org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor.completed(Nio2Endpoint.java:396)
                at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
                at sun.nio.ch.Invoker$2.run(Invoker.java:218)
                at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
                at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(T

Reason is code has been compiled on Java 9+ which is not backward compatible.

(In reply to Allan from comment #25)
> Hi Mark, I have carried out a number of connectivity tests.  Result as
> follow.
> 
> Tomcat 8.5.72-4795df9 and 9.0.57-115334b with Oracle Java JDK 9,11,13
> (specifically 9.0.4, 11.0.2, 13.0.2) on RHEL 7 and Windows
> HTTP - pass
> HTTPS NIO - pass
> HTTPS NIO2 - pass.
> 
> Tomcat 8.5.72-4795df9 and 9.0.57-115334b with Oracle Java JRE 8
> (1.8.0.212b31 or jdk_1.8.0.251) on RHEL 7 and Windows
> HTTP - failed
> HTTPS NIO - failed
> HTTPS NIO2 - failed
> See errors from logs below.

Michael is right. I built the tests builds on my development machine which uses Java 11 by default and I didn't change it. Release builds are always made with Java 8 for Tomcat 9 and Java 7 for Tomcat 8.

It is good that your testing confirms that the issue is fixed for Java 11. I am confident that it will also be fixed for Java 8. That said, if you want test builds that will run on Java 8 just let me know.

(In reply to Mark Thomas from comment #27)
> (In reply to Allan from comment #25)
> > Hi Mark, I have carried out a number of connectivity tests.  Result as
> > follow.
> > 
> > Tomcat 8.5.72-4795df9 and 9.0.57-115334b with Oracle Java JDK 9,11,13
> > (specifically 9.0.4, 11.0.2, 13.0.2) on RHEL 7 and Windows
> > HTTP - pass
> > HTTPS NIO - pass
> > HTTPS NIO2 - pass.
> > 
> > Tomcat 8.5.72-4795df9 and 9.0.57-115334b with Oracle Java JRE 8
> > (1.8.0.212b31 or jdk_1.8.0.251) on RHEL 7 and Windows
> > HTTP - failed
> > HTTPS NIO - failed
> > HTTPS NIO2 - failed
> > See errors from logs below.
> 
> Michael is right. I built the tests builds on my development machine which
> uses Java 11 by default and I didn't change it. Release builds are always
> made with Java 8 for Tomcat 9 and Java 7 for Tomcat 8.
> 
> It is good that your testing confirms that the issue is fixed for Java 11. I
> am confident that it will also be fixed for Java 8. That said, if you want
> test builds that will run on Java 8 just let me know.

Thanks for the confirmation. I thought that maybe the case. There is no need to provide a Java 8 test image as you have already explained that this will be covered in the coming release. Lets hope all goes well.

Other then that, I just want to thank you and your team for the quick turn around to resolve this issue.  You guys are awesome.