I have detected what appears to be a regression in 8.5.20 with JSSE keystores since 8.5.16. With my limited understanding I'm unable to pinpoint the exact cause to a certainty after poking around a bit, so I thought I'd pass what info I have along and get some thoughts.

Below is the error message I am getting:
21-Aug-2017 15:01:57.989 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector [Connector[Http11Nio2ProtocolCryptovault-25005
]]
 org.apache.catalina.LifecycleException: Failed to initialize component [Connector[Http11Nio2ProtocolCryptovault-25005]]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:549)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:607)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:630)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:49
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:311)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:494)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:999)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: java.security.KeyStoreException: Cannot get key bytes, not PKCS#8 encoded
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:85)
        at org.apache.tomcat.util.net.Nio2Endpoint.bind(Nio2Endpoint.java:163)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:982)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:244)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:620)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:66)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:997)
        ... 13 more
Caused by: java.security.KeyStoreException: Cannot get key bytes, not PKCS#8 encoded
        at sun.security.provider.KeyProtector.protect(KeyProtector.java:174)
        at sun.security.provider.JavaKeyStore.engineSetKeyEntry(JavaKeyStore.java:267)
        at sun.security.provider.JavaKeyStore$JKS.engineSetKeyEntry(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:117)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineSetKeyEntry(JavaKeyStore.java:70)
        at java.security.KeyStore.setKeyEntry(KeyStore.java:1140)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.getKeyManagers(JSSEUtil.java:226)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112)
        ... 20 mor

I did notice that because of the revision mentioned above (to JSSEUtil.java), KeyStore objects that aren't PEM encoded are loaded as an in-memory JKS keystore and a call to setKeyEntry is made. I may be wrong, but I think this is causing the failure. The hint I had to go from is that the documentation for the second form of setKeyEntry requires the key bytes to be PKCS8 encoded since this underlying keystore is JKS[1]... but we cannot guarantee that the getKey[2] call returned a Key that is PKCS8 encoded. With the implementation I am using, it's unclear what the encoding is for the Key object, but since PKCS11 is a common interface for hardware crypto, I'm sure many different types (or none at all) are possible. Looking into the source for engineSetKeyEntry(String alias, byte[] key, Certificate[] chain), I see that a call to protect() is made which does the check for PKCS8 encoding. This appears to explain the exception.

Unfortunately... I'm not sure where to go from there (if that even is the issue). It wouldn't help to switch to setKeyEntry(String alias, byte[] key, Certificate[] chain) since that also has the same PKCS8 encoding requirement. I also don't think it would be possible to obtain the raw key bytes since a hardware crypto device would certainly block such an operation and it would be out of Tomcat's place to try to convert among encoding formats.

Thoughts greatly appreciated

[1] https://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html#setKeyEntry(java.lang.String,%20java.security.Key,%20char[],%20java.security.cert.Certificate[]
[2] https://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html#getKey(java.lang.String, char[])