Created attachment 36855 [details]
hs_err_pid6113 log

Problematic frame:
# C  [libtcnative-1.so+0x20d99a]  apr_pool_cleanup_kill+0x3a

Created attachment 36856 [details]
hs_err_pid14911 log

# Problematic frame:
# C  [libtcnative-1.so+0x14d462]  X509_VERIFY_PARAM_free+0x12

Both those crashes are during shutdown of the NIO2 connector when it is using OpenSSL for TLS.

It would be worth trying an upgrade to 8.5.47 and Tomcat Native 1.2.23 although I don't see anything in the changelog that obviously matches this issue.

(In reply to Mark Thomas from comment #3)
> Both those crashes are during shutdown of the NIO2 connector when it is
> using OpenSSL for TLS.
> 
> It would be worth trying an upgrade to 8.5.47 and Tomcat Native 1.2.23
> although I don't see anything in the changelog that obviously matches this
> issue.

I wonder whether the APR connector would crash here too.

It would be useful to test Tomcat 9 with NIO.
The stack indicates a crash on stop only, are the context reloads related to it ?

(In reply to Remy Maucherat from comment #5)
> It would be useful to test Tomcat 9 with NIO.
> The stack indicates a crash on stop only, are the context reloads related to
> it ?

Tested with tomcat 9.0.27 with NIO2
Loaded APR based Apache Tomcat Native library [1.2.23] using APR version [1.7.0].
OpenSSL successfully initialized [OpenSSL 1.1.1c  28 May 2019]
Please find the attached logs.(hs_err_pid27914,SERVER2.server.runlog,SERVER2.System.0)
Context reload are of during deployment of the webapp.




2019-10-25 14:35:16.602, SEVERE, module.Communication.HTTP - sachin inside3 getSSLHostConfig _default_
2019-10-25 14:35:16.602, WARNING, org.apache.tomcat.util.net.SSLHostConfigorg.apache.tomcat.util.net.SSLHostConfig setProtocols
 - The protocol [TLSv1.1] was added to the list of protocols on the SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
2019-10-25 14:35:16.602, WARNING, org.apache.tomcat.util.net.SSLHostConfigorg.apache.tomcat.util.net.SSLHostConfig setProtocols
 - The protocol [TLSv1] was added to the list of protocols on the SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
2019-10-25 14:35:16.609, INFO, org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.AprLifecycleListener lifecycleEvent
 - Loaded APR based Apache Tomcat Native library [1.2.23] using APR version [1.7.0].
2019-10-25 14:35:16.609, INFO, org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.AprLifecycleListener lifecycleEvent
 - APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
2019-10-25 14:35:16.609, INFO, org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.AprLifecycleListener lifecycleEvent
 - APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
2019-10-25 14:35:16.613, INFO, org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.AprLifecycleListener initializeSSL
 - OpenSSL successfully initialized [OpenSSL 1.1.1c  28 May 2019]
2019-10-25 14:35:16.620, INFO, org.apache.coyote.http11.Http11Nio2Protocolorg.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol
 - The ["https-openssl-nio2-6090"] connector has been configured to support negotiation to [h2] via ALPN

Created attachment 36858 [details]
JVM log on tomcat9

Created attachment 36859 [details]
runlog on tomcat9

Created attachment 36860 [details]
application log on tomcat9

# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libtcnative-1.so+0x2c61be]  apr_pool_cleanup_kill+0x3a
#



Stack: [0x00007f91865ec000,0x00007f91866ed000],  sp=0x00007f91866eb110,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtcnative-1.so+0x2c61be]  apr_pool_cleanup_kill+0x3a
C  [libtcnative-1.so+0x2c632c]  apr_pool_cleanup_run+0x2b
j  org.apache.tomcat.jni.SSLContext.free(J)I+0
j  org.apache.tomcat.util.net.openssl.OpenSSLContext.destroy()V+25
j  org.apache.tomcat.util.net.AbstractEndpoint.releaseSSLContext(Lorg/apache/tomcat/util/net/SSLHostConfig;)V+50
j  org.apache.tomcat.util.net.AbstractEndpoint.destroySsl()V+43
j  org.apache.tomcat.util.net.Nio2Endpoint.unbind()V+16
j  org.apache.tomcat.util.net.AbstractEndpoint.destroy()V+11
j  org.apache.coyote.AbstractProtocol.destroy()V+48
j  org.apache.catalina.connector.Connector.destroyInternal()V+11
j  org.apache.catalina.util.LifecycleBase.destroy()V+239
j  cmg.protocols.http.EmbeddedTomcat.stopService(Ljava/lang/String;)V+97
j  cmg.protocols.http.TomcatController.removeConnector(Ljava/lang/String;)Z+4
j  cmg.stdapp.restservices.frontend.RestWebServicesFE.deactivate()V+31
j  cmg.services.ConfigurableNodeImpl.stop(Lcom/ericsson/cm/common/configurationmanagement/javabeans/online/ConfigId;ZZZ)Z+146
j  cmg.communicationmanager.controller.NodeConfigurationHandler.handleManagerRequest(Lcom/ericsson/cm/common/messagingframework/javabeans/Message;)V+1032
j  cmg.communicationmanager.controller.ManagerRequestHandler$RequestHandlerThread.run()V+299
J 4767 C1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f9205dedeb4 [0x00007f9205dece80+0x1034]

(In reply to sachin.pipal1 from comment #6)
> (In reply to Remy Maucherat from comment #5)
> > It would be useful to test Tomcat 9 with NIO.
> > The stack indicates a crash on stop only, are the context reloads related to
> > it ?
> 
> Tested with tomcat 9.0.27 with NIO2
> Loaded APR based Apache Tomcat Native library [1.2.23] using APR version
> [1.7.0].
> OpenSSL successfully initialized [OpenSSL 1.1.1c  28 May 2019]
> Please find the attached
> logs.(hs_err_pid27914,SERVER2.server.runlog,SERVER2.System.0)
> Context reload are of during deployment of the webapp.
> 
> 
> 
> 
> 2019-10-25 14:35:16.602, WARNING,
> org.apache.tomcat.util.net.SSLHostConfigorg.apache.tomcat.util.net.
> SSLHostConfig setProtocols
>  - The protocol [TLSv1.1] was added to the list of protocols on the
> SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
> 2019-10-25 14:35:16.602, WARNING,
> org.apache.tomcat.util.net.SSLHostConfigorg.apache.tomcat.util.net.
> SSLHostConfig setProtocols
>  - The protocol [TLSv1] was added to the list of protocols on the
> SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
> 2019-10-25 14:35:16.609, INFO,
> org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.
> AprLifecycleListener lifecycleEvent
>  - Loaded APR based Apache Tomcat Native library [1.2.23] using APR version
> [1.7.0].
> 2019-10-25 14:35:16.609, INFO,
> org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.
> AprLifecycleListener lifecycleEvent
>  - APR capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 2019-10-25 14:35:16.609, INFO,
> org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.
> AprLifecycleListener lifecycleEvent
>  - APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 2019-10-25 14:35:16.613, INFO,
> org.apache.catalina.core.AprLifecycleListenerorg.apache.catalina.core.
> AprLifecycleListener initializeSSL
>  - OpenSSL successfully initialized [OpenSSL 1.1.1c  28 May 2019]
> 2019-10-25 14:35:16.620, INFO,
> org.apache.coyote.http11.Http11Nio2Protocolorg.apache.coyote.http11.
> AbstractHttp11Protocol configureUpgradeProtocol
>  - The ["https-openssl-nio2-6090"] connector has been configured to support
> negotiation to [h2] via ALPN

Ok, so this is clearly embedded (don't see why it would make a difference yet) and it crashes on shutdown.

Can you also test NIO (the default connector) with Tomcat 9 ?
Also can you give details about your use (we need to be able to reproduce the crash) ?

(In reply to Remy Maucherat from comment #12)
> Ok, so this is clearly embedded (don't see why it would make a difference
> yet) and it crashes on shutdown.
> 
> Can you also test NIO (the default connector) with Tomcat 9 ?
> Also can you give details about your use (we need to be able to reproduce
> the crash) ?


Same error encountered with  tomcat 9 +NIO.

We are using embedded tomcat and deploying multiple restful services over HTTP2 protcol.Tomcat crashes randomly within 10 -12 minutes while stopping/starting the one of the multiple deployed rest web application and getting Libtcnative crashes error in log.

(In reply to sachin.pipal1 from comment #13)
> (In reply to Remy Maucherat from comment #12)
> > Ok, so this is clearly embedded (don't see why it would make a difference
> > yet) and it crashes on shutdown.
> > 
> > Can you also test NIO (the default connector) with Tomcat 9 ?
> > Also can you give details about your use (we need to be able to reproduce
> > the crash) ?
> 
> 
> Same error encountered with  tomcat 9 +NIO.
> 
> We are using embedded tomcat and deploying multiple restful services over
> HTTP2 protcol.Tomcat crashes randomly within 10 -12 minutes while
> stopping the one of the multiple deployed rest web application and
> getting Libtcnative crashes error in log.

We got and read your latest comment with the additional testing.
However, pinging developers for updates is not acceptable behavior. Please don't do this again.

I have tried, and failed, to recreate this. Please provide the simplest possible test case that re-creates the issue.

Marking resolved this issue, as the issue was with the custom code while creating the AprLifeccycleListner the reference was getting overridden.

Thanks to all who supported during the time.

Thanks
Sachin

Was this caused by custom native code of custom Java code?

I would argue that, unless you went into the C stack/heap with a hatchet, (ab)using the tcnative code shouldn't cause a JVM crash. Exceptions, dropped connections, etc., sure. But the tcnative code should be protecting itself against improper usage.

Can you please share your custom code surrounding your use of AprLifecycleListener? Maybe there really is a bug, here.

Since this was using embedded with a crash only on shutdown, and given comment 17, I would say the issue was some sort of misuse of the AprLifecycleListener or the handling of the native library. The debug given didn't show an issue with the SSL context close (all it does is prevent a crash caused by a double free).