Bug 63405 - Tomcat 7.0.91.0 EXCEPTION_ACCESS_VIOLATION - Problematic frame tcnative-1.dll+0x802e
Summary: Tomcat 7.0.91.0 EXCEPTION_ACCESS_VIOLATION - Problematic frame tcnative-1.dll...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat Native
Classification: Unclassified
Component: Library (show other bugs)
Version: unspecified
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-05-06 04:55 UTC by kd
Modified: 2020-08-20 15:24 UTC (History)
0 users



Attachments
Tomcat crash error file (110.60 KB, text/plain)
2019-05-06 04:55 UTC, kd
Details

Note You need to log in before you can comment on or make changes to this bug.
Description kd 2019-05-06 04:55:04 UTC
Created attachment 36571 [details]
Tomcat crash error file

Hi,

During application performance testing on Tomcat 7.0.91.0, Tomcat crashed multiple times with EXCEPTION_ACCESS_VIOLATION error as follows. PFA error file generated after crash.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000018000802e, pid=97772, tid=0x000000000000c740
#
# JRE version: OpenJDK Runtime Environment (8.0_192-b01) (build 1.8.0_192-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.192-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [tcnative-1.dll+0x802e]
#
# Core dump written. Default location: D:\nice_systems\RTServer\Tomcat\hs_err_pid97772.mdmp
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Comment 1 Mark Thomas 2019-06-20 13:42:48 UTC
Could you please provide your connector configuration and a description of the requests being sent at the time. It is particularly helpful to know what type(s) of requests were being used (WebSocket, Comet, sendfile, static files, servlets, etc.)

Without a test case we can use to reproduce this, we need to narrow down the volume of code we need to manually inspect to try and figure out what the root cause is.
Comment 2 kd 2019-07-17 09:53:59 UTC
server.xml

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
  <Connector port="7077" protocol="HTTP/1.1"
               maxThreads="1000" connectionTimeout="20000"
               redirectPort="7078"
               server="Web" />
	
	<Connector port="7078" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="1000" connectionTimeout="20000" scheme="https" secure="true"
               SSLEnabled="true" 
               SSLCertificateFile="D:\Tomcat\certs\server.crt"
               SSLCertificateKeyFile="D:\Tomcat\certs\server.pem"
               SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
               SSLCipherSuite="ALL:!aNULL:!ADH:!eNULL:!LOW:!MEDIUM:!EXP:RC4+RSA:+HIGH"
               server="Web" />
    
    
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"
			   digest="MD5"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
       
      </Host>
    </Engine>
  </Service>
</Server>

REST API were invoked at the time of crash.
We also use Websocket for client server communication.
Please let me know if you need more details.
Comment 3 Michael Osipov 2020-04-06 13:33:19 UTC
Does this still occur with the most recent versions of Tomcat, libtcnative and
libapr?
Comment 4 Mark Thomas 2020-08-20 15:24:58 UTC
The crash log indicates that the crash occurred in the Poller.

I've reviewed the changes since Tomcat 7.0.91 and Tomcat Native 1.2.17 and I don't see any references in the change log for fixing issues that could trigger a crash like this.

There is a significant change in 7.0.99 onwards where the code was refactored to use a single pollset rather than multiple.

I have also compared the 7.0.x code to 8.5.x and while a direct comparison is not possible due to 7.0.x having to support Comet, I don't see any obvious differences that could trigger a crash. I did make a few cosmetic changes to bring 7.0.x into line with 8.5.x and I also fixed one threading issue although the 8.5.x changelog indicated that that was related to timeouts.

At this point I suspect a bug in the code that handled multiple pollsets. I am therefore going to resolve this as fixed. If you still see this issue, or one like it please:
- update to the latest available major Tomact version (7.0.x, 8.5.x, 9.0.x ot 10.0.x) and latest available Tomcat native version
- retest

Anything you can do to narrow down how to trigger the issue is helpful and steps that reproduce are ideal.