Bug 56313 - Tomcat 8 crashes in tcnative-1.dll+0x7923
Summary: Tomcat 8 crashes in tcnative-1.dll+0x7923
Status: RESOLVED DUPLICATE of bug 57653
Alias: None
Product: Tomcat Native
Classification: Unclassified
Component: Library (show other bugs)
Version: 1.1.29
Hardware: PC All
: P2 major with 10 votes (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
: 55797 56415 57140 (view as bug list)
Depends on:
Reported: 2014-03-25 19:07 UTC by Timcesq
Modified: 2017-08-23 21:57 UTC (History)
3 users (show)

the log from the crash (52.63 KB, text/plain)
2014-03-25 19:07 UTC, Timcesq
another log - same issue (52.41 KB, text/plain)
2014-03-25 19:09 UTC, Timcesq
third one - same issue. Just to show that this is a recurring thing. (52.56 KB, text/plain)
2014-03-25 19:40 UTC, Timcesq
JMeter test plan (8.59 KB, application/xml)
2014-03-26 17:31 UTC, Timcesq

Note You need to log in before you can comment on or make changes to this bug.
Description Timcesq 2014-03-25 19:07:06 UTC
Created attachment 31435 [details]
the log from the crash

Tomcat 8 crashed when ~1000 users were browsing a website (JSP front-end, Struts-2 backend with Java Persistence API, Hibernate, c3p0, MySQL).

No crashes are observed when using JAVA NIO (not tested with JAVA BIO).

The attached file is from the crash.
Comment 1 Timcesq 2014-03-25 19:09:41 UTC
Created attachment 31436 [details]
another log - same issue
Comment 2 Timcesq 2014-03-25 19:15:54 UTC
APR version is 1.5. OpenSSL 1.0.1f.
Comment 3 Timcesq 2014-03-25 19:40:24 UTC
Created attachment 31437 [details]
third one - same issue. Just to show that this is a recurring thing.
Comment 4 Konstantin Kolinko 2014-03-25 19:45:26 UTC
> Tomcat 8 crashed

What exactly version of Tomcat 8.0.x ?
Comment 5 Timcesq 2014-03-25 19:47:27 UTC
8.0.3 - currently the latest one, AFAIK.
Comment 6 Christopher Schultz 2014-03-25 19:51:55 UTC
Does high-load appear to be a required factor?
Comment 7 Timcesq 2014-03-25 19:59:22 UTC
Apaprently - yes. I wasn't able to reproduce it with low number of concurrent users. But it's hard to say if it would never crash or it's just that it would crash -- only much later.

I used JMeter to simulate high load.

The whole server works flawlessly (including very low request-response latency) until the very end when it crashes. Nothing exceptional happens in the system: no unusual disk activity, no other CPU- or RAM-hungry processes. It just crashes suddenly for no obvious reason.
Comment 8 Timcesq 2014-03-25 20:02:45 UTC
With 4000 users, it crashes in less than a minute (Used JMeter to test).
Comment 9 Christopher Schultz 2014-03-26 17:02:27 UTC
Could you provide a JMeter test case? I'm sure that would be helpful. I'm not win32 dev, but I'm sure something that can reproduce the issue would be helpful to whoever looks at this.
Comment 10 Timcesq 2014-03-26 17:31:53 UTC
Created attachment 31441 [details]
JMeter test plan
Comment 11 Timcesq 2014-03-26 19:03:09 UTC
I used a debugger to locate the crash address in the source code. It looks like the crash occurs inside the function "Java_org_apache_tomcat_jni_Poll_poll", specifically inside the following block:

for (i = 0; i < num; i++) {
    tcn_socket_t *s = (tcn_socket_t *)fd->client_data;
    p->set[i*2+0] = (jlong)(fd->rtnevents);
    p->set[i*2+1] = P2J(s);
    if (remove) {
        apr_pollset_remove(p->pollset, fd);
        APR_RING_REMOVE(s->pe, link);
        APR_RING_INSERT_TAIL(&p->dead_ring, s->pe, tcn_pfde_t, link);
        s->pe = NULL;
    else {
        /* Update last active with the current time
        * after the poll call.
        s->last_active = now;
    fd ++;

The crash address suggests that a read of the "s->pe" variable inside the "if(remove)" block fails (the variable is null).
Comment 12 Timcesq 2014-03-26 19:04:54 UTC
What I meant is that the s->pe is NULL, so a read from that address crashes the application.
Comment 13 Timcesq 2014-03-26 19:07:10 UTC
A quick workaround would be to check for NULL before doing the read. However, if the library is designed so that there shouldn't be any NULLs at this point in the code, such patch would only mask the underlying bug (a race condition maybe?).
Comment 14 Pavel 2015-03-15 09:59:23 UTC
Same issue here, tried using the latest tomcat 7/8 but getting the same error and tomcat crashes.

Any updates on this? (currently the only solution is to switch to NIO)
Comment 15 Mark Thomas 2015-03-15 23:48:39 UTC

*** This bug has been marked as a duplicate of bug 57653 ***
Comment 16 Mark Thomas 2017-08-23 21:52:37 UTC
*** Bug 55797 has been marked as a duplicate of this bug. ***
Comment 17 Mark Thomas 2017-08-23 21:56:32 UTC
*** Bug 56415 has been marked as a duplicate of this bug. ***
Comment 18 Mark Thomas 2017-08-23 21:57:18 UTC
*** Bug 57140 has been marked as a duplicate of this bug. ***