Bug 47714 - Response mixed between users
Response mixed between users
Status: REOPENED
Product: Tomcat Connectors
Classification: Unclassified
Component: mod_jk
1.2.28
Other Linux
: P1 critical with 11 votes (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2009-08-20 09:04 UTC by sameem.ahmed
Modified: 2015-03-14 14:07 UTC (History)
6 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description sameem.ahmed 2009-08-20 09:04:36 UTC
Hi,

We are migrating our application to JBoss container version 4.2.x with Apache web server in the front. We are using the mod_jk connector version 1.2.28. Our Apache version is 2.2.6. AJP13 worker is used for communication between web server and Tomcat.

We have picked up an issue during load testing where the response is being mixed between users - one user is able to see reponse of another.  

Is this a bug in mod_jk?

We are planning to switch to mod_proxy_ajp. Will it help to resolve this issue?

We can't take this to production until this issue is resolved. Kindly advice.

Thanks & Regards,
Sameem Ahmed
Comment 1 jfclere 2009-08-20 09:22:33 UTC
AS 4.2.x = jbossweb-2.0.x
jbossweb-2.0.x ~ tc-6.0.x

the bug could be in mod_jk as well as in the AJP connector, try mod_proxy_ajp if it works then it is most probably a mod_jk bugs otherwise it is in the AJP connector and you could use the jboss forums for help.
Comment 2 Derek 2009-11-18 12:11:52 UTC
Has this bug where "responses are being mixed between users" been confirmed in mod_jk? If so, would it apply to all the previous versions of mod_JK?
Comment 3 Kevin J Walters 2009-11-26 07:03:34 UTC
1.2.21 can exhibit request/response mixing - could be the same issue? One way this can occur is some event that triggers a bug where the http response from tomcat ends up buffered. The next request to that apache worker process will then pick up the previous response from the persistent tcp connection. For this particular type of bug, evidence of an ongoing issue can be seen in a few ways:

1) buffered tcp data between tomcat and apache that stays at the same level - easiest to observe when load is very light - run netstat -a a few times and look for established tcp connections with non 0 data in queues.

2) the responses will be way too fast because they are instant due to previous response already waiting

3) if your using prefork and logging worker process pids, you can look for the pattern by comparing things like response size in logs for URLs served by each pid

4) quiesce the server and then fire a series of requests for a known servlet and verify responses match expected result

A workaround or preventative measure is DisableReuse
Comment 4 Pid 2009-11-26 07:18:52 UTC
(In reply to comment #0)
> Hi,
> 
> We are migrating our application to JBoss container version 4.2.x with Apache
> web server in the front. We are using the mod_jk connector version 1.2.28. Our
> Apache version is 2.2.6. AJP13 worker is used for communication between web
> server and Tomcat.
> 
> We have picked up an issue during load testing where the response is being
> mixed between users - one user is able to see reponse of another.  

Might sound like an dumb question, but what are you load testing against?

Have you eliminated the possibility that the application has an error which might cause this?
Comment 5 Sudhir Reddy 2009-12-16 20:38:24 UTC
I too happen to notice the same issue on our production boxes with mod_jk-1.2.28, DisableReuse fixed this issue.

We are using apache <---> mod_jk-1.2.28 <---AJP13---> tomcat-5.5.23

tomcat logs shows HTTP 302 being returned and apache logs shows HTTP 200

I am trying to reproduce the same on our non-production boxes. The only possible scenario I can think of is the following:
* mod_jk: Request 1 sent to tomcat got timeout after 2 mins while waiting for a response
* mod_jk : Request 2 reuses the socket of the Request 1 and reads the response of the Request 1 from tomcat.

Please let  me know if you think the above scenario is not possible.

We do have timeouts configured for 2 mins and retries not set(defaults to 2).

We did see lots of HTTP 400 error messages(Bad Requests) also caused due to mod_jk and got fixed when we used DisableReuse
Comment 6 Rainer Jung 2009-12-17 02:23:29 UTC
(In reply to comment #5)

Which platform?

Are you using tcnative aka the APR connector? What happens if you disable it?
Comment 7 Sudhir Reddy 2009-12-18 02:57:42 UTC
> Which platform?

Linux (RHEL)

> Are you using tcnative aka the APR connector? What happens if you disable it?
Yes. I am not able to reproduce this in non-production. Can't try this on production boxes :)

---------------------
<Server>
  <!-- This enables use of the apache portable runtime if apr package is installed -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"/>
....
</SERVER>
---------------------


BTW is there a chance that this can cause server side connection aborts too ? We did see couple of them, not sure if its related to mod_jk.
Comment 8 Pawel Tucholski 2010-03-26 15:17:54 UTC
Hi

I think we are facing an issue similiar or same as the described and we have also mod_jk 1.2.28. However we can not reproduce it in the test env or during performance testing.

Could you give me some hint on how to reproduce the issue you were facing?

Is this the scenario to reproduce it?

* mod_jk: Request 1 sent to tomcat got timeout after 2 mins while waiting for a
response
* mod_jk : Request 2 reuses the socket of the Request 1 and reads the response
of the Request 1 from tomcat.

The thing is that we don't set the socket_timeout (default is infinity) so that does not fit to the described scenario.

Is it a true statement that DisableResuse set to false is solving your issue?

Cheers

Pawel
Comment 9 Sudhir Reddy 2010-03-26 15:51:10 UTC
> Is it a true statement that DisableResuse set to false is solving your issue?

Yes, I have not seen another instance after setting this flag on our production boxes.
Comment 10 Paul Hinds 2010-03-29 09:53:34 UTC
To follow up Pawels comments in this chain, we went live with DiableReuse recently and it did not resolve the issue for us.

We may be looking at a similar issue, but perhaps not this bug. We seem to be experiencing resubmition of POST data, rather than response mixing. Data in a previous unrelated POST request is somehow merged with a subsequent request from a different user.

Currently we still do not know the root cause so can not confirm this is a mod_jk issue.  

It is still critical for us.
Comment 11 Sudhir Reddy 2010-03-29 15:01:32 UTC
> We seem to be experiencing resubmition of POST data, rather than response mixing.

Just in case you are not aware ....

mod_jk retries http post requests in case of timeout in getting response from
tomcat with default configurations(no retries or recovery_options specified).

See : https://issues.apache.org/bugzilla/show_bug.cgi?id=48400

As part of the DisableReuse, i did set recovery_options to not retry post data and have not seen this issue after that. May be setting both will fix your issue ?

For some reason, i was not able to reproduce this behaviour(response swaping) in non-production boxes.
Comment 12 Kevin J Walters 2010-03-29 17:50:15 UTC
(In reply to comment #10)
> Data in a previous unrelated POST request is somehow merged with
> a subsequent request from a different user.

A general comment and not one aimed at your application - the majority of incidents of this i have seen are from application bugs rather than infrastructure ones. It's very easy to mis-share data in a single process, multi-threaded environment. These will often only reveal themselves under truly concurrent request load from multiple login accounts.

Perhaps it's worth doing something with userids / unique ids / http headers / logging to allow you to match up requests and responses and verify the correct data is being sent back from the application layer.
Comment 13 Paul Hinds 2010-04-01 06:31:31 UTC
We have now found out about issues that are resolved in jk .30 that look related, to do with reposting buffers.

We did not disable retries.

"mod_jk retries http post requests in case of timeout in getting response from
tomcat with default configurations(no retries or recovery_options specified)."

We will do this anyway, since our application does not like "retries", they can (in theory) result in two or more transactions being processed.

I think next test will be with .30 , no reuse, no reties and a new tomcat from JBoss.

BTW: We are pretty sure our application is not at fault.

I will post results of that test when I get them.
Comment 14 Nageshwar 2010-04-02 05:56:02 UTC
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x48b8e1fe, pid=1412, tid=716
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) Client VM (14.2-b01 mixed mode windows-x86 )
# Problematic frame:
# C  [tcnative-1.dll+0xe1fe]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x48710400):  JavaThread "http-7000-1" daemon [_thread_in_native, id=716, stack(0x4a140000,0x4a180000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000034

Registers:
EAX=0x00000001, EBX=0x00000000, ECX=0x00000000, EDX=0x00000001
ESP=0x4a17ebd8, EBP=0x4a17ebe0, ESI=0x00000000, EDI=0x48710400
EIP=0x48b8e1fe, EFLAGS=0x00010246

Top of Stack: (sp=0x4a17ebd8)
0x4a17ebd8:   48710400 4a17ec1c 4a17ebf8 48b8111f
0x4a17ebe8:   4a17ebf4 00000001 00000000 00000000
0x4a17ebf8:   4a17ec28 00934f8f 48710510 4a17ec1c
0x4a17ec08:   00000001 49d3f6b0 00000000 00000001
0x4a17ec18:   00000001 42b3f598 48710400 48710400
0x4a17ec28:   4a17ed28 007d6e48 49d3f6b0 00000000
0x4a17ec38:   4a17ec5c 48710400 4a17ec5c 6d99918b
0x4a17ec48:   48710400 6d9991b2 42de44b0 48710400 

Instructions: (pc=0x48b8e1fe)
0x48b8e1ee:   5d c2 0c 00 83 f8 01 0f 85 80 00 00 00 8b 75 10
0x48b8e1fe:   8b 46 34 85 c0 74 3a 8b 56 14 c7 42 14 10 00 00 


Stack: [0x4a140000,0x4a180000],  sp=0x4a17ebd8,  free space=250k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [tcnative-1.dll+0xe1fe]
C  [tcnative-1.dll+0x111f]
J  org.apache.tomcat.jni.Address.get(IJ)J
J  org.apache.coyote.http11.Http11AprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
j  org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+56
J  org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String;
J  org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/String;
J  com.ecolog.generic.Session.getSession(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;)Lcom/ecolog/generic/Session;
j  org.apache.jsp.warehouse_v3.cartitems_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+71
j  org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
j  org.apache.jasper.servlet.JspServletWrapper.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V+395
j  org.apache.jasper.servlet.JspServlet.serviceJspFile(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;Ljava/lang/Throwable;Z)V+134
j  org.apache.jasper.servlet.JspServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+436
J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V
J  org.apache.coyote.http11.Http11AprProcessor.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;
J  org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+147
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [jvm.dll+0xecf9c]
V  [jvm.dll+0x1741e1]
V  [jvm.dll+0xed167]
V  [jvm.dll+0xed1dd]
V  [jvm.dll+0x116290]
V  [jvm.dll+0x1d0424]
V  [jvm.dll+0x173e5c]
C  [MSVCR71.dll+0xb381]
C  [KERNEL32.dll+0xb3bc]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  org.apache.tomcat.jni.Address.get(IJ)J
J  org.apache.coyote.http11.Http11AprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
j  org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+56
J  org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String;
J  org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/String;
J  com.ecolog.generic.Session.getSession(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;)Lcom/ecolog/generic/Session;
j  org.apache.jsp.warehouse_v3.cartitems_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+71
j  org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
j  org.apache.jasper.servlet.JspServletWrapper.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V+395
j  org.apache.jasper.servlet.JspServlet.serviceJspFile(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;Ljava/lang/Throwable;Z)V+134
j  org.apache.jasper.servlet.JspServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+436
J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V
J  org.apache.coyote.http11.Http11AprProcessor.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;
J  org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+147
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x4906ec00 JavaThread "ajp-8009-1" daemon [_thread_blocked, id=204, stack(0x4c240000,0x4c280000)]
  0x4919a800 JavaThread "http-7000-13" daemon [_thread_blocked, id=5484, stack(0x4c1c0000,0x4c200000)]
  0x4919a400 JavaThread "http-7000-12" daemon [_thread_blocked, id=5480, stack(0x4c180000,0x4c1c0000)]
  0x490ad400 JavaThread "http-7000-11" daemon [_thread_blocked, id=3920, stack(0x4c140000,0x4c180000)]
  0x48fc8400 JavaThread "http-7000-10" daemon [_thread_blocked, id=3876, stack(0x4c100000,0x4c140000)]
  0x48ebd400 JavaThread "http-7000-9" daemon [_thread_blocked, id=3832, stack(0x4c0c0000,0x4c100000)]
  0x48ebcc00 JavaThread "http-7000-8" daemon [_thread_blocked, id=3828, stack(0x4c080000,0x4c0c0000)]
  0x49022800 JavaThread "http-7000-7" daemon [_thread_blocked, id=1056, stack(0x4c040000,0x4c080000)]
  0x48f24400 JavaThread "http-7000-6" daemon [_thread_blocked, id=584, stack(0x4a800000,0x4a840000)]
  0x48b25400 JavaThread "AWT-Windows" daemon [_thread_in_native, id=3640, stack(0x4a7c0000,0x4a800000)]
  0x48e7ac00 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=2052, stack(0x4a740000,0x4a780000)]
  0x48b39400 JavaThread "http-7000-5" daemon [_thread_blocked, id=976, stack(0x4a700000,0x4a740000)]
  0x48aa6400 JavaThread "http-7000-4" daemon [_thread_blocked, id=1696, stack(0x4a6c0000,0x4a700000)]
  0x48f26400 JavaThread "http-7000-3" daemon [_thread_blocked, id=1652, stack(0x4a680000,0x4a6c0000)]
  0x48afe800 JavaThread "http-7000-2" daemon [_thread_blocked, id=1064, stack(0x4a640000,0x4a680000)]
=>0x48710400 JavaThread "http-7000-1" daemon [_thread_in_native, id=716, stack(0x4a140000,0x4a180000)]
  0x48e68400 JavaThread "ajp-8009-CometPoller-7" daemon [_thread_blocked, id=3592, stack(0x4a100000,0x4a140000)]
  0x48e67000 JavaThread "ajp-8009-CometPoller-6" daemon [_thread_blocked, id=3588, stack(0x4a0c0000,0x4a100000)]
  0x48e62400 JavaThread "ajp-8009-CometPoller-5" daemon [_thread_blocked, id=3584, stack(0x4a080000,0x4a0c0000)]
  0x48e60c00 JavaThread "ajp-8009-CometPoller-4" daemon [_thread_blocked, id=3304, stack(0x4a040000,0x4a080000)]
  0x48e5f800 JavaThread "ajp-8009-CometPoller-3" daemon [_thread_blocked, id=3316, stack(0x4a000000,0x4a040000)]
  0x48e5e000 JavaThread "ajp-8009-CometPoller-2" daemon [_thread_blocked, id=640, stack(0x49fc0000,0x4a000000)]
  0x48e59400 JavaThread "ajp-8009-CometPoller-1" daemon [_thread_blocked, id=1608, stack(0x49f80000,0x49fc0000)]
  0x48e58000 JavaThread "ajp-8009-CometPoller-0" daemon [_thread_blocked, id=1660, stack(0x49f40000,0x49f80000)]
  0x48e56800 JavaThread "ajp-8009-Poller-7" daemon [_thread_blocked, id=644, stack(0x49f00000,0x49f40000)]
  0x48e55400 JavaThread "ajp-8009-Poller-6" daemon [_thread_blocked, id=632, stack(0x49ec0000,0x49f00000)]
  0x48e2c800 JavaThread "ajp-8009-Poller-5" daemon [_thread_blocked, id=1668, stack(0x49e80000,0x49ec0000)]
  0x48b53000 JavaThread "ajp-8009-Poller-4" daemon [_thread_blocked, id=636, stack(0x49a40000,0x49a80000)]
  0x48b51800 JavaThread "ajp-8009-Poller-3" daemon [_thread_blocked, id=3364, stack(0x49a00000,0x49a40000)]
  0x48b50400 JavaThread "ajp-8009-Poller-2" daemon [_thread_blocked, id=3368, stack(0x499c0000,0x49a00000)]
  0x48ae2c00 JavaThread "ajp-8009-Poller-1" daemon [_thread_blocked, id=3372, stack(0x49980000,0x499c0000)]
  0x48ae1c00 JavaThread "ajp-8009-Poller-0" daemon [_thread_blocked, id=3376, stack(0x49940000,0x49980000)]
  0x48b14400 JavaThread "ajp-8009-Acceptor-0" daemon [_thread_blocked, id=3380, stack(0x49900000,0x49940000)]
  0x48a66800 JavaThread "http-7000-Sendfile-0" daemon [_thread_blocked, id=3392, stack(0x498c0000,0x49900000)]
  0x48a65400 JavaThread "http-7000-CometPoller-7" daemon [_thread_blocked, id=3404, stack(0x49880000,0x498c0000)]
  0x489f3c00 JavaThread "http-7000-CometPoller-6" daemon [_thread_blocked, id=3396, stack(0x49840000,0x49880000)]
  0x489f2400 JavaThread "http-7000-CometPoller-5" daemon [_thread_blocked, id=3400, stack(0x49800000,0x49840000)]
  0x48e50000 JavaThread "http-7000-CometPoller-4" daemon [_thread_blocked, id=3408, stack(0x497c0000,0x49800000)]
  0x48e4b800 JavaThread "http-7000-CometPoller-3" daemon [_thread_blocked, id=3412, stack(0x49780000,0x497c0000)]
  0x48a62c00 JavaThread "http-7000-CometPoller-2" daemon [_thread_blocked, id=3416, stack(0x49740000,0x49780000)]
  0x48a61400 JavaThread "http-7000-CometPoller-1" daemon [_thread_blocked, id=3420, stack(0x49700000,0x49740000)]
  0x489e9c00 JavaThread "http-7000-CometPoller-0" daemon [_thread_blocked, id=3428, stack(0x496c0000,0x49700000)]
  0x489e8800 JavaThread "http-7000-Poller-7" daemon [_thread_blocked, id=3424, stack(0x49680000,0x496c0000)]
  0x48a71400 JavaThread "http-7000-Poller-6" daemon [_thread_blocked, id=908, stack(0x49640000,0x49680000)]
  0x48a70400 JavaThread "http-7000-Poller-5" daemon [_thread_blocked, id=3432, stack(0x49400000,0x49440000)]
  0x48657800 JavaThread "http-7000-Poller-4" daemon [_thread_blocked, id=3440, stack(0x493c0000,0x49400000)]
  0x48656000 JavaThread "http-7000-Poller-3" daemon [_thread_blocked, id=3564, stack(0x49380000,0x493c0000)]
  0x4871ec00 JavaThread "http-7000-Poller-2" daemon [_thread_blocked, id=3568, stack(0x49340000,0x49380000)]
  0x4871d400 JavaThread "http-7000-Poller-1" daemon [_thread_blocked, id=3560, stack(0x49300000,0x49340000)]
  0x48a35c00 JavaThread "http-7000-Poller-0" daemon [_thread_blocked, id=3572, stack(0x492c0000,0x49300000)]
  0x489f0400 JavaThread "http-7000-Acceptor-0" daemon [_thread_blocked, id=3576, stack(0x49280000,0x492c0000)]
  0x48706800 JavaThread "Thread-1" [_thread_blocked, id=2152, stack(0x48940000,0x48980000)]
  0x486cfc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2084, stack(0x488c0000,0x48900000)]
  0x486ca400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1596, stack(0x48880000,0x488c0000)]
  0x486c8c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=1928, stack(0x48840000,0x48880000)]
  0x486c7800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1920, stack(0x48800000,0x48840000)]
  0x486b4400 JavaThread "Finalizer" daemon [_thread_blocked, id=1768, stack(0x487c0000,0x48800000)]
  0x486b2c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=1764, stack(0x48780000,0x487c0000)]
  0x00546400 JavaThread "main" [_thread_in_native, id=1416, stack(0x00440000,0x00480000)]

Other Threads:
  0x486afc00 VMThread [stack: 0x48740000,0x48780000] [id=1760]
  0x486d2800 WatcherThread [stack: 0x48900000,0x48940000] [id=2088]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 def new generation   total 36352K, used 5720K [0x02600000, 0x04d70000, 0x074c0000)
  eden space 32320K,  12% used [0x02600000, 0x029c9d60, 0x04590000)
  from space 4032K,  45% used [0x04980000, 0x04b4c600, 0x04d70000)
  to   space 4032K,   0% used [0x04590000, 0x04590000, 0x04980000)
 tenured generation   total 483968K, used 220495K [0x074c0000, 0x24d60000, 0x42600000)
   the space 483968K,  45% used [0x074c0000, 0x14c13c50, 0x14c13e00, 0x24d60000)
 compacting perm gen  total 27648K, used 27539K [0x42600000, 0x44100000, 0x46600000)
   the space 27648K,  99% used [0x42600000, 0x440e4d98, 0x440e4e00, 0x44100000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x0040f000 	C:\Tomcat\bin\tomcat6.exe
0x77f80000 - 0x77ffc000 	C:\WINNT\system32\ntdll.dll
0x7c570000 - 0x7c624000 	C:\WINNT\system32\KERNEL32.dll
0x77e10000 - 0x77e6f000 	C:\WINNT\system32\USER32.dll
0x77f40000 - 0x77f7d000 	C:\WINNT\system32\GDI32.dll
0x7c2d0000 - 0x7c335000 	C:\WINNT\system32\ADVAPI32.dll
0x77d30000 - 0x77d9f000 	C:\WINNT\system32\RPCRT4.dll
0x78000000 - 0x78045000 	C:\WINNT\system32\MSVCRT.dll
0x7cf30000 - 0x7d176000 	C:\WINNT\system32\SHELL32.dll
0x70a70000 - 0x70ad6000 	C:\WINNT\system32\SHLWAPI.dll
0x71710000 - 0x71794000 	C:\WINNT\system32\COMCTL32.dll
0x6d800000 - 0x6da8b000 	C:\Java\jre6\bin\client\jvm.dll
0x77570000 - 0x775a0000 	C:\WINNT\system32\WINMM.dll
0x7c360000 - 0x7c3b6000 	C:\WINNT\system32\MSVCR71.dll
0x6d7b0000 - 0x6d7bc000 	C:\Java\jre6\bin\verify.dll
0x6d330000 - 0x6d34f000 	C:\Java\jre6\bin\java.dll
0x6d290000 - 0x6d298000 	C:\Java\jre6\bin\hpi.dll
0x690a0000 - 0x690ab000 	C:\WINNT\system32\PSAPI.DLL
0x6d7f0000 - 0x6d7ff000 	C:\Java\jre6\bin\zip.dll
0x48b80000 - 0x48c51000 	C:\Tomcat\bin\tcnative-1.dll
0x75030000 - 0x75044000 	C:\WINNT\system32\WS2_32.dll
0x75020000 - 0x75028000 	C:\WINNT\system32\WS2HELP.DLL
0x74ff0000 - 0x75002000 	C:\WINNT\system32\MSWSOCK.dll
0x77980000 - 0x779a5000 	C:\WINNT\system32\DNSAPI.dll
0x75050000 - 0x75058000 	C:\WINNT\system32\WSOCK32.dll
0x7ca00000 - 0x7ca23000 	C:\WINNT\system32\rsaenh.dll
0x7ce20000 - 0x7cf0f000 	C:\WINNT\system32\ole32.dll
0x7c0f0000 - 0x7c154000 	C:\WINNT\system32\USERENV.dll
0x7c740000 - 0x7c7cc000 	C:\WINNT\system32\CRYPT32.dll
0x77430000 - 0x77441000 	C:\WINNT\system32\MSASN1.dll
0x74fd0000 - 0x74fed000 	C:\WINNT\system32\msafd.dll
0x75010000 - 0x75017000 	C:\WINNT\System32\wshtcpip.dll
0x6d610000 - 0x6d623000 	C:\Java\jre6\bin\net.dll
0x782c0000 - 0x782cc000 	C:\WINNT\System32\rnr20.dll
0x77340000 - 0x77353000 	C:\WINNT\system32\iphlpapi.dll
0x77520000 - 0x77525000 	C:\WINNT\system32\ICMP.dll
0x77320000 - 0x77337000 	C:\WINNT\system32\MPRAPI.dll
0x75150000 - 0x75160000 	C:\WINNT\system32\SAMLIB.DLL
0x7cdc0000 - 0x7ce10000 	C:\WINNT\system32\NETAPI32.DLL
0x7c340000 - 0x7c34e000 	C:\WINNT\system32\Secur32.dll
0x77bf0000 - 0x77c01000 	C:\WINNT\system32\NTDSAPI.dll
0x77950000 - 0x7797b000 	C:\WINNT\system32\WLDAP32.DLL
0x751c0000 - 0x751c6000 	C:\WINNT\system32\NETRAP.dll
0x779b0000 - 0x77a4c000 	C:\WINNT\system32\OLEAUT32.DLL
0x773b0000 - 0x773df000 	C:\WINNT\system32\ACTIVEDS.DLL
0x77380000 - 0x773a3000 	C:\WINNT\system32\ADSLDPC.DLL
0x77830000 - 0x7783e000 	C:\WINNT\system32\RTUTILS.DLL
0x77880000 - 0x7790e000 	C:\WINNT\system32\SETUPAPI.DLL
0x774e0000 - 0x77514000 	C:\WINNT\system32\RASAPI32.dll
0x774c0000 - 0x774d1000 	C:\WINNT\system32\rasman.dll
0x77530000 - 0x77552000 	C:\WINNT\system32\TAPI32.dll
0x77360000 - 0x77379000 	C:\WINNT\system32\DHCPCSVC.DLL
0x777e0000 - 0x777e8000 	C:\WINNT\System32\winrnr.dll
0x777f0000 - 0x777f5000 	C:\WINNT\system32\rasadhlp.dll
0x6d000000 - 0x6d14a000 	C:\Java\jre6\bin\awt.dll
0x77800000 - 0x7781e000 	C:\WINNT\system32\WINSPOOL.DRV
0x76620000 - 0x76631000 	C:\WINNT\system32\MPR.DLL
0x75e60000 - 0x75e7a000 	C:\WINNT\system32\IMM32.dll

VM Arguments:
jvm_args: -Dcatalina.home=C:\Tomcat -Dcatalina.base=C:\Tomcat -Djava.endorsed.dirs=C:\Tomcat\common\endorsed -Djava.io.tmpdir=C:\Tomcat\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Tomcat\conf\logging.properties vfprintf -Xms512m -Xmx1024m 
java_command: <unknown>
Launcher Type: generic

Environment Variables:
PATH=d:\d2k\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\Program Files\Support Tools\;d:\d2k\jdk\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 7 Stepping 10, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows 2000 Build 2195 Service Pack 4

CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Client VM (14.2-b01) for windows-x86 JRE (1.6.0_16-b01), built on Jul 31 2009 11:26:58 by "java_re" with MS VC++ 7.1

time: Wed Mar 31 09:08:33 2010
elapsed time: 21683 seconds
Comment 15 Konstantin Kolinko 2010-04-02 10:06:38 UTC
(In reply to comment #14)
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #

I see that the above is also posted separately, as bug 49038. There is no explanation, how that is related to this issue, so I assume this is submitter's fault to post it here.

He also changed Product/Component/Version/Platform for this issue.

Please ignore comment #14, see bug 49038 if you want to follow it. I am resetting Product etc. to their previous values.
Comment 16 Paul Hinds 2010-06-22 04:05:20 UTC
FYI 

We are now running succesfully with modjk .30 , no reuse, no reties, and a new tomcat 4.3.0_CP07, provided by JBoss.  

This combination seems to be OK, and we have not seen further issues in production systems.   I can not confirm that the jk upgrade is what made the difference since we upgraded both modjk and TC at the same time.  

We were unable to duplicate the issues in test environments.
Comment 17 Konstantin Kolinko 2014-06-03 22:20:34 UTC
(In reply to Paul Hinds from comment #16)
Thank you for follow-up.

No reports of such issues for the current version of mod_jk.
Closing as WORKSFORME.
Comment 18 Victor 2014-07-24 14:57:49 UTC
We are seeing this issue with one of our new products.  This product produces many more requests than previous ones.  We had two instances this week where a users session was misplaced, or there data files showed up in a list within another users browser.

I'm trying to reproduce using a command line tool.  I haven't used JMeter.
Comment 19 Kevin J Walters 2014-07-24 15:55:20 UTC
My general conclusion in this area is anything that's effectively doing rpc should be using some sort of id to match requests and responses and allow the client to discard/warn/retry on mismatches. This feature can be found in sun rpc from 1988 and probably earlier implementations, it's not a new concept.

This type of bug where the the wrong response ends up with the client and in this case often the web browser can be extremely hazardous. It's a worryingly common bug on connection based transports, pools of connections and multi-threaded clients.
Comment 20 Victor 2014-07-28 18:28:43 UTC
(In reply to Kevin J Walters from comment #19)
> My general conclusion in this area is anything that's effectively doing rpc
> should be using some sort of id to match requests and responses and allow
> the client to discard/warn/retry on mismatches. This feature can be found in
> sun rpc from 1988 and probably earlier implementations, it's not a new
> concept.
> 
> This type of bug where the the wrong response ends up with the client and in
> this case often the web browser can be extremely hazardous. It's a
> worryingly common bug on connection based transports, pools of connections
> and multi-threaded clients.

Kevin,

Thank you for your advice.  Do your represent Apache?  I.E. Is this Apache.org's official advice to users? 

Victor
Comment 21 Christopher Schultz 2014-07-28 19:33:46 UTC
(In reply to Victor from comment #20)
> Kevin,
> 
> Thank you for your advice.  Do your represent Apache?  I.E. Is this
> Apache.org's official advice to users? 

There is no "Apache.org official advice to users." If you wish to discuss this more, please take the discussion to the users' list. It seems the original bug report here has been satisfied. Victor, if you have new evidence that Tomcat and/or mod_jk has a bug, please post them on the users' list and we'll let you know if you should file another bug report.

Usually this kind of thing turns out to be a bug in the web application, but it does not rule-out a but in Tomcat/mod_jk.

Resetting resolution to FIXED WORKSFORME.
Comment 22 Ben Erridge 2014-09-08 14:44:07 UTC
This needs to be reopened it happens daily on our production system we are using the latest mod_jk 1.2.40

I also turned off all retrying and connection reuse.

switching to mod_proxy resolves the issue.
Comment 23 Sudhir Reddy 2014-09-08 15:05:05 UTC
I was unable to reproduce, but describing the scenario seen just in case it helps in reproducing for others ...

I had seen multiple instances of this happening(Response of one of the previous request sent to another request) on our production servers.

We were sending wrong Content-Length from Tomcat application(More than the actual length) and our tomcat logs showed the correct response size being written to output channel and Apache logs showed less size. Upon looking at Apache logs, saw that this response length and the served content matched with an earlier request.

Fixed it by sending correct Content-Length and disabled connection reuse in mod_jk.
Comment 24 Ben Erridge 2014-09-08 15:08:49 UTC
I found a post about this and the guy was nice enough to post a JMeter script that can reproduce the issue. I have not tested it yet but looks promising.

http://jeecookbook.blogspot.com/2013/07/modjk-session-mixed-between-users.html
Comment 25 Christopher Schultz 2014-09-09 16:16:19 UTC
(In reply to Ben Erridge from comment #22)
> This needs to be reopened it happens daily on our production system we are
> using the latest mod_jk 1.2.40
> 
> I also turned off all retrying and connection reuse.
> 
> switching to mod_proxy resolves the issue.

What version of Tomcat are you using?
Comment 26 Ben Erridge 2014-09-09 16:44:14 UTC
We are using Glassfish V3.1.2.2 in production. We have the same application running in other datacenters with other Load Balancers, none have exhibited this behaviour except the one fronted by Mod_JK.

I have screen shots clearly showing response mix ups. Also because glassfish sends set cookie each response users often take on the identity of other users. This was happening 14-20 times a day. We have a session origin filter which detects when sessions span request IP's but this is not always effective when many users have the same source ip.

Once we switched to mod_proxy this never happens.

I much prefer mod_jk to mod_proxy so I am happy to provide any details necessary to getting to the bottom of this.

I did setup a test platform and make a simple JMeter test but have been unsuccessful in replicating the issue. Our production environment is much more complex than my test environment was. 

Also I tried to contact  Olivier Schmitt who is the gentlemen who has been successful in replicating this in a test environment; But as yet he has not responded to my inquiry about his test rig.
Comment 27 Rami Stefanidis 2014-10-17 15:18:03 UTC
.
Comment 28 Philippe Mouawad 2015-03-14 14:04:54 UTC
(In reply to Ben Erridge from comment #24)
> I found a post about this and the guy was nice enough to post a JMeter
> script that can reproduce the issue. I have not tested it yet but looks
> promising.
> 
> http://jeecookbook.blogspot.com/2013/07/modjk-session-mixed-between-users.
> html

Hi,
the jmeter script (session.bsh) is in fact not correct to detect the session mix as it just compared Request jsessionid cookie et response jsession cookie header setting.
But this can be a regular case when during a load test 1 tomcat becomes unresponsive, mod_jk will direct the request to another Tomcat leading to creation of a new JSESSIONID and you end up having a "False Session Mix".

Regards
Philippe M.
Comment 29 Philippe Mouawad 2015-03-14 14:07:06 UTC
(In reply to Ben Erridge from comment #26)
> We are using Glassfish V3.1.2.2 in production. We have the same application
> running in other datacenters with other Load Balancers, none have exhibited
> this behaviour except the one fronted by Mod_JK.
> 
> I have screen shots clearly showing response mix ups. Also because glassfish
> sends set cookie each response users often take on the identity of other
> users. This was happening 14-20 times a day. We have a session origin filter
> which detects when sessions span request IP's but this is not always
> effective when many users have the same source ip.
> 
> Once we switched to mod_proxy this never happens.
> 
> I much prefer mod_jk to mod_proxy so I am happy to provide any details
> necessary to getting to the bottom of this.
> 
> I did setup a test platform and make a simple JMeter test but have been
> unsuccessful in replicating the issue. Our production environment is much
> more complex than my test environment was. 
> 
> Also I tried to contact  Olivier Schmitt who is the gentlemen who has been
> successful in replicating this in a test environment; But as yet he has not
> responded to my inquiry about his test rig.


In fact the blog :
http://jeecookbook.blogspot.fr/2013/07/modjk-session-mixed-between-users.html
does not really reproduce the case as the way of testing the Mix can trigger false positive as per my previous comment at 2015-03-14 14:04:54 UTC.

If you are really able to reproduce it here, it would be nice to provide a full test case with if possible a simple usable application.

Thanks

Philippe M.