Bug 36155 - tomcat chooses wrong host if using mod_jk
tomcat chooses wrong host if using mod_jk
Status: RESOLVED FIXED
Product: Tomcat Connectors
Classification: Unclassified
Component: Common
unspecified
All All
: P2 normal (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2005-08-12 05:08 UTC by Sven
Modified: 2011-04-11 16:23 UTC (History)
0 users



Attachments
testcase for the problem i described (42.46 KB, application/x-compressed-tar)
2006-01-12 20:58 UTC, Sven
Details
patch for this bug (1.45 KB, patch)
2007-03-02 22:00 UTC, Sven
Details | Diff
new patch (1.86 KB, patch)
2007-03-03 15:53 UTC, Sven
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sven 2005-08-12 05:08:52 UTC
Hi,

my config is as follows:

tomcat 5.5.9 knows two hosts, let's say the host "host1" and the host "host2".
http://host1/ is basically a jsp-page echoing the string host1 - http://host2/
echos the string host2.

apache 2.0.54 is configured with two vhost with ServerName host1 and host2. Both
servers have a ServerAlias too - the one has ServerAlias alias1, the other has
ServerAlias alias2

mod-jk 1.2.14 only knows one worker. My worker.properties looks like this:
worker.list=worker1
worker.worker1.type=ajp13
worker.worker1.host=127.0.0.1

A very simple config. Both apache vhosts contain a "JkMount *.jsp worker1"

Apache's UseCanonialName is on and Tomcats AJP-connector has useIPVHosts turned
on. Therefor tomcat will chose the host by the server-name that mod_jk sends.

So calling http://alias1/ results in a page containing the string host1, and
calling http://alias2/ results in a page containing the string host2.

The problem:
Sometimes (not always and very selden) http://alias1 returns the string host2
and http://alias2 returns the string host1.

I'm not sure, where the problem is, but i'm almost sure it's mod_jk - although i
don't know, what should go wrong.

I now defined a second worker with the same address and each of apache's vhosts
use a different worker. The error didn't come up yet.
Comment 1 patrick ong 2005-12-25 18:23:29 UTC
Dont know whether related or not.
Migrated some old 4.0 serlvet to 4.1 with mod_jk - 1.2.0

After I upgraded mod_jk to 1.2.15
my tomcat-4.1.31 start to point to wrong host.
(the serlvet binaries are copied from another host,
 my point is Previous version did not give me problem)

patricko@staff.singnet.com.sg
Comment 2 Sven 2005-12-25 19:13:55 UTC
So the problem still exists with mod_jk 1.2.15 and Tomcat 5.5.12.

If two VirtualHosts use the same worker, Tomcat receives the wrong ServerName.
IMHO, it has something todo with some cache or some other flaw.
(The caches are worker-wise, aren't they?)

Does somebody care about this?

@Patrick: i don't know if your problem is related. For me, the problem is
reproducable, but it does not go wrong always:
first HTTP-request works, second doesn't, third works again, some random ...
Comment 3 Yoav Shapira 2006-01-10 20:12:50 UTC
Patrick, I don't think your problem is related.  Try users@tomcat.apache.org for
help.  Sven, I can't reproduce your problem.  If you're still experiencing it
and can post the configuration files for Apache, Tomcat, mod_jk, please do.
Comment 4 Sven 2006-01-10 20:29:35 UTC
I'll do my best, to reproduce it with a small testcase.
Comment 5 Sven 2006-01-12 20:58:04 UTC
Created attachment 17404 [details]
testcase for the problem i described

There are two hosts in tomcat: host1 and host2
They point to /tmp/host1 and /tmp/host2 (which is also in the tgz)
There are two hosts in apache: host1/alias1 and host2/alias2
There are two workers defined, but only one is in use from both virtualhosts
Put the line "127.0.0.1 host1 host2 alias1 alias2" in /etc/hosts

If you run "./test.sh alias1" and "./test.sh alias2" in two terminals you
should see only "host1" in one window, and "host2" in the other window.

What i experience is, that "./test.sh alias2" output "host1" sometimes - or
"./test.sh alias1" outputs "host2". When i change apache's virtualhosts to use
two different workers, the problem is gone.
Comment 6 Sven 2006-01-12 21:43:42 UTC
you can also start "./test.sh host1" and "./test.sh host2" with the same strange
result: "./test.sh host1" sometimes outputs "host2", and "./test.sh host2"
sometimes outputs "host1"
Comment 7 Dan Carwin 2006-01-17 19:24:16 UTC
Sven,
 Both workers on the same port? Why not match the "host" name in
 workers.properties to the "host" container name in server.xml?  Why point
 both at 127.0.0.1?

 Seems the proper config is to use names for the workers' host values instead
 of the same (port and) ip addy in both cases.

-Dan
Comment 8 Sven 2006-01-18 01:15:40 UTC
@Dan: i want to use only one worker - and the config attached to this bug only
uses one worker - though two are declared.
Using two workers that point to the same tomcat is a workaround for me, so that
the bug described here doesn't rise.
I don't understand, what you mean with "using names" in the worker-file. That
doesn't make sense to me since the proper DNS-name would be "localhost" in both
cases. The AJP-Connector only listenes on localhost. The names used in the
worker-file are _not_ sent to Tomcat by mod_jk, AFAIK.
Comment 9 Sven 2006-01-18 17:51:38 UTC
The comment by Dan sounded much like "your config cannot work at all". So i
think i need to explain a little more:

The config does work, if you:
- restart apache, then run only "./test.sh alias1": you will only see host1 on
the console
- restart apache, then run only "./test.sh alias2": you will only see host2 on
the console
- use worker1 for the one virtualhost, and use worker2 for the other, then run
"./test.sh alias1" and "./test.sh alias2" in parallel: both scripts return the
right results.

I hope that you agree with me, that tomcat chooses the hosts like it should.
Note, that the server.xml says useIPVHosts="true" - so tomcat does AFAIK not
look at the host-header, but at some other information that mod_jk sends. Since
i also use "UseCanonicalName On" for apache, mod_jk is supposed to send the
ServerName declared for the virtualhost.

The config does not work, if you use the same worker for both virtualhosts -
which is "the right thing" IMHO since there is also only one tomcat running.
What do i mean, with "it does not work" - well:
Start "./test.sh alias1" and "./test.sh alias2" in parallel. If you're lucky, it
works for 10 requests and you see what you should: "host1" returned by the first
command, and "host2" returned by the second command.

The result changes for me, and the at least one command "goes crazy" and returns
something else: either the command only returns host1 instead of host2 (or vice
versa of course) or sometimes i even get an alternating sequence of host1 and
host2 by one of the commands - or even by both.

Especially the alternating output is a case, that should not occur. The choice
made by tomcat is deterministic AFAIK and tomcat is only considering the data
that it gets by mod_jk, so how can there be an alternating sequence?

My conclusion is, that mod_jk sends the wrong servername, or perhaps sending the
servername is optional and tomcat just "reuses" the information it has been
given by a former AJP-request on the same connection - but i have no clue about
those internal things.
Comment 10 Mladen Turk 2006-03-17 09:25:47 UTC
I was not able to reproduce the problem.
With your test files and mod_jk from SVN it works OK.
Comment 11 Sven 2007-03-02 20:13:11 UTC
I found the bug.
It's still present in Tomcat 6.0.10.
I will supply details soon, as well as a patch.
Comment 12 Sven 2007-03-02 21:59:03 UTC
So the bug is in the class org.apache.tomcat.util.buf.MessageBytes.

The method setBytes doesn't clear the char-buffer. And the method setChars
doesn't clear the byte-buffer.

If useIPVHosts is enabled, then the parameter "host" of the
org.apache.tomcat.util.http.mapper.map()-method is a MessageBytes-Object of the
type T_BYTES. The method MessageBytes.toChars()-method is called. But this
method exists, if the charbuffer is already filled.

And in deed, the char-buffer is already filled. While the byte-buffer contains
the value "host1", the char-buffer may still contain the value "host2" because
the MessageBytes objects are reused. And instead of converting the
byte-buffer-value "host1" to chars, the old value "host2" is used. This resulted
in the bug.

So the following patch fixes the problem.


With your permission, i would like to clean up the class MessageBytes.
It really needs it - IMHO.
Comment 13 Sven 2007-03-02 22:00:42 UTC
Created attachment 19659 [details]
patch for this bug

The patch should apply cleanly to Tomcat 6.0.10
Comment 14 Sven 2007-03-03 00:40:45 UTC
I discovered, that calling byteC.recycle() also resets the Encoding that might
have been set. But actually the setEncoding() method is used nowhere.
And calling byteC.reset() doesn't work, because the reset()-method leaves the
ByteChunk in an illegal state:  the array-pointer is set to null, but the
internal "isSet" which controlls the result of isNull() is not reset to false.

I'm quite unhappy with the class MessageBytes. For example setInt/setLong also
forget to clear the char-buffer. toBytes() ignores the charset. In many cases i
discovered strange handeling of non-ASCII characters. So i don't aim to clean it
up. There are too many side-effects that might be hurt by any modification.
Comment 15 Remy Maucherat 2007-03-03 02:00:04 UTC
-1 for the patch. If the problem exists (I really doubt it), a better solution
will have to be found.
Comment 16 Sven 2007-03-03 06:25:02 UTC
(In reply to comment #15)
> -1 for the patch. If the problem exists (I really doubt it), a better solution
> will have to be found.

Simply one question: Why?
setBytes() invalidates everything - except the value within charC is kept.

Anyway: there might be one missing recycle call somewhere. I'm already looking
for it.
Comment 17 Sven 2007-03-03 06:57:31 UTC
So in org.apache.coyote.Request.recylce(), this.localNameMB.recycle() is not
called. If i add that call, the problem is gone too (my first patch has been
reverted before testing).

The problem is: remoteAddrMB.recycle(), remoteHostMB.recycle(),
localAddrMB.recycle() is also not called. And i have no clue why not or is it
should actually be called.


Anyway: At least localNameMB gets reused without being recycled. So the internal
charC contains an old value. setBytes() doesn't clear internal the charC-cache.
toChars() reuses the old value. Hence the bug.


I admit, that the semantics of the MessageBytes class are a ridle to me. I don't
know, why setBytes() shouldn't invalidate the charC cache (even though it
invalidates all other cached values).
Somebody please explain the semantics of this class to me! And decide, whether
setBytes() should clear charC, or not.
Comment 18 Sven 2007-03-03 07:04:53 UTC
Additional information:

the value of localNameMB is set in AjpProcessor.prepareRequest() (line 645) by
calling "requestHandlerMessage.getBytes(request.localName())". This getBytes
method calls setBytes() of localNameMB.

Above that line, remoteAddrMB and remoteHostMB are also set with the same
method. And they also might not have been recycled. I don't know for sure.
Comment 19 Remy Maucherat 2007-03-03 14:45:08 UTC
(In reply to comment #17)
> And decide, whether setBytes() should clear charC, or not.

It seems you're eager to contribute and stuff, and I think that's good, but if I
said -1 to your patch, this means the answer to this question is "no". This will
cause problems, and is also inefficient.

I think this is a special case with AJP, where recycle should possibly be called
on the localName MessageBytes before reading the localName field (it should be
fairly explicit that the localName/Addr are otherwise never recycled in the
Request object). This is indeed a special case with the IP vhost feature (which
IMO is not a very good feature, but it's another story).
Comment 20 Sven 2007-03-03 15:51:41 UTC
Do you understand, why we need to recycle the localName?
(short reason: two ajp-requests with different localName over same AJP connection)


So i have the feeling, that you want this fix to be AJP specific.
I will try to fix it in that way.

I tried to fix AjpProcessor/AjpAprProcessor. But the fix only worked, when i did
it in JkCoyoteHandler. I simply changed the code from
  request.recycle();
to
  request.recycle();
  request.localName().recycle();


I don't feel well leaving the other 3 fields (remoteAddr, remoteHost, localAddr)
unrecycled. Somehow i feel, like this issue also applies to the other fields.
But i don't have enough knowledge to be sure. So i leave this up to you.


Now i will attach a new patch. Maybe you like it :-)
Comment 21 Sven 2007-03-03 15:53:02 UTC
Created attachment 19660 [details]
new patch
Comment 22 william.barker 2007-05-20 15:35:47 UTC
I've committed a more general fix for the JK Connector (i.e. 
JkCoyoteHandler).  I can port it to the APR and AJP Connectors if/when Remy 
and Mladen doen't veto it.
Comment 23 Sven 2007-05-20 18:30:57 UTC
> I've committed a more general fix for the JK Connector (i.e. 
> JkCoyoteHandler).  I can port it to the APR and AJP Connectors if/when Remy 
> and Mladen doen't veto it.

Thanks a lot!

I took a look at your commits. I think, the JK Connector is fixed now. I will
test it soon. (At the moment, i don't have much time).
Comment 24 Sven 2007-05-22 16:56:34 UTC
(In reply to comment #23)
> I took a look at your commits. I think, the JK Connector is fixed now. I will
> test it soon. (At the moment, i don't have much time).

I ran my tests, and yes: it's fixed.

Sweet!
Comment 25 Sven 2007-08-06 07:17:45 UTC
So William has made a start. His patch works, and could be ported to the other 2
connectors.

But i think he's waiting for some reaction by Remy and Mladen.
(see comment #22)

Let's get this done, so this can be marked fixed.
Comment 26 Tim Whittington 2011-04-11 16:23:08 UTC
I think this is as fixed as it's ever going to get (and the connectors have moved on a lot since this was raised).