Daniel, please post the headers that your web browser is sending to Apache
httpd. Also, please tell us the version of mod_jk (or mod_proxy_ajp) you are using.

Christ, I don't use web browser to send data to apache httpd. How do I display
the  httpd header? Can you tell me a command how to do that, turn on the debug
or something?

The version of mod_proxy_ajp is Apache/2.2.3
The version of mod_jk is I don't know how to see them?

You need to tell us if you are using mod_jk or mod_proxy_ajp, because there is
some question as to which component contains this problem.

Check your Apache httpd configuration for how you connect Apache httpd to
Tomcat. If you are using mod_proxy_ajp, then you will have configuration
directives like "ProxyPass /myApp ajp://localhost:8080/myApp". If you are using
mod_jk, you will have directives such as "JkMount".

If you are using mod_jk, we'll need to know the version number. Sometimes the
name of the module being loaded into Apache httpd has the version number right
in the filename (i.e. mod_jk_1.2.13.so). If not, you can look into the file to
find out what version you have:

$ strings /usr/lib/apache2-extramodules/mod_jk.so | grep "mod_jk/"
[output]
mod_jk/1.2.18

The version I have checked here is 1.2.18. You should upgrade to the latest
version of mod_jk (currently 1.2.23) if possible.

I believe I am using mod_proxy_ajp connector because this is what I have on my
httpd.conf:
ProxyPass /axis ajp://192.168.11.10:8009/axis
ProxyPassReverse /axis ajp://192.168.11.10:8009/axis

The version of mod_proxy_ajp is Apache/2.2.3

(In reply to comment #2)
> Christ, I don't use web browser to send data to apache httpd. How do I 
display
> the  httpd header? Can you tell me a command how to do that, turn on the 
debug
> or something?
> The version of mod_proxy_ajp is Apache/2.2.3
> The version of mod_jk is I don't know how to see them?

Since it sounds like your configuration is pretty much just out of the box, 
then edit $CATALINA_HOME/conf/logging.properties and add a line that looks 
like:

org.apache.jk.level = FINE

Since we only care about the Content-Length, doing something like:

LogFormat "%h %l %u %t \"%r\" %>s %b %{content-length}i"

in httpd.conf should be enough.

(In reply to comment #3)
> The version I have checked here is 1.2.18. You should upgrade to the latest
> version of mod_jk (currently 1.2.23) if possible.

Can you actually read all of the request body?  It shouldn't throw an 
exception with a 3GB request body while parsing the header, and you should get 
dropped connections (at least if you aren't using the latest from SVN), but it 
looks to me like mod_jk will truncate the input (unless sizeof(int) >= 16 on 
your platform).

Created attachment 20361 [details]
Rough patch for http 2.2.x only

mod_proxy_ajp is now fine with POSTs > 2GB
mod_jk fails to send any content for POSTs > 2GB

I have attached a patch that fixes mod_jk for me but:
- it only fixes this for httpd 2.2.x
- I haven't looked at any of the other servers
- I don't trust my c coding skills enough to commit it

Hopefully, one of the mod_jk folks will pick this up and turn it into a
reasonable patch.

Changing component as testing shows mod_proxy_ajp now works but mod_jk still fails.

(In reply to comment #7)
> Created an attachment (id=20361) [edit]
> Rough patch for http 2.2.x only
> mod_proxy_ajp is now fine with POSTs > 2GB
> mod_jk fails to send any content for POSTs > 2GB
> I have attached a patch that fixes mod_jk for me but:
> - it only fixes this for httpd 2.2.x
> - I haven't looked at any of the other servers
> - I don't trust my c coding skills enough to commit it
> Hopefully, one of the mod_jk folks will pick this up and turn it into a
> reasonable patch.


It won't compile for httpd 1.3.x or IIS.  But it looks like a 
s/apr_off_t/jk_uint64_t/g should turn into a resonable patch.

Quibbles:  
1) The jk_ajp12_worker.c is no longer supported, so that one can be dropped 
(but it is harmless to include it).
2) There look like there are a couple of tab-to-space problems in the patch.

I'm also in it, but didn't yet test my patch. It will use jk_uint64_t anf the
apache parts are not very far away from your patch.

I will commit after my tests, but then we will still need to test the Windows
and IIS+Netscape parts.

I committed a patch for mod_jk. A dev-tarball of mod_jk can be found at

  http://people.apache.org/~rjung/mod_jk-dev/HugeContent/

This needs to be tested well, especially in combination with Bill Barker's
patches for the Tomcat connectors.

Please report any test results here.

I have a simple test that submits 4G data and counts the bytes received.

Tested with:
Tomcat 5.5.HEAD, Win XP SP2 Home, using Java APR connector.

Apache 1.3.37, mod_jk 1.2.24-dev built locally with httpd on the same box as
Tomcat fails with a 400 - Invalid content length. The error appears to be httpd
generated.

Apache 2.2.4, mod_jk 1.2.24-dev built locally with httpd on the same box as
Tomcat works (all 4G data is received)

IIS 5, Win2k running on VMWare, mod_jk 1.2.24-dev built locally is *very* slow
for smaller files and fails completely (connection is dropped) for 4G files.

Mark, thanks a lot.

I think the failing with httpd 1.3 is not that bad. It's very outdated now and
support for huge content could be handled as yet another reason to update. From
the mod_jk perspective everything is analogous, but it relies an the web server
being able to accept such a large value in the content length header.

Concerning IIS: The IIS special part of the patch is close to being non
existant. All the relevant parts are the same as for IIS. So I strongly assume,
that the slowlyness hasn't been introduced by this fix. Maybe it's a VMWare problem.

I hope Mladen will retest IIS basic performance to double check.

Thanks again for the test!

I'll let this BZ open and we can close it in a couple of days, if no negative
side effects come up.

Now I did some tests as well. I tested downloads and not uploads. Three test cases:

a) size a little below 2^31 Bytes
b) size a little above 2^31 Bytes
c) size a little above 2^32 Bytes

Platform Solaris 10 Sparc, Tomcat 5.5.24 candidate using Java APR connector,
mod_jk 1.2.24-dev.

All Apache builds 32 Bit binaries.

Apache 1.3.37: a: OK, b: OK, but "0" as size in access log, c: OK, but size mod
2^ 32 in access log

Apache 2.0.59: a: OK, b: OK, but negative size in access log, c: OK, but size
mod 2^ 32 in access log

Apache 2.2.4: a,b,c: OK, correct sizes in access log

But: as soon as the size goes over 2^31, I don't get a Content-Length header
back. I needed to fake a little: when I use static content Tomcat tries to load
it into memory, so I used a little servlet to produce the download content on
the fly. But the servlet API call to set the content length header only allows
an int as the argument. I instead used the setHeader method for Content-Length.
This works well below 2^31, but above the Tomcat AJP connector does not handle
the header back to mod_jk. Nevertheless wget could retrieve the complete
message, and I *think* Tomcat transparently switched to transfer encoding chunked.

Same test on same system with Sun Webserver 6.1 SP 7 64Bit: a,b,c are OK with
correct length in access log, but again the Content-Length header seems not to
come back from Tomcat when size is bigger than 2^31.

Download speed was the same for Sun, Apache 1.3, 2.0 and 2.2 (12MB/s; of course
this is not a performance test, it's just one huge download on a slow machine).
Using the direct http connector of Tomcat resulted in app. 15% faster download.

The absoulte number is not important (the servlet was doing cracy stuff, so
generating the content was CPU bound).

The Tomcat access log by the way, shows correct length below 2^31, it shows a
"-" for case b and the size mod 2^32 for case c.