Please provide the output of your error_log file. Sorry for being confused
regarding your size observations.
You mean requests below 2 GB are fine and request above 2 GB only work sometimes?

The proxy is cutting off data less than 2 Meg, not G.  The test URL is ~8M and
on some requests I may only get 500K.  However, sometimes (using wget, but never
firefox, it seems) I will get the whole file, though wget had to retry and
managed to get all of the content.

I went to the page:

http://wiki.dlib.indiana.edu/confluence/pages/viewpageattachments.action?pageId=25

and clicked on the boot.iso link.  The original file is 8,161,280 bytes.  These
messages were generated in the log:
[Mon Aug 27 16:37:47 2007] [error] ajp_check_msg_header() got bad signature d3e6
[Mon Aug 27 16:37:47 2007] [error] ajp_ilink_receive() received bad header
[Mon Aug 27 16:37:47 2007] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Aug 27 16:37:47 2007] [error] (120007)APR does not understand this error
code: proxy: send body failed to 156.56.241.30:8079 (feta.dlib.indiana.edu)

and only 663,544 bytes were transferred.


If I go to 
http://wiki.dlib.indiana.edu:8070/confluence/pages/viewpageattachments.action?pageId=25

which is the tomcat instance without the proxy, clicking on the boot.iso link
gives me the entire file as expected.

(In reply to comment #2)
These
> messages were generated in the log:
> [Mon Aug 27 16:37:47 2007] [error] ajp_check_msg_header() got bad signature d3e6

It seems that your Tomcat is sending wrong data. So please:

- Increase the loglevel of your Apache to debug.
- Try sniffing the ajp traffic between your Apache and your Tomcat and attach
the sniff file.

I've done the proxy from my workstation's apache (2.2.4 vs 2.2.3 of the
production server) and it still does the same thing.  Here's the error messages
generated when setting LogLevel debug and requesting the file.  I will attach a
dump from wireshark of the transaction.

[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_ajp.c(44): proxy: AJP:
canonicalising URL
//feta.dlib.indiana.edu:8079/confluence/download/attachments/25/boot.iso
[Tue Aug 28 08:20:19 2007] [debug] proxy_util.c(1378): [client 127.0.0.1] proxy:
ajp: found worker ajp://feta.dlib.indiana.edu:8079/confluence/ for
ajp://feta.dlib.indiana.edu:8079/confluence/download/attachments/25/boot.iso,
referer: http://localhost/confluence/pages/viewpageattachments.action?pageId=25
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy.c(777): Running scheme ajp handler
(attempt 0)
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_http.c(1652): proxy: HTTP:
declining URL
ajp://feta.dlib.indiana.edu:8079/confluence/download/attachments/25/boot.iso
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_ajp.c(507): proxy: AJP: serving URL
ajp://feta.dlib.indiana.edu:8079/confluence/download/attachments/25/boot.iso
[Tue Aug 28 08:20:19 2007] [debug] proxy_util.c(1798): proxy: AJP: has acquired
connection for (feta.dlib.indiana.edu)
[Tue Aug 28 08:20:19 2007] [debug] proxy_util.c(1859): proxy: connecting
ajp://feta.dlib.indiana.edu:8079/confluence/download/attachments/25/boot.iso to
feta.dlib.indiana.edu:8079
[Tue Aug 28 08:20:19 2007] [debug] proxy_util.c(1955): proxy: connected
/confluence/download/attachments/25/boot.iso to feta.dlib.indiana.edu:8079
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(218): Into ajp_marshal_into_msgb
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[0] [Host] = [localhost]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[1] [User-Agent] = [Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.5)
Gecko/20070718 Fedora/2.0.0.5-1.fc7 Firefox/2.0.0.5]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[2] [Accept] =
[text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[3] [Accept-Language] = [en-us,en;q=0.5]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[4] [Accept-Encoding] = [gzip,deflate]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[5] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[6] [Keep-Alive] = [300]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[7] [Connection] = [keep-alive]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[8] [Referer] =
[http://localhost/confluence/pages/viewpageattachments.action?pageId=25]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[9] [Cookie] = [JSESSIONID=AEAB20432F03EE55B61C3DF792948B91;
wikiuserUserID=1; wikiuserUserName=Admin;
wikiuserToken=0a2ca7abde82342684d88f18c80498bf]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[10] [Max-Forwards] = [10]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(424): ajp_marshal_into_msgb: Done
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_ajp.c(188): proxy: APR_BUCKET_IS_EOS
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_ajp.c(193): proxy: data to read
(max 8186 at 4)
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_ajp.c(208): proxy: got 0 bytes of data
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 04
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 04
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(484): ajp_unmarshal_response:
status = 200
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(495): ajp_unmarshal_response:
Number of headers is = 5
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(557): ajp_unmarshal_response:
Header[0] [Last-Modified] = [Mon, 27 Aug 2007 13:52:44 GMT]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(557): ajp_unmarshal_response:
Header[1] [ETag] = ["1188222764000"]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(557): ajp_unmarshal_response:
Header[2] [Content-Disposition] = [inline; filename="boot.iso"]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(557): ajp_unmarshal_response:
Header[3] [Content-Type] = [application/x-cd-image;charset=UTF-8]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(567): ajp_unmarshal_response:
ap_set_content_type done
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(557): ajp_unmarshal_response:
Header[4] [Content-Length] = [8161280]
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(643): ajp_read_header:
ajp_ilink_received 03
[Tue Aug 28 08:20:19 2007] [debug] ajp_header.c(653): ajp_parse_type: got 03
[Tue Aug 28 08:20:19 2007] [error] ajp_check_msg_header() got bad signature 5528
[Tue Aug 28 08:20:19 2007] [error] ajp_ilink_receive() received bad header
[Tue Aug 28 08:20:19 2007] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Aug 28 08:20:19 2007] [debug] mod_proxy_ajp.c(395): (120007)APR does not
understand this error code: ajp_read_header failed
[Tue Aug 28 08:20:19 2007] [error] (120007)APR does not understand this error
code: proxy: send body failed to 156.56.241.30:8079 (feta.dlib.indiana.edu)
[Tue Aug 28 08:20:19 2007] [debug] proxy_util.c(1816): proxy: AJP: has released
connection for (feta.dlib.indiana.edu)

Created attachment 20719 [details]
pcap dump from wireshark of truncated transaction

This is the proxy request from my workstation to the tomcat server via
mod_proxy.

Thanks for the feedback. According to my first analysis of the pcap dump Tomcat
sents the data correctly. The debug messages from httpd are not sufficient for
me for further analysis. It seems that some debug messages get written to the
main server error log and not to the error log of the virtual host which you
added. Can you please also add the corresponding entries from the main server log?

I'm not sure where to look, since I didn't put the redirect into a virtual host.
 I set "LogLevel" to debug -- is there a different setting I should have used? 
What error message are you looking for?  Maybe I can grep around and see where
it went.

Sorry for creating confusion. I assume that you use a virtual host and that this
virtual host contains an ErrorLog directive. I guess you put excerpts from this
file into your report. There should be also an ErrorLog directive outside of any
virtual host and I need the appropriate messages of this file. Of course this
requires that you set LogLevel to debug outside of the virtual host.
If this is not the case could you please repeat your debugging session and
attach the messages from both logs plus the dump.
And it does not matter that the ProxyPass is outside of the virtual host. It
only matters that there is one with a separate ErrorLog directive.

Created attachment 20732 [details]
Error Log

Full error log with truncated output

I've attached the full error log from a startup and a few hits ending with the
truncated output download.  This should be all the logging messages there are.

Thanks for the quick response, but I see no failing request in your error_log.
Your request to /confluence/download/attachments/25/boot.iso results in a 304
(not modified). Please clear the cache of your browser and try again.

Forgot one thing: I am sorry but I need a network dump AND the error_log for the
same failing request as I need to analyse this data together. Sorry for the
inconvenience.

Created attachment 20740 [details]
packet dump for truncated request (part 1 of 2)

Here's a packet dump from the startup of apache to after the truncated request.
 I had to split it due to size.

Created attachment 20741 [details]
packet dump for truncated request (part 2 of 2)

This is part 2 of 2.

Created attachment 20742 [details]
Error log for truncated transaction

This is the error log in question

Thanks for the update. I am still missing the debug message I expected :-(. This
is not your fault. I do not know why it does not show up. I need to think about it.

Are there any thoughts on this issue? Brian, did you resolve this?

I resolved it in a round-about sort of way:  I used the http proxy instead of
the ajp proxy.  

We're seeing this problem too. The tomcat configuration works fine with Apache
1.3, but we recently decided to move to Apache2 and it's broken for large files
sent with AJP.

This is with Apache 2.2.4 on gentoo, tomcat 5.5.25, so slightly more up to date
than Brian's case, and it's still there.

This is preventing us moving to Apache2 at the moment...

Some more testing on our systems:

* truncation happens reliably on any ajp response page longer than 1 MB, and
intermittently on responses in the 100 - 500 kB range (the longer the response,
the more likely to have a problem)
* problem is there for apache 2.2.6 server (latest download) as well.
* However, the problem is only there when the tomcat server is on a linux box.
Running the identical tomcat server on a Mac OS X box resulted in no ajp
truncation problems.
* We've tried this with apache-tomcat-5.5.20 and 5.5.25, same issue.
* on the apache side in debug mode we're seeing the following error messages
when it happens:
    [error] ajp_check_msg_header() got bad signature 3d22
    [error] ajp_ilink_receive() received bad header
    [error] ajp_read_header: ajp_ilink_receive failed
...

except the '3d22' may be some other random 4-hex number
* on the tomcat side we're seeing an IOException (5.5.25):
    at org.apache.coyote.ajp.AjpAprProcessor.flush(AjpAprProcessor.java:1199)
    at
org.apache.coyote.ajp.AjpAprProcessor$SocketOutputBuffer.doWrite(AjpAprProcessor.java:1284)
    at org.apache.coyote.Response.doWrite(Response.java:560)
    at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:352)
...

though presumably tomcat is seeing the IO error because apache has closed the
connection?

However, the problem is gone when we use tomcat 6.0.14. Maybe time to switch to
that... It would be nice to know what was causing this issue though!

What AJP connector did you use with 5.5.x? Despite one of my first comments I
now think that the Tomcat connector sends wrong data, but I did not have the
time to check this further and prove it.. It would be useful to know whether you
used the "classic" blocking pure java ajp connector or the APR ajp connector.

(In reply to comment #22)
> What AJP connector did you use with 5.5.x? Despite one of my first comments I
> now think that the Tomcat connector sends wrong data, but I did not have the
> time to check this further and prove it.. It would be useful to know whether you
> used the "classic" blocking pure java ajp connector or the APR ajp connector.

Here's the server.xml line:
    <Connector port="8009" enableLookups="false" redirectPort="8443"
protocol="AJP/1.3" />

- i.e. no mention of APR. On the other hand, the tomcat stack trace started in:

org.apache.coyote.ajp.AjpAprProcessor

which sounds like an APR version of the connector. I'm afraid I'm not familiar
enough with that to know if that answers the question or not! 

Ah, if APR = Apache Portable Runtime, then no, we're not using that at the
moment, just pure java on the tomcat side.

(In reply to comment #23)

> 
> Here's the server.xml line:
>     <Connector port="8009" enableLookups="false" redirectPort="8443"
> protocol="AJP/1.3" />

This should use the APR connector if the APR libraries are present. If not the
pure java connector is used.

> 
> - i.e. no mention of APR. On the other hand, the tomcat stack trace started in:
> 
> org.apache.coyote.ajp.AjpAprProcessor

Ups. My fault I should have read your answer more closely. So yes, you used the
APR (= Apache Portable Runtime) connector in the case where the error occurred.
Maybe you only used the APR connector on Linux and not on Mac OS (as the APR
libraries were possibly not present on Mac OS).

Yup, APR's the problem! Those tricky sysadmin's, I hadn't realized it was set up
on our server.

I removed the /usr/lib/libtcnative* libraries, restarted the tomcat-5.5.20
server, and the truncation problem was gone.  Restored them, and the truncations
are back.

So, who's responsible for the APR AJP stuff?!

It turns out we didn't have the latest 'tomcat-native' package for gentoo. The
one that was causing problems was version 1.1.7. We've now updated to
tomcat-native-1.1.10 and the truncation problem is gone!

It seems likely this was the same problem Brian was having - should probably
confirm this, but if so this bug can probably be considered resolved. Thanks.

No further feedback. I assume it was a bug in tcnative and upgrading it fixed
the problem.

(In reply to comment #28)
> No further feedback. I assume it was a bug in tcnative and upgrading it fixed
> the problem.

Hmm, I'm afraid we are still seeing this here with tcnative 1.12.
When starting tomcat with "-Djava.library.path=/usr/local/apr/lib" we get:

ajp_check_msg_header() got bad signature 420
[error] ajp_ilink_receive() received bad header
[error] ajp_read_header: ajp_ilink_receive failed
[error] (120007)APR does not understand this error
code: proxy: send body failed to 127.0.0.1:8009

When starting tomcat without this env variable it runs fine.

- Redhat EL4 x86_64 U6, kernel 2.6.9-67.0.7.ELsmp x86_64

- java version "1.5.0_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_13-b05, mixed mode) 

- Apache Tomcat/5.5.25

- ./httpd -V
Server version: Apache/2.2.8 (Unix)
Server built:   Mar 31 2008 15:17:07
Server's Module Magic Number: 20051115:11
Server loaded:  APR 1.2.12, APR-Util 1.2.12
Compiled using: APR 1.2.12, APR-Util 1.2.12
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

- # ll /usr/local/apr/lib/
insgesamt 2316
-rw-r--r--  1 root root 1502768 31. Mär 15:24 libtcnative-1.a
-rwxr-xr-x  1 root root     953 31. Mär 15:24 libtcnative-1.la
lrwxrwxrwx  1 root root      23 31. Mär 15:24 libtcnative-1.so -> libtcnative-1.so.0.1.12
lrwxrwxrwx  1 root root      23 31. Mär 15:24 libtcnative-1.so.0 -> libtcnative-1.so.0.1.12
-rwxr-xr-x  1 root root  850307 31. Mär 15:24 libtcnative-1.so.0.1.12
drwxr-xr-x  2 root root    4096 31. Mär 15:24 pkgconfig

Can you confirm that this is actually a bug in HTTPD...

Since we're no longer using it, I can't directly confirm it is an httpd issue, but since it doesn't appear when using mod_jk or when connecting to the tomcat instance directly, it does kind of point that way.