Can you try this with 2.2.0?

(In reply to comment #1)
> Can you try this with 2.2.0?

Sorry for late response. 

I tested Apache 2.2.0, and error again with about 6000 simultaneous 
connections, though the error messages were slightly different. 

[Mon Dec 19 10:41:03 2005] [error] (110)Connection timed out: proxy: AJP: 
attempt to connect to 127.0.0.1:8009 (localhost) failed
[Mon Dec 19 10:41:03 2005] [error] proxy: AJP: failed to make connection to 
backend: localhost
[Mon Dec 19 10:41:04 2005] [error] (110)Connection timed out: proxy: AJP: 
attempt to connect to 127.0.0.1:8009 (localhost) failed
[Mon Dec 19 10:41:04 2005] [error] proxy: AJP: failed to make connection to 
backend: localhost
[Mon Dec 19 10:41:04 2005] [error] (110)Connection timed out: proxy: AJP: 
attempt to connect to 127.0.0.1:8009 (localhost) failed
[Mon Dec 19 10:41:04 2005] [error] proxy: AJP: failed to make connection to 
backend: localhost
[Mon Dec 19 10:41:07 2005] [error] (104)Connection reset by peer: 
ajp_ilink_receive() can't receive header
[Mon Dec 19 10:41:07 2005] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Dec 19 10:41:07 2005] [error] (120006)APR does not understand this error 
code: proxy: read response failed from (null) (localhost)
[Mon Dec 19 10:41:08 2005] [error] (104)Connection reset by peer: 
ajp_ilink_receive() can't receive header
[Mon Dec 19 10:41:08 2005] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Dec 19 10:41:08 2005] [error] (120006)APR does not understand this error 
code: proxy: read response failed from (null) (localhost)
....
....
[Mon Dec 19 10:42:55 2005] [error] (104)Connection reset by peer: 
ajp_ilink_receive() can't receive header
[Mon Dec 19 10:42:55 2005] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Dec 19 10:42:55 2005] [error] (120006)APR does not understand this error 
code: proxy: read response failed from (null) (localhost)
[Mon Dec 19 10:42:57 2005] [error] (104)Connection reset by peer: 
ajp_ilink_receive() can't receive header
[Mon Dec 19 10:42:57 2005] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Dec 19 10:42:57 2005] [error] (120006)APR does not understand this error 
code: proxy: read response failed from (null) (localhost)

In TOMCAT5.5.9 log, found 
"Dec 19, 2005 10:18:04 AM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (8,000) are currently busy, waiting. Increase maxThreads 
(8,000) or check the servlet status"

If using mod_jk, works perfectly!

Thx, Xuekun

Please execute the following command if the problem occurs again and post the
result of:

netstat -tn | egrep "^tcp.*:8009" | tr -s " " | cut -f6 -d " " | sort | uniq -c

What is your ulimit -n setting for httpd and Tomcat?
You said it works with mod_jk. Please let me know which version of mod_jk and
post your worker.properties please.

(In reply to comment #3)
> Please execute the following command if the problem occurs again and post the
> result of:
> netstat -tn | egrep "^tcp.*:8009" | tr -s " " | cut -f6 -d " " | sort | uniq -
c
  16213 ESTABLISHED
   1240 SYN_SENT

wowo...

> What is your ulimit -n setting for httpd and Tomcat?

110490

> You said it works with mod_jk. Please let me know which version of mod_jk and
> post your worker.properties please.

mod_jk 1.2.14

workers.tomcat_home=/root/jakarta-tomcat-5.5.9
workers.java_home=$JAVA_HOME
ps=/
worker.list=ajp13
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009

Thx, Xuekun

(In reply to comment #4)
> (In reply to comment #3)
> > Please execute the following command if the problem occurs again and post the
> > result of:
> > netstat -tn | egrep "^tcp.*:8009" | tr -s " " | cut -f6 -d " " | sort | uniq -
> c
>   16213 ESTABLISHED
>    1240 SYN_SENT
>

Hmm. Strange. I currently do not see how that much connections could be created with

MaxClients         24000
ServerLimit     50
ThreadsPerChild     600

and

ProxyPass /jspdir ajp://localhost:8009/jspdir max=100

which sets max to 100. This should result in 50 * 100 = 5000 connections at
most. Just a guess right now but maybe somewhere in the proxy module the socket
does not get closed properly. Needs some more investigation.

> 
> workers.tomcat_home=/root/jakarta-tomcat-5.5.9
> workers.java_home=$JAVA_HOME
> ps=/
> worker.list=ajp13
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009

That looks like you only use one connection to Tomcat per process as the
cachesize property defaults to 1. That would be 50 connections to Tomcat in
total at max. So this is not comparable to the test above. Try setting

worker.ajp13.cachesize=100

and test again.

(In reply to comment #5)
> most. Just a guess right now but maybe somewhere in the proxy module the 
socket
> does not get closed properly. Needs some more investigation.

> That looks like you only use one connection to Tomcat per process as the
> cachesize property defaults to 1. That would be 50 connections to Tomcat in
> total at max. So this is not comparable to the test above. Try setting
> worker.ajp13.cachesize=100
> and test again.

When setting worker.ajp13.cachesize=100, all my testings are passed without 
errors. 

And I also checked the connections to localhost:8009 with your commands, the 
max number is about 600, so the max connections is about 300.  

Thx, Xuekun

The main difference that I see between mod_proxy_ajp and mod_jk is that 
is_socket_connected in mod_jk does (translated to APR):
   if(APR_STATUS_IS_EAGAIN(socket_status))
      return 1;
   else
      return 0;

instead of:
   if(APR_STATUS_IS_EOF(socket_status))
      return 0;
   else
      return 1;

On my Solaris test box, socket_status is only ever EOF or EAGAIN, so it makes 
no difference.  I don't know how generally this is true.

mod_jk will also try to re-establish the connection if the first response from 
Tomcat fails, but the window between is_socket_connected and ajp_send_header 
has to be unbelievably small.

(In reply to comment #7)
> mod_jk will also try to re-establish the connection if the first response 
from 
> Tomcat fails, but the window between is_socket_connected and ajp_send_header 
> has to be unbelievably small.

Just for chuckles, can you re-try your mod_jk tests with including:
  worker.ajp13.recovery_options=3

in your worker.properties file?  What this will tell us is if mod_jk really is 
actually recovering during the window between is_socket_connected and 
ajp_send_header.

What the h*ll, I want to know ;-)

(In reply to comment #9)
> What the h*ll, I want to know ;-)

? Sorry, can you elaborate what do you want to know? 

(In reply to comment #10)
> (In reply to comment #9)
> > What the h*ll, I want to know ;-)
> ? Sorry, can you elaborate what do you want to know? 

Adding myself to the CC list, so I get notified of changes.

(In reply to comment #8)
> (In reply to comment #7)
> > mod_jk will also try to re-establish the connection if the first response 
> from 
> > Tomcat fails, but the window between is_socket_connected and 
ajp_send_header 
> > has to be unbelievably small.
> Just for chuckles, can you re-try your mod_jk tests with including:
>   worker.ajp13.recovery_options=3
> in your worker.properties file?  What this will tell us is if mod_jk really 
is 
> actually recovering during the window between is_socket_connected and 
> ajp_send_header.

I tested with worker.ajp13.recovery_options=3, and all work fine without 
errors. 

> I tested with worker.ajp13.recovery_options=3, and all work fine without 
> errors. 

Other than patching proxy_util.c as in #7, and trying again, I don't see a way 
forward. (not that I expect this to do anything interesting :).

(In reply to comment #13)
> > I tested with worker.ajp13.recovery_options=3, and all work fine without 
> > errors. 
> Other than patching proxy_util.c as in #7, and trying again, I don't see a 
way 
> forward. (not that I expect this to do anything interesting :).

I tested the patch in #7 before, still same errors. 

We're seeing the same errors for mod_proxy_ajp and wanted to know how y'all were
trying out mod_jk w/ apache 2.2.3 given that the latest source distribution for
mod_jk has no support for 2.2; it only supports 1.3 and 2.0

We have the same problem RHEL5 / /usr/lib64/httpd/modules/mod_proxy_ajp.so from
httpd 2.2.3. Messages start to appear from 50 concurrent connections.

(In reply to comment #16)
> We have the same problem RHEL5 / /usr/lib64/httpd/modules/mod_proxy_ajp.so from
> httpd 2.2.3. Messages start to appear from 50 concurrent connections.

Also seeing the same problem on FC6 x86_64 with using tcnative (native apr) and
tomcat 5.5.23.

[Fri Aug 10 16:36:40 2007] [error] ajp_check_msg_header() got bad signature 420
[Fri Aug 10 16:36:40 2007] [error] ajp_ilink_receive() received bad header
[Fri Aug 10 16:36:40 2007] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Aug 10 16:36:40 2007] [error] (120007)APR does not understand this error
code: proxy: read response failed from 127.0.0.1:8009 (localhost)

Mostly works fine but every now and then with only a few users these errors pop
up. And someone gets a 503 service temporarily unavailable.

The config in proxy_ajp.conf is:
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

If any more info is needed just ask, really need this one fixed.

We had the same problem and finally, we found that it happened when Apache
MaxClient < Tomcat maxThread.
Now, it works fine.

If tomcat maxThread is reach then we got 503 until we restart tomcat.


Also, we had a configuration error because we tried to use maxProcessor in
tomcat's server.xml wich is not used in tomcat 6. The correct parameter is
maxThreads.

Plateform: RHEL5, x64.

(In reply to comment #17)
> (In reply to comment #16)
> > We have the same problem RHEL5 / /usr/lib64/httpd/modules/mod_proxy_ajp.so from
> > httpd 2.2.3. Messages start to appear from 50 concurrent connections.
> 
> Also seeing the same problem on FC6 x86_64 with using tcnative (native apr) and
> tomcat 5.5.23.
> 
> [Fri Aug 10 16:36:40 2007] [error] ajp_check_msg_header() got bad signature 420

This case is different from the other cases. It seems that your Tomcat is
sending wrong data. Please open a new report for this and try the following:

1. Does this still happen without tcnative?
2. Try upgrading Tomcat and tcnative

(In reply to comment #18)
> We had the same problem and finally, we found that it happened when Apache
> MaxClient < Tomcat maxThread.
> Now, it works fine.
> 
> If tomcat maxThread is reach then we got 503 until we restart tomcat.

I have two Apaches and two Tomcats. 

          -> Tomcat A
Apache 1 -|
          -> Tomcat B

          -> Tomcat A
Apache 2 -|
          -> Tomcat B

Apache's MaxClients is 1024 and Tomcat's maxThreads is also 1024. What's the
recommend setting to avoid ajp_ilink_receive() failed and client-side HTTP 503
error?

(In reply to comment #20)

> 
> I have two Apaches and two Tomcats. 
> 
>           -> Tomcat A
> Apache 1 -|
>           -> Tomcat B
> 
>           -> Tomcat A
> Apache 2 -|
>           -> Tomcat B
> 
> Apache's MaxClients is 1024 and Tomcat's maxThreads is also 1024. What's the
> recommend setting to avoid ajp_ilink_receive() failed and client-side HTTP 503
> error?

2048, but this is is insane. Try using the APR AJP connector
(http://tomcat.apache.org/tomcat-5.5-doc/apr.html). This allows you to limit the
number of threads to the number that really need to do work.

Tried APR with tcnative.

Now the error.log shows the following (sometimes):
[Tue Sep 18 14:48:05 2007] [error] ajp_check_msg_header() got bad signature 420
[Tue Sep 18 14:48:05 2007] [error] ajp_ilink_receive() received bad header
[Tue Sep 18 14:48:05 2007] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Sep 18 14:48:05 2007] [error] (120007)APR does not understand this error
code: proxy: read response failed from xxx.x.xxx.xx:20009 (xxx.x.xxx.xx)

tomcat log at the same time:
java.io.IOException
        at org.apache.coyote.ajp.AjpAprProcessor.flush(AjpAprProcessor.java:1197)
        at
org.apache.coyote.ajp.AjpAprProcessor$SocketOutputBuffer.doWrite(AjpAprProcessor.java:1282)
        at org.apache.coyote.Response.doWrite(Response.java:560)
        at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)

(In reply to comment #22)
> Tried APR with tcnative.
> 
> Now the error.log shows the following (sometimes):
> [Tue Sep 18 14:48:05 2007] [error] ajp_check_msg_header() got bad signature 420

This looks like PR43220 to me. Hope I find time any time soon to start working
on it again.

Hi, as this bug is hapenning on my platform ( Debian Etch x86_64 ), and this bug
is still tagged as NEEDINFO, here is a little extract of Apache debug dump
showing the problem. Hope it'll help:

[Wed Sep 26 17:14:52 2007] [debug] mod_proxy_ajp.c(44): proxy: AJP:
canonicalising URL //localhost:8009/test/dummy.gif
[Wed Sep 26 17:14:52 2007] [debug] proxy_util.c(1378): [client 84.14.90.18]
proxy: ajp: found worker ajp://localhost:8009/ for
ajp://localhost:8009/test/dummy.gif, referer:
http://dev.resamania.com/test/dummy.css
[Wed Sep 26 17:14:52 2007] [debug] mod_proxy.c(756): Running scheme ajp handler
(attempt 0)
[Wed Sep 26 17:14:52 2007] [debug] mod_proxy_ajp.c(497): proxy: AJP: serving URL
ajp://localhost:8009/test/dummy.gif
[Wed Sep 26 17:14:52 2007] [debug] proxy_util.c(1798): proxy: AJP: has acquired
connection for (localhost)
[Wed Sep 26 17:14:52 2007] [debug] proxy_util.c(1858): proxy: connecting
ajp://localhost:8009/test/dummy.gif to localhost:8009
[Wed Sep 26 17:14:52 2007] [debug] proxy_util.c(1951): proxy: connected
/test/dummy.gif to localhost:8009
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(218): Into ajp_marshal_into_msgb
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[0] [Host] = [dev.resamania.com]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[1] [User-Agent] = [Mozilla/5.0 (Windows; U; Windows NT 5.1; fr;
rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[2] [Accept] = [image/png,*/*;q=0.5]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[3] [Accept-Language] = [fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[4] [Accept-Encoding] = [gzip,deflate]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[5] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[6] [Keep-Alive] = [300]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[7] [Connection] = [keep-alive]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[8] [Referer] = [http://myhost//test/dummy.css]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[9] [Cookie] = [login=a; JSESSIONID=012345678901234567890123456789]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[10] [Authorization] = [....]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[11] [Pragma] = [no-cache]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[12] [Cache-Control] = [no-cache]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(284): ajp_marshal_into_msgb:
Header[13] [Max-Forwards] = [10]
[Wed Sep 26 17:14:52 2007] [debug] ajp_header.c(424): ajp_marshal_into_msgb: Done
[Wed Sep 26 17:14:52 2007] [debug] mod_proxy_ajp.c(187): proxy: APR_BUCKET_IS_EOS
[Wed Sep 26 17:14:52 2007] [debug] mod_proxy_ajp.c(192): proxy: data to read
(max 8186 at 4)
[Wed Sep 26 17:14:52 2007] [debug] mod_proxy_ajp.c(207): proxy: got 0 bytes of data
[Wed Sep 26 17:14:52 2007] [error] ajp_read_header: ajp_ilink_receive failed
[Wed Sep 26 17:14:52 2007] [error] (120007)APR does not understand this error
code: proxy: read response failed from 127.0.0.1:8009 (localhost)

Mathieu

Some tests with mod_jk (1.2.28 packaged inside Debian Etch x86_64 )
===================================================================

Just a bit more of information, people around are saying it works well with
mod_jk, but in my opinion it simply work better. I.e Mod_jk handles the
"connection crashing".... and try to handle the HTTP query through a new free
connector, here is a little extract of my mod_jk (1.2.28) error messages :

[Wed Sep 26 18:10:25 2007][16884:10592] [info] 
ajp_send_request::jk_ajp_common.c (1178): Socket 23 is not connected any more
(errno=-1)
[Wed Sep 26 18:10:25 2007][16884:10592] [info] 
ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try
another pooled connection
[Wed Sep 26 18:10:25 2007][16884:10592] [info] 
ajp_send_request::jk_ajp_common.c (1224): All endpoints are disconnected or dead
[Wed Sep 26 18:10:25 2007][16884:10592] [info]  ajp_service::jk_ajp_common.c
(1783): Sending request to tomcat failed,  recoverable operation attempt=1

Hope it helps.

Mathieu 

I don't know if this is relevant to this bug, but I saw this error reliably:

[error] ajp_check_msg_header() got bad signature 420
[error] ajp_ilink_receive() received bad header
[error] ajp_read_header: ajp_ilink_receive failed
[error] (120007)APR does not understand this error
code: proxy: read response failed from 127.0.0.1:8009 (localhost)

when using Apache 2.2.3 & mod_proxy_ajp to proxy a JBoss 4.2.2.GA / Tomcat 5.5.x app.

I found that my JBoss app wasn't setting the content-length response header.  For some reason, this caused my proxy to fail (503) after a number of requests.  The number of requests that before failure varied, but the error above always coincided with the 503s.  Just adding the Content-Length header in my JBoss/Tomcat app fixed it for me.  It may be relevant to mention that my proxied app was serving PDFs (w/ MIME application/pdf), not text/html.

We have also a lot of pdf download requests (axis webservice serving byte content). I switched to http a year ago and we had no problems since then. The problem only occurs in production environment so it's difficult for me to verify your workaround.

Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.