Bug 37770 - proxy: error reading status line from remote server (null)
Summary: proxy: error reading status line from remote server (null)
Status: RESOLVED WORKSFORME
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.2.3
Hardware: Sun Solaris
: P1 blocker with 29 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL: http://extra.gatema.cz/Test42/Data.asmx
Keywords: ErrorMessage
: 32731 37876 39499 39629 41150 44079 45588 46504 57520 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-12-03 13:40 UTC by Christophe Yayon
Modified: 2024-01-16 07:46 UTC (History)
28 users (show)



Attachments
backend connection race condition fix (untested) (2.88 KB, patch)
2007-12-07 07:31 UTC, Aleksey Midenkov
Details | Diff
Patch against trunk (1.71 KB, patch)
2008-07-31 13:23 UTC, Ruediger Pluem
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christophe Yayon 2005-12-03 13:40:17 UTC
Hi all,

I have configured httpd-2.2.0 (release) with mod_proxy and virtual hosting and
when i stress it, even low (10 simultaneous users with Mercury Load Runner) i
get  some errors messages and status code 502 on the client browser.
The problem occurs only when i do some load tests, when i request from single
client i have no problem...

My error.log :
[Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: error reading
status line from remote server (null), referer: http://portail.polen.xxxxx.fr/
[Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: Error reading
from remote server returned by /xxxx.gif, referer: http://portail.polen.xxxxx.fr/
[Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: error reading
status line from remote server (null), referer: http://portail.polen.xxxxx.fr/polen/
[Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: Error reading
from remote server returned by /polen/styles/polen.css, referer:
http://portail.polen.xxxxx.fr/polen/
[Sat Dec 03 13:16:26 2005] [error] [client 10.133.2.63] proxy: error reading
status line from remote server (null), referer: http://portail.polen.xxxx.fr/polen/
[Sat Dec 03 13:16:26 2005] [error] [client 10.133.2.63] proxy: Error reading
from remote server returned by /polen/images/droite2.gif, referer:
http://portail.polen.sofinco.fr/polen/
[Sat Dec 03 13:16:28 2005] [error] [client 10.133.2.63] (9)Bad file descriptor:
proxy: error reading response


My config :
<VirtualHost *>
ServerName portail.polen.xxxxxxx.fr
CustomLog /var/log/apache21/access-portail.log combined
<Location />
Order deny,allow
Allow from all
</Location>
RewriteEngine On
RewriteRule "(.*)"
"http://uxdev14.xxxxx.fr:8080%{REQUEST_URI}" [P,L]
</VirtualHost>


I have  tried with mod_proxy_balancer, mod_rewrite [P] (which call mod_proxy)
and of course mod_proxy...
The problem seems coming from mod_proxy.
Comment 1 Ruediger Pluem 2005-12-03 21:51:06 UTC
Merge from Christophe's mail to the dev list
(http://mail-archives.apache.org/mod_mbox/httpd-dev/200511.mbox/ajax/%3c4388AC09.2000907@nbux.com%3e):

Hi all,

i have just tried with the last 2.1.10 and i have the same problem :

1. in error logs :
[Sat Nov 26 19:29:55 2005] [error] [client 10.133.2.63] proxy: error reading
status line from remote server (null), referer:
http://portail.polen.sofinco.fr/polen/
[Sat Nov 26 19:29:55 2005] [error] [client 10.133.2.63] proxy: Error reading
from remote server returned by /polen/images/passe.gif, referer:
http://portail.polen.sofinco.fr/polen/
[Sat Nov 26 19:29:56 2005] [notice] child pid 25976 exit signal Segmentation
fault (11), possible coredump in /tmp


2. i used gdb to get a trace :
gdb /opt/apache21/bin/httpd /tmp/core.25976
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux"...Using host libthread_db library
"/lib/tls/libthread_db.so.1".

Core was generated by `/opt/apache21/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.

warning: current_sos: Can't read pathname for load map: Input/output error

Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /opt/apache21/lib/libaprutil-1.so.0...done.
Loaded symbols for /opt/apache21/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libdb-4.3.so...done.
Loaded symbols for /usr/lib/libdb-4.3.so
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /opt/apache21/lib/libapr-1.so.0...done.
Loaded symbols for /opt/apache21/lib/libapr-1.so.0
Reading symbols from /lib/tls/librt.so.1...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /lib/tls/libcrypt.so.1...done.
Loaded symbols for /lib/tls/libcrypt.so.1
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/tls/libdl.so.2...done.
Loaded symbols for /lib/tls/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/tls/libnss_compat.so.2...done.
Loaded symbols for /lib/tls/libnss_compat.so.2
Reading symbols from /lib/tls/libnsl.so.1...done.
Loaded symbols for /lib/tls/libnsl.so.1
Reading symbols from /lib/tls/libnss_nis.so.2...done.
Loaded symbols for /lib/tls/libnss_nis.so.2
Reading symbols from /lib/tls/libnss_files.so.2...done.
Loaded symbols for /lib/tls/libnss_files.so.2
Reading symbols from /opt/apache21/modules/mod_authz_host.so...done.
Loaded symbols for /opt/apache21/modules/mod_authz_host.so
Reading symbols from /opt/apache21/modules/mod_dumpio.so...done.
Loaded symbols for /opt/apache21/modules/mod_dumpio.so
Reading symbols from /opt/apache21/modules/mod_include.so...done.
Loaded symbols for /opt/apache21/modules/mod_include.so
Reading symbols from /opt/apache21/modules/mod_log_config.so...done.
Loaded symbols for /opt/apache21/modules/mod_log_config.so
Reading symbols from /opt/apache21/modules/mod_logio.so...done.
Loaded symbols for /opt/apache21/modules/mod_logio.so
Reading symbols from /opt/apache21/modules/mod_env.so...done.
Loaded symbols for /opt/apache21/modules/mod_env.so
Reading symbols from /opt/apache21/modules/mod_expires.so...done.
Loaded symbols for /opt/apache21/modules/mod_expires.so
Reading symbols from /opt/apache21/modules/mod_headers.so...done.
Loaded symbols for /opt/apache21/modules/mod_headers.so
Reading symbols from /opt/apache21/modules/mod_ident.so...done.
Loaded symbols for /opt/apache21/modules/mod_ident.so
Reading symbols from /opt/apache21/modules/mod_usertrack.so...done.
Loaded symbols for /opt/apache21/modules/mod_usertrack.so
Reading symbols from /opt/apache21/modules/mod_unique_id.so...done.
Loaded symbols for /opt/apache21/modules/mod_unique_id.so
Reading symbols from /opt/apache21/modules/mod_setenvif.so...done.
Loaded symbols for /opt/apache21/modules/mod_setenvif.so
Reading symbols from /opt/apache21/modules/mod_version.so...done.
Loaded symbols for /opt/apache21/modules/mod_version.so
Reading symbols from /opt/apache21/modules/mod_proxy.so...done.
Loaded symbols for /opt/apache21/modules/mod_proxy.so
Reading symbols from /opt/apache21/modules/mod_proxy_connect.so...done.
Loaded symbols for /opt/apache21/modules/mod_proxy_connect.so
Reading symbols from /opt/apache21/modules/mod_proxy_http.so...done.
Loaded symbols for /opt/apache21/modules/mod_proxy_http.so
Reading symbols from /opt/apache21/modules/mod_proxy_balancer.so...done.
Loaded symbols for /opt/apache21/modules/mod_proxy_balancer.so
Reading symbols from /opt/apache21/modules/mod_mime.so...done.
Loaded symbols for /opt/apache21/modules/mod_mime.so
Reading symbols from /opt/apache21/modules/mod_status.so...done.
Loaded symbols for /opt/apache21/modules/mod_status.so
Reading symbols from /opt/apache21/modules/mod_autoindex.so...done.
Loaded symbols for /opt/apache21/modules/mod_autoindex.so
Reading symbols from /opt/apache21/modules/mod_asis.so...done.
Loaded symbols for /opt/apache21/modules/mod_asis.so
Reading symbols from /opt/apache21/modules/mod_info.so...done.
Loaded symbols for /opt/apache21/modules/mod_info.so
Reading symbols from /opt/apache21/modules/mod_vhost_alias.so...done.
Loaded symbols for /opt/apache21/modules/mod_vhost_alias.so
Reading symbols from /opt/apache21/modules/mod_dir.so...done.
Loaded symbols for /opt/apache21/modules/mod_dir.so
Reading symbols from /opt/apache21/modules/mod_alias.so...done.
Loaded symbols for /opt/apache21/modules/mod_alias.so
Reading symbols from /opt/apache21/modules/mod_rewrite.so...done.
Loaded symbols for /opt/apache21/modules/mod_rewrite.so
Reading symbols from /opt/apache21/modules/mod_cache.so...done.
Loaded symbols for /opt/apache21/modules/mod_cache.so
Reading symbols from /opt/apache21/modules/mod_mem_cache.so...done.
Loaded symbols for /opt/apache21/modules/mod_mem_cache.so
Reading symbols from /opt/apache21/modules/mod_ssl.so...done.
Loaded symbols for /opt/apache21/modules/mod_ssl.so
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/tls/libnss_dns.so.2...done.
Loaded symbols for /lib/tls/libnss_dns.so.2
Reading symbols from /lib/tls/libresolv.so.2...done.
Loaded symbols for /lib/tls/libresolv.so.2
#0  apr_brigade_cleanup (data=0x8398980) at buckets/apr_brigade.c:44
44              apr_bucket_delete(e);
(gdb) bt full
#0  apr_brigade_cleanup (data=0x8398980) at buckets/apr_brigade.c:44
No locals.
#1  0xb7f24e7d in apr_brigade_destroy (b=0x8398980) at buckets/apr_brigade.c:53
No locals.
#2  0x080726f4 in ap_core_output_filter (f=0xaa083538, b=0x8398980) at
core_filters.c:876
        hdtr = {headers = 0xb41c3ff8, numheaders = 134740504, trailers =
0x8392910, numtrailers = 0}
        bytes_sent = 0
        flags = 0
        nbytes = 1429
        nvec_trailers = 0
        vec = {{iov_base = 0x838d208, iov_len = 206}, {iov_base = 0x8394a09,
iov_len = 1223}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0xaa083aa0, iov_len
= 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217623}, {iov_base = 0x0,
    iov_len = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217643},
{iov_base = 0x8392910, iov_len = 0}, {iov_base = 0xb41c3f30, iov_len = 0},
{iov_base = 0x838e938, iov_len = 0}, {iov_base = 0xb41c3f68, iov_len =
134646597}, {iov_base = 0x0,
    iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x8393960, iov_len
= 137967992}, {iov_base = 0xb41c3ff8, iov_len = 134740481}, {iov_base =
0x8393960, iov_len = 137967992}}
        flen = Variable "flen" is not available.
(gdb) thread apply all bt full

Thread 27 (process 25976):
#0  0x00000293 in ?? ()
No symbol table info available.
#1  0x00000000 in ?? ()
No symbol table info available.
#2  0x00000000 in ?? ()
No symbol table info available.
#3  0x00000000 in ?? ()
No symbol table info available.
#4  0x00000000 in ?? ()
No symbol table info available.
#5  0x00000000 in ?? ()
No symbol table info available.
#6  0x00000000 in ?? ()
No symbol table info available.
#7  0x00000000 in ?? ()
No symbol table info available.
#8  0x00000000 in ?? ()
No symbol table info available.
#9  0x00000000 in ?? ()
---Type <return> to continue, or q <return> to quit---

Comment 2 Ruediger Pluem 2005-12-03 21:54:00 UTC
(In reply to comment #1)

[..cut..]

> 1. in error logs :
> [Sat Nov 26 19:29:55 2005] [error] [client 10.133.2.63] proxy: error reading
> status line from remote server (null), referer:
> http://portail.polen.sofinco.fr/polen/

Strange that the remote server is null.

[..cut..]

> (gdb) bt full
> #0  apr_brigade_cleanup (data=0x8398980) at buckets/apr_brigade.c:44
> No locals.
> #1  0xb7f24e7d in apr_brigade_destroy (b=0x8398980) at buckets/apr_brigade.c:53
> No locals.
> #2  0x080726f4 in ap_core_output_filter (f=0xaa083538, b=0x8398980) at
> core_filters.c:876
>         hdtr = {headers = 0xb41c3ff8, numheaders = 134740504, trailers =
> 0x8392910, numtrailers = 0}
>         bytes_sent = 0
>         flags = 0
>         nbytes = 1429
>         nvec_trailers = 0
>         vec = {{iov_base = 0x838d208, iov_len = 206}, {iov_base = 0x8394a09,
> iov_len = 1223}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0xaa083aa0, iov_len
> = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217623}, {iov_base = 0x0,
>     iov_len = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217643},
> {iov_base = 0x8392910, iov_len = 0}, {iov_base = 0xb41c3f30, iov_len = 0},
> {iov_base = 0x838e938, iov_len = 0}, {iov_base = 0xb41c3f68, iov_len =
> 134646597}, {iov_base = 0x0,
>     iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x8393960, iov_len
> = 137967992}, {iov_base = 0xb41c3ff8, iov_len = 134740481}, {iov_base =
> 0x8393960, iov_len = 137967992}}
>         flen = Variable "flen" is not available.
> (gdb) thread apply all bt full


Possibly a corrupted brigade. I am a little bit astonished that the stacktrace
isn't longer.
So could you please do the following:

1. Add the contents of .gdbinit from the root directory of the 2.1.10 tar ball
to your .gdbinit
   (see http://httpd.apache.org/dev/debugging.html#gdb)

2. Issue the following commands:

gdb /opt/apache21/bin/httpd /tmp/core.25976
bt
frame 2
dump_brigade b
dump_filters f

[..cut..]
Comment 3 Ruediger Pluem 2005-12-03 22:04:29 UTC
Please set your LogLevel also to debug and let us know the contents of the error
log once the problem happens again.
Comment 4 Christophe Yayon 2005-12-04 08:28:59 UTC
Hum, very very strange, when i have LogLevel debug, problem NEVER occur !, but i
have any other LogLevel (info, notice, warn, crit, ...) the problem occur again
(but no more detailed log information than info - which i posted).

I have retry 10 times, with much of stress (500 simultaneous users) but no luck
... no more error message like my bug report post !

Sorry
Comment 5 Christophe Yayon 2005-12-04 15:20:40 UTC
Hi again,

Here is more informations about the problem, sometimes i even get a segmentation
fault (11) :

I have done some differents stress tests :

1. with a debug LogLevel : no problem (see my last post), i have the problem
only with any other level (warn, notice, info, crit, ...)

2. with others backends servers (simple .html on an apache 2.0.54 instead of a
.jsp on a Tomcat-5.5.12) : no problem. Of course, no error on Tomcat backend
server, and when i bypass reverse proxy, or when i use apache 2.0.55 (instead of
2.2.0 with mod_proxy), i don't get any error.

3. sometimes i get a segmentation fault  and here a trace (i have used .gdbinit
of httpd-2.2.0 directory) :

------------------
gdb /opt/apache22/bin/httpd /tmp/core.7637 
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux"...Using host libthread_db library
"/lib/tls/libthread_db.so.1".

Core was generated by `/opt/apache22/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.

warning: current_sos: Can't read pathname for load map: Input/output error

Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /opt/apache22/lib/libaprutil-1.so.0...done.
Loaded symbols for /opt/apache22/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libdb-4.3.so...done.
Loaded symbols for /usr/lib/libdb-4.3.so
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /opt/apache22/lib/libapr-1.so.0...done.
Loaded symbols for /opt/apache22/lib/libapr-1.so.0
Reading symbols from /lib/tls/librt.so.1...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /lib/tls/libcrypt.so.1...done.
Loaded symbols for /lib/tls/libcrypt.so.1
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/tls/libdl.so.2...done.
Loaded symbols for /lib/tls/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/tls/libnss_compat.so.2...done.
Loaded symbols for /lib/tls/libnss_compat.so.2
Reading symbols from /lib/tls/libnsl.so.1...done.
Loaded symbols for /lib/tls/libnsl.so.1
Reading symbols from /lib/tls/libnss_nis.so.2...done.
Loaded symbols for /lib/tls/libnss_nis.so.2
Reading symbols from /lib/tls/libnss_files.so.2...done.
Loaded symbols for /lib/tls/libnss_files.so.2
Reading symbols from /opt/apache22/modules/mod_authz_host.so...done.
Loaded symbols for /opt/apache22/modules/mod_authz_host.so
Reading symbols from /opt/apache22/modules/mod_dumpio.so...done.
Loaded symbols for /opt/apache22/modules/mod_dumpio.so
Reading symbols from /opt/apache22/modules/mod_include.so...done.
Loaded symbols for /opt/apache22/modules/mod_include.so
Reading symbols from /opt/apache22/modules/mod_log_config.so...done.
Loaded symbols for /opt/apache22/modules/mod_log_config.so
Reading symbols from /opt/apache22/modules/mod_logio.so...done.
Loaded symbols for /opt/apache22/modules/mod_logio.so
Reading symbols from /opt/apache22/modules/mod_env.so...done.
Loaded symbols for /opt/apache22/modules/mod_env.so
Reading symbols from /opt/apache22/modules/mod_expires.so...done.
Loaded symbols for /opt/apache22/modules/mod_expires.so
Reading symbols from /opt/apache22/modules/mod_headers.so...done.
Loaded symbols for /opt/apache22/modules/mod_headers.so
Reading symbols from /opt/apache22/modules/mod_ident.so...done.
Loaded symbols for /opt/apache22/modules/mod_ident.so
Reading symbols from /opt/apache22/modules/mod_usertrack.so...done.
Loaded symbols for /opt/apache22/modules/mod_usertrack.so
Reading symbols from /opt/apache22/modules/mod_unique_id.so...done.
Loaded symbols for /opt/apache22/modules/mod_unique_id.so
Reading symbols from /opt/apache22/modules/mod_setenvif.so...done.
Loaded symbols for /opt/apache22/modules/mod_setenvif.so
Reading symbols from /opt/apache22/modules/mod_version.so...done.
Loaded symbols for /opt/apache22/modules/mod_version.so
Reading symbols from /opt/apache22/modules/mod_proxy.so...done.
Loaded symbols for /opt/apache22/modules/mod_proxy.so
Reading symbols from /opt/apache22/modules/mod_proxy_connect.so...done.
Loaded symbols for /opt/apache22/modules/mod_proxy_connect.so
Reading symbols from /opt/apache22/modules/mod_proxy_http.so...done.
Loaded symbols for /opt/apache22/modules/mod_proxy_http.so
Reading symbols from /opt/apache22/modules/mod_proxy_balancer.so...done.
Loaded symbols for /opt/apache22/modules/mod_proxy_balancer.so
Reading symbols from /opt/apache22/modules/mod_mime.so...done.
Loaded symbols for /opt/apache22/modules/mod_mime.so
Reading symbols from /opt/apache22/modules/mod_status.so...done.
Loaded symbols for /opt/apache22/modules/mod_status.so
Reading symbols from /opt/apache22/modules/mod_autoindex.so...done.
Loaded symbols for /opt/apache22/modules/mod_autoindex.so
Reading symbols from /opt/apache22/modules/mod_asis.so...done.
Loaded symbols for /opt/apache22/modules/mod_asis.so
Reading symbols from /opt/apache22/modules/mod_info.so...done.
Loaded symbols for /opt/apache22/modules/mod_info.so
Reading symbols from /opt/apache22/modules/mod_vhost_alias.so...done.
Loaded symbols for /opt/apache22/modules/mod_vhost_alias.so
Reading symbols from /opt/apache22/modules/mod_dir.so...done.
Loaded symbols for /opt/apache22/modules/mod_dir.so
Reading symbols from /opt/apache22/modules/mod_alias.so...done.
Loaded symbols for /opt/apache22/modules/mod_alias.so
Reading symbols from /opt/apache22/modules/mod_rewrite.so...done.
Loaded symbols for /opt/apache22/modules/mod_rewrite.so
Reading symbols from /opt/apache22/modules/mod_cache.so...done.
Loaded symbols for /opt/apache22/modules/mod_cache.so
Reading symbols from /opt/apache22/modules/mod_mem_cache.so...done.
Loaded symbols for /opt/apache22/modules/mod_mem_cache.so
Reading symbols from /opt/apache22/modules/mod_ssl.so...done.
Loaded symbols for /opt/apache22/modules/mod_ssl.so
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/tls/libnss_dns.so.2...done.
Loaded symbols for /lib/tls/libnss_dns.so.2
Reading symbols from /lib/tls/libresolv.so.2...done.
Loaded symbols for /lib/tls/libresolv.so.2
#0  0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0
(gdb) bt
#0  0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0
#1  0x08085ea7 in ap_mpm_pod_check (pod=0xfffffe00) at pod.c:54
#2  0x08083f69 in child_main (child_num_arg=Variable "child_num_arg" is not
available.
) at worker.c:1233
#3  0x080840d0 in make_child (s=0x80a7d88, slot=0) at worker.c:1316
#4  0x0808417b in startup_children (number_to_start=2) at worker.c:1350
#5  0x08084ce1 in ap_mpm_run (_pconf=0x80a60a8, plog=0x80d4160, s=0x80a7d88) at
worker.c:1700
#6  0x08061bef in main (argc=3, argv=0xbfe76ca4) at main.c:712
(gdb) frame 2
#2  0x08083f69 in child_main (child_num_arg=Variable "child_num_arg" is not
available.
) at worker.c:1233
1233                rv = ap_mpm_pod_check(pod);
(gdb) dump_brigade b
No symbol "b" in current context.
(gdb) dump_filters f
No symbol "f" in current context.
--------------------------------


Comment 6 Ruediger Pluem 2005-12-04 15:32:43 UTC
Please do the following things:

1. Apply the steps (dump_brigade and co) I had described to the old core dump
(core.25976).
2. Apply the command the following commands to your latest core dump
(/tmp/core.7637):

info threads
thread apply all bt
Comment 7 Christophe Yayon 2005-12-04 15:43:07 UTC
I have just tried with prefork mpm method instead of worker, and no more problem...

I appears that it is worker mpm with is the origin of the problem...

Should i open a new bug report ?
Comment 8 Christophe Yayon 2005-12-04 15:47:03 UTC
1. sorry i have deleted it ...

2. 
(gdb) info threads
  27 process 7640  0x00000212 in ?? ()
  26 process 7641  0x00000212 in ?? ()
  25 process 7642  0x00000212 in ?? ()
  24 process 7643  0x00000212 in ?? ()
  23 process 7644  0x00000212 in ?? ()
  22 process 7645  0x00000212 in ?? ()
  21 process 7646  0x00000246 in ?? ()
  20 process 7647  0x00000246 in ?? ()
  19 process 7648  0x00000212 in ?? ()
  18 process 7649  0x00000212 in ?? ()
  17 process 7650  0x00000212 in ?? ()
  16 process 7651  0x00000212 in ?? ()
  15 process 7652  0x00000212 in ?? ()
  14 process 7653  0x00000212 in ?? ()
  13 process 7654  0x00000212 in ?? ()
  12 process 7655  0x00000212 in ?? ()
  11 process 7656  0x00000212 in ?? ()
  10 process 7657  0x00000246 in ?? ()
  9 process 7659  0x00000212 in ?? ()
  8 process 7660  0x00000246 in ?? ()
  7 process 7661  0x00010292 in ?? ()
  6 process 7662  0x00000212 in ?? ()
  5 process 7663  0x00000212 in ?? ()
  4 process 7664  0x00000212 in ?? ()
  3 process 7665  0x00000246 in ?? ()
  2 process 7666  0x00000293 in ?? ()
* 1 process 7637  0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0
(gdb) thread apply all bt

Thread 27 (process 7640):
#0  0x00000212 in ?? ()
Cannot access memory at address 0x0
#0  0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0


see my last post, when i use mpm prefork, no more problem, it appears that it
come from worker...

should i open a new bug report ?
Comment 9 Ruediger Pluem 2005-12-04 16:19:50 UTC
No, you do not need to create a new bug report. We keep this one open.
Comment 10 Jerry Baker 2005-12-12 19:38:59 UTC
*** Bug 37876 has been marked as a duplicate of this bug. ***
Comment 11 Jerry Baker 2005-12-12 19:39:47 UTC
OS -> All since I see this on WinXP (even the crash).
Comment 12 Jerry Baker 2005-12-12 21:14:39 UTC
Apologies for the bug spam, but I thought I should note that this isn't
happening to me when load testing. This is happening in real-world usage. The
crashes are coming about every two hours or so. Apache 2.2.0's proxy is unusable
in a production environment.
Comment 13 Ruediger Pluem 2005-12-12 21:47:00 UTC
Christophe said that he cannot reproduce the error with LogLevel debug. Are you
able to reproduce the error with LogLevel debug? If yes, it would be very
helpful if you could attach the error log here.
Comment 14 Jerry Baker 2005-12-12 22:51:08 UTC
[Mon Dec 12 13:49:44 2005] [debug] mod_proxy_http.c(54): proxy: HTTP:
canonicalising URL //shserver2.ssl.berkeley.edu/
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1382): [client 127.0.0.1] proxy:
*: found forward proxy worker for http://shserver2.ssl.berkeley.edu/
[Mon Dec 12 13:49:44 2005] [debug] mod_proxy.c(736): Running scheme http handler
(attempt 0)
[Mon Dec 12 13:49:44 2005] [debug] mod_proxy_http.c(1661): proxy: HTTP: serving
URL http://shserver2.ssl.berkeley.edu/
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1754): proxy: HTTP: has acquired
connection for (*)
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1811): proxy: connecting
http://shserver2.ssl.berkeley.edu/ to shserver2.ssl.berkeley.edu:80
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1911): proxy: connected / to
shserver2.ssl.berkeley.edu:80
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(2005): proxy: HTTP: fam 2 socket
created to connect to *
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(2101): proxy: HTTP: connection
complete to 66.28.250.122:80 (shserver2.ssl.berkeley.edu)
[Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1769): proxy: (null): has
released connection for (*)
[Mon Dec 12 13:49:44 2005] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Dec 12 13:49:44 2005] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by http://shserver2.ssl.berkeley.edu/
Comment 15 Ruediger Pluem 2006-05-12 12:49:31 UTC
httpd 2.2.2 contains many proxy fixes. Could you please check if the problem is
still there with 2.2.2?
Comment 16 Mitch McKenzie 2006-05-16 15:19:31 UTC
We have tried(In reply to comment #15)
> httpd 2.2.2 contains many proxy fixes. Could you please check if the problem 
is
> still there with 2.2.2?

We upgraded 6 of our proxy servers to 2.2.2 and are still experiencing problems.

Is anyone else able to reproduce the problem in 2.2.2???

Thanks.
Comment 17 James 2006-05-16 17:07:23 UTC
I to have seen this with 2.2.2 and also the latest cvs. Everything works ok with
prefork mpm.

The problem also only accours for us when we use an IIS backend device, with an
apache its ok. From a tcpdump the only diff I can see between the two sessions
is that the IIS closes the connection when the keep-alive time out is reached
with a TCP RST packet, while the apache does a Fin/Ack handshake. When IIS sends
this RST packet the next request to the mod_proxy device thats hits the worker
with the RST connection returns the 502 error msg.

Our fix (read hack) was to set the timeout to 24hours on all the backend devices.
Comment 18 Christophe Yayon 2006-05-16 17:19:49 UTC
hi all,

i have just tried with 2.2.2 and apache/mod_proxy_ajp+balancer as backend
(mod_proxy_balancer -> mod_proxy_balancer/ajp -> tomcat ajp) with the same load
test in my first post, and i have only status line error message when i get a
timeout from backend which is normal in my case...

i used apache 2.2.2 with worker mpm...

we are trying another load test and production switch if it ok thursday night...
i return you friday, i hope it will be ok...
Comment 19 Christophe Yayon 2006-05-17 05:57:47 UTC
does anyone encounter segmentation fault with mpm worker ?
Comment 20 Alexander Hartner 2006-05-17 10:11:29 UTC
We are experiencing the same problem using Apache 2.2 on Windows 2000 Server.
The backend is a combination of Tomcat / JBoss and IIS. Yet the problem only
seems to affect IIS. 

C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe -l
Compiled in modules:
  core.c
  mod_win32.c
  mpm_winnt.c
  http_core.c
  mod_so.c

C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe -v
Server version: Apache/2.2.2
Server built:   Apr 29 2006 18:32:31

This is an extract from the log file:

[Tue May 16 15:55:50 2006] [notice] Parent: Received restart signal --
Restarting the server.
[Tue May 16 15:55:50 2006] [notice] Apache/2.2.2 (Win32) configured -- resuming
normal operations
[Tue May 16 15:55:50 2006] [notice] Server built: Apr 29 2006 18:32:31
[Tue May 16 15:55:50 2006] [notice] Child 2624: Exit event signaled. Child
process is ending.
[Tue May 16 15:55:50 2006] [notice] Parent: Created child process 1864
[Tue May 16 15:55:50 2006] [notice] Child 1864: Child process is running
[Tue May 16 15:55:51 2006] [notice] Child 2624: Released the start mutex
[Tue May 16 15:55:51 2006] [notice] Child 1864: Acquired the start mutex.
[Tue May 16 15:55:51 2006] [notice] Child 1864: Starting 250 worker threads.
[Tue May 16 15:55:51 2006] [notice] Child 1864: Starting thread to listen on
port 8080.
[Tue May 16 15:55:52 2006] [notice] Child 2624: Waiting for 250 worker threads
to exit.
[Tue May 16 15:55:52 2006] [notice] Child 2624: All worker threads have exited.
[Tue May 16 15:55:52 2006] [notice] Child 2624: Child process is exiting
[Tue May 16 16:54:10 2006] [error] [client 81.187.164.45] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 16:54:10 2006] [error] [client 81.187.164.45] proxy: Error reading
from remote server returned by /support/
[Tue May 16 18:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 18:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Tue May 16 19:48:42 2006] [error] [client 207.46.98.44] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 19:48:42 2006] [error] [client 207.46.98.44] proxy: Error reading
from remote server returned by /robots.txt
[Tue May 16 19:59:10 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 19:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Tue May 16 20:56:43 2006] [error] [client 80.176.89.29] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 20:56:43 2006] [error] [client 80.176.89.29] proxy: Error reading
from remote server returned by /support/helpdesk/viewrequest.asp
[Tue May 16 20:58:22 2006] [error] [client 207.46.98.44] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 20:58:22 2006] [error] [client 207.46.98.44] proxy: Error reading
from remote server returned by /default.asp
[Tue May 16 21:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 21:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Tue May 16 22:59:10 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 22:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Tue May 16 23:59:10 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Tue May 16 23:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 00:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 00:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 01:33:59 2006] [error] [client 207.46.98.44] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 01:33:59 2006] [error] [client 207.46.98.44] proxy: Error reading
from remote server returned by /robots.txt
[Wed May 17 01:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 01:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 02:11:08 2006] [error] [client 68.142.212.239] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 02:11:08 2006] [error] [client 68.142.212.239] proxy: Error reading
from remote server returned by /robots.txt
[Wed May 17 02:59:08 2006] [error] [client 68.142.212.239] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 02:59:08 2006] [error] [client 68.142.212.239] proxy: Error reading
from remote server returned by /img/front/fnt_utilities.jpg
[Wed May 17 02:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 02:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 03:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 03:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 04:48:46 2006] [warn] (OS 64)The specified network name is no longer
available.  : winnt_accept: Asynchronous AcceptEx failed.
[Wed May 17 04:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 04:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 05:59:10 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 05:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 06:59:10 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 06:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 07:59:11 2006] [error] [client 81.29.74.130] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 07:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading
from remote server returned by /support/huns/post.asp
[Wed May 17 08:29:25 2006] [error] [client 81.187.164.45] proxy: error reading
status line from remote server 192.168.99.88
[Wed May 17 08:29:25 2006] [error] [client 81.187.164.45] proxy: Error reading
from remote server returned by /support/helpdesk/viewrequest.asp
[Wed May 17 09:01:38 2006] [notice] Parent: Received restart signal --
Restarting the server.
[Wed May 17 09:01:38 2006] [notice] Apache/2.2.2 (Win32) configured -- resuming
normal operations
Comment 21 Mitch McKenzie 2006-05-19 14:43:09 UTC
We installed 2.2.2 and turned debugging on but it does not appear to give us 
any further information.

Here is a portion of the error log that contains the errors:

[Fri May 19 04:02:28 2006] [error] [client 69.60.122.139] proxy: error reading 
status line from remote server 192.198.54.130
[Fri May 19 04:02:28 2006] [error] [client 69.60.122.139] proxy: Error reading 
from remote server returned by /community/ruralmeded/model.htm
[Fri May 19 05:02:27 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/calm.htm
[Fri May 19 05:02:27 2006] [debug] proxy_util.c(1378): [client 72.30.133.212] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 05:02:27 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 05:02:27 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 05:02:27 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 05:23:12 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/security.htm
[Fri May 19 05:23:12 2006] [debug] proxy_util.c(1378): [client 72.30.101.219] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/security.htm
[Fri May 19 05:23:12 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/security.htm
[Fri May 19 05:23:12 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/security.htm to 192.198.54.130:80
[Fri May 19 05:23:12 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/security.htm to 192.198.54.130:80
[Fri May 19 05:59:07 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/
[Fri May 19 05:59:07 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/, referer: http://www.ianr.unl.edu/
[Fri May 19 05:59:07 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/
[Fri May 19 05:59:07 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/ to 192.198.54.130:80
[Fri May 19 05:59:07 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/ to 192.198.54.130:80
[Fri May 19 06:06:19 2006] [debug] proxy_util.c(1378): [client 65.214.39.180] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/5/unmc.css, referer: 
http://www.unmc.edu/bioterrorism/ommrslist.htm
[Fri May 19 06:06:19 2006] [debug] proxy_util.c(1378): [client 65.214.39.180] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/footer.js, referer: 
http://www.unmc.edu/bioterrorism/ommrslist.htm
[Fri May 19 06:37:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:37:19 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:37:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:37:19 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:37:19 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:47:07 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/faq.htm
[Fri May 19 06:47:07 2006] [debug] proxy_util.c(1378): [client 68.142.249.93] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/faq.htm
[Fri May 19 06:47:07 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/faq.htm
[Fri May 19 06:47:07 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/faq.htm to 192.198.54.130:80
[Fri May 19 06:47:07 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/faq.htm to 192.198.54.130:80
[Fri May 19 06:48:37 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/
[Fri May 19 06:48:37 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/
[Fri May 19 06:48:37 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/
[Fri May 19 06:48:37 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/ to 192.198.54.130:80
[Fri May 19 06:48:37 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/ to 192.198.54.130:80
[Fri May 19 06:49:29 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:49:29 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/nphl.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 06:49:29 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:49:29 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:49:29 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:49:32 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:49:32 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/sambol.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 06:49:32 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:49:32 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:49:32 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:49:42 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/inthenews.htm
[Fri May 19 06:49:42 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/inthenews.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 06:49:42 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/inthenews.htm
[Fri May 19 06:49:42 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/inthenews.htm to 192.198.54.130:80
[Fri May 19 06:49:42 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/inthenews.htm to 192.198.54.130:80
[Fri May 19 06:52:30 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:52:30 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:52:30 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:52:30 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:52:30 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:52:33 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 06:52:33 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 06:52:33 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 06:52:33 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 06:52:33 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 06:59:06 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/partners.htm
[Fri May 19 06:59:06 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/partners.htm
[Fri May 19 06:59:06 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/partners.htm
[Fri May 19 06:59:06 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/partners.htm to 192.198.54.130:80
[Fri May 19 06:59:06 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/partners.htm to 192.198.54.130:80
[Fri May 19 07:51:38 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising 
URL //192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_erro
rs.htm
[Fri May 19 07:51:38 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err
ors.htm
[Fri May 19 07:51:38 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL 
http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err
ors.htm
[Fri May 19 07:51:38 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err
ors.htm to 192.198.54.130:80
[Fri May 19 07:51:38 2006] [debug] proxy_util.c(1951): proxy: 
connected /Community/ruralmeded/admissions/admissions_and_medical_errors.htm to 
192.198.54.130:80
[Fri May 19 08:00:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/coping.htm
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1378): [client 65.38.102.153] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/coping.htm
[Fri May 19 08:00:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/coping.htm
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/coping.htm to 192.198.54.130:80
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/coping.htm to 192.198.54.130:80
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1378): [client 65.38.102.153] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/footer.js, referer: 
http://www.unmc.edu/bioterrorism/coping.htm
[Fri May 19 08:14:21 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/calm.htm
[Fri May 19 08:14:21 2006] [debug] proxy_util.c(1378): [client 65.214.44.82] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 08:14:21 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 08:14:21 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 08:14:21 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/minimed/smallpox.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:20:15 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/system.htm
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/system.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:20:15 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/system.htm
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/system.htm to 192.198.54.130:80
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/system.htm to 192.198.54.130:80
[Fri May 19 08:20:16 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 08:20:16 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/newsarchive.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:20:16 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 08:20:16 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 08:20:16 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 08:27:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/
[Fri May 19 08:27:19 2006] [debug] proxy_util.c(1378): [client 64.34.145.194] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/
[Fri May 19 08:27:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/
[Fri May 19 08:27:19 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/ to 192.198.54.130:80
[Fri May 19 08:27:19 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/ to 192.198.54.130:80
[Fri May 19 08:44:53 2006] [debug] proxy_util.c(1378): [client 204.248.22.161] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/headerbar.js, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/, referer: 
http://images.google.com/imgres?
imgurl=http://www.unmc.edu/rural/mederrors/images/logo-
graphics.jpg&imgrefurl=http://www.unmc.edu/rural/mederrors/&h=181&w=209&sz=11&tb
nid=vKuIcgUlFtYkKM:&tbnh=86&tbnw=100&hl=en&start=19&prev=/images%3Fq%
3Dmedication%2Berrors%26svnum%3D10%26hl%3Den%26lr%3D
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/ to 192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/ to 192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/department.js, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/images/logo-partners.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/images/logo-partners.jpg, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/images/logo-partners.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/images/logo-partners.jpg to 
192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/images/logo-partners.jpg to 192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/images/logo-contact.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/images/logo-contact.jpg, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/images/logo-contact.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/images/logo-contact.jpg to 
192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/images/logo-contact.jpg to 192.198.54.130:80
[Fri May 19 08:50:26 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/images/getacro.gif, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:54:32 2006] [error] [client 24.252.13.39] request failed: error 
reading the headers
[Fri May 19 09:16:43 2006] [debug] proxy_util.c(1378): [client 137.197.145.122] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/5/unmc.css, referer: http://www.unmc.edu/bioterrorism/
[Fri May 19 09:16:43 2006] [debug] proxy_util.c(1378): [client 137.197.145.122] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/footer.js, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 09:32:06 2006] [debug] proxy_util.c(1378): [client 138.163.0.38] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/headerbar.js, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 09:32:19 2006] [debug] proxy_util.c(1378): [client 138.163.0.44] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/5/unmc.css, referer: 
http://www.unmc.edu/bioterrorism/default.htm
Comment 22 Mitch McKenzie 2006-05-19 14:43:54 UTC
We installed 2.2.2 and turned debugging on but it does not appear to give us 
any further information.

Here is a portion of the error log that contains the errors:

[Fri May 19 04:02:28 2006] [error] [client 69.60.122.139] proxy: error reading 
status line from remote server 192.198.54.130
[Fri May 19 04:02:28 2006] [error] [client 69.60.122.139] proxy: Error reading 
from remote server returned by /community/ruralmeded/model.htm
[Fri May 19 05:02:27 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/calm.htm
[Fri May 19 05:02:27 2006] [debug] proxy_util.c(1378): [client 72.30.133.212] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 05:02:27 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 05:02:27 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 05:02:27 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 05:23:12 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/security.htm
[Fri May 19 05:23:12 2006] [debug] proxy_util.c(1378): [client 72.30.101.219] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/security.htm
[Fri May 19 05:23:12 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/security.htm
[Fri May 19 05:23:12 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/security.htm to 192.198.54.130:80
[Fri May 19 05:23:12 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/security.htm to 192.198.54.130:80
[Fri May 19 05:59:07 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/
[Fri May 19 05:59:07 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/, referer: http://www.ianr.unl.edu/
[Fri May 19 05:59:07 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/
[Fri May 19 05:59:07 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/ to 192.198.54.130:80
[Fri May 19 05:59:07 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/ to 192.198.54.130:80
[Fri May 19 06:06:19 2006] [debug] proxy_util.c(1378): [client 65.214.39.180] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/5/unmc.css, referer: 
http://www.unmc.edu/bioterrorism/ommrslist.htm
[Fri May 19 06:06:19 2006] [debug] proxy_util.c(1378): [client 65.214.39.180] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/footer.js, referer: 
http://www.unmc.edu/bioterrorism/ommrslist.htm
[Fri May 19 06:37:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:37:19 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:37:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:37:19 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:37:19 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:47:07 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/faq.htm
[Fri May 19 06:47:07 2006] [debug] proxy_util.c(1378): [client 68.142.249.93] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/faq.htm
[Fri May 19 06:47:07 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/faq.htm
[Fri May 19 06:47:07 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/faq.htm to 192.198.54.130:80
[Fri May 19 06:47:07 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/faq.htm to 192.198.54.130:80
[Fri May 19 06:48:37 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/
[Fri May 19 06:48:37 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/
[Fri May 19 06:48:37 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/
[Fri May 19 06:48:37 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/ to 192.198.54.130:80
[Fri May 19 06:48:37 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/ to 192.198.54.130:80
[Fri May 19 06:49:29 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:49:29 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/nphl.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 06:49:29 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/nphl.htm
[Fri May 19 06:49:29 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:49:29 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/nphl.htm to 192.198.54.130:80
[Fri May 19 06:49:32 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:49:32 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/sambol.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 06:49:32 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:49:32 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:49:32 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:49:42 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/inthenews.htm
[Fri May 19 06:49:42 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/inthenews.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 06:49:42 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/inthenews.htm
[Fri May 19 06:49:42 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/inthenews.htm to 192.198.54.130:80
[Fri May 19 06:49:42 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/inthenews.htm to 192.198.54.130:80
[Fri May 19 06:52:30 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:52:30 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:52:30 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/sambol.htm
[Fri May 19 06:52:30 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:52:30 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/sambol.htm to 192.198.54.130:80
[Fri May 19 06:52:33 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 06:52:33 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 06:52:33 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 06:52:33 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 06:52:33 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 06:59:06 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/partners.htm
[Fri May 19 06:59:06 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/partners.htm
[Fri May 19 06:59:06 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/partners.htm
[Fri May 19 06:59:06 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/partners.htm to 192.198.54.130:80
[Fri May 19 06:59:06 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/partners.htm to 192.198.54.130:80
[Fri May 19 07:51:38 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising 
URL //192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_erro
rs.htm
[Fri May 19 07:51:38 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err
ors.htm
[Fri May 19 07:51:38 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL 
http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err
ors.htm
[Fri May 19 07:51:38 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err
ors.htm to 192.198.54.130:80
[Fri May 19 07:51:38 2006] [debug] proxy_util.c(1951): proxy: 
connected /Community/ruralmeded/admissions/admissions_and_medical_errors.htm to 
192.198.54.130:80
[Fri May 19 08:00:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/coping.htm
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1378): [client 65.38.102.153] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/coping.htm
[Fri May 19 08:00:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/coping.htm
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/coping.htm to 192.198.54.130:80
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/coping.htm to 192.198.54.130:80
[Fri May 19 08:00:19 2006] [debug] proxy_util.c(1378): [client 65.38.102.153] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/footer.js, referer: 
http://www.unmc.edu/bioterrorism/coping.htm
[Fri May 19 08:14:21 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/calm.htm
[Fri May 19 08:14:21 2006] [debug] proxy_util.c(1378): [client 65.214.44.82] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 08:14:21 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/calm.htm
[Fri May 19 08:14:21 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 08:14:21 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/calm.htm to 192.198.54.130:80
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/minimed/smallpox.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:20:15 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/system.htm
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/system.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:20:15 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/system.htm
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/system.htm to 192.198.54.130:80
[Fri May 19 08:20:15 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/system.htm to 192.198.54.130:80
[Fri May 19 08:20:16 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 08:20:16 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/newsarchive.htm, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:20:16 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/newsarchive.htm
[Fri May 19 08:20:16 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 08:20:16 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/newsarchive.htm to 192.198.54.130:80
[Fri May 19 08:27:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/bioterrorism/
[Fri May 19 08:27:19 2006] [debug] proxy_util.c(1378): [client 64.34.145.194] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/bioterrorism/
[Fri May 19 08:27:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/bioterrorism/
[Fri May 19 08:27:19 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/bioterrorism/ to 192.198.54.130:80
[Fri May 19 08:27:19 2006] [debug] proxy_util.c(1951): proxy: 
connected /bioterrorism/ to 192.198.54.130:80
[Fri May 19 08:44:53 2006] [debug] proxy_util.c(1378): [client 204.248.22.161] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/headerbar.js, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/, referer: 
http://images.google.com/imgres?
imgurl=http://www.unmc.edu/rural/mederrors/images/logo-
graphics.jpg&imgrefurl=http://www.unmc.edu/rural/mederrors/&h=181&w=209&sz=11&tb
nid=vKuIcgUlFtYkKM:&tbnh=86&tbnw=100&hl=en&start=19&prev=/images%3Fq%
3Dmedication%2Berrors%26svnum%3D10%26hl%3Den%26lr%3D
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/ to 192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/ to 192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/department.js, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/images/logo-partners.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/images/logo-partners.jpg, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/images/logo-partners.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/images/logo-partners.jpg to 
192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/images/logo-partners.jpg to 192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //192.198.54.130/rural/mederrors/images/logo-contact.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/mederrors/images/logo-contact.jpg, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving 
URL http://192.198.54.130/rural/mederrors/images/logo-contact.jpg
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://192.198.54.130/rural/mederrors/images/logo-contact.jpg to 
192.198.54.130:80
[Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: 
connected /rural/mederrors/images/logo-contact.jpg to 192.198.54.130:80
[Fri May 19 08:50:26 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/rural/images/getacro.gif, referer: 
http://www.unmc.edu/rural/mederrors/
[Fri May 19 08:54:32 2006] [error] [client 24.252.13.39] request failed: error 
reading the headers
[Fri May 19 09:16:43 2006] [debug] proxy_util.c(1378): [client 137.197.145.122] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/5/unmc.css, referer: http://www.unmc.edu/bioterrorism/
[Fri May 19 09:16:43 2006] [debug] proxy_util.c(1378): [client 137.197.145.122] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/footer.js, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 09:32:06 2006] [debug] proxy_util.c(1378): [client 138.163.0.38] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/template04/headerbar.js, referer: 
http://www.unmc.edu/bioterrorism/
[Fri May 19 09:32:19 2006] [debug] proxy_util.c(1378): [client 138.163.0.44] 
proxy: http: found worker http://192.198.54.130/ for 
http://192.198.54.130/5/unmc.css, referer: 
http://www.unmc.edu/bioterrorism/default.htm
Comment 23 Mitch McKenzie 2006-05-19 14:44:41 UTC
oops...

sorry, didn't mean to post twice.
Comment 24 Ruediger Pluem 2006-05-25 12:40:39 UTC
Does anybody still observe segfaults with 2.2.2 or is it "only" the "error reading
status line" that is left?
Comment 25 Christophe Yayon 2006-05-25 17:43:46 UTC
not me...
Comment 26 Mitch McKenzie 2006-05-30 21:51:34 UTC
nope... no segfaults.
Comment 27 Olivier BOEL 2006-06-26 13:47:56 UTC
Same problem, here :

Configuration : Apache 2.2.2 (with worker MPM) on Solaris 9 proxying to an IIS 
6.0 back-end server.

No core dump produced.

As I understand, truss (see below) shows that the process uses a file (25, in 
the example below) handle to communicate with the back-end server but does not 
check whether it is still usable or not.
This file handle seems to be used from the connection pool ("As of Apache 2.1, 
the ability to use pooled connections to a backend server is available").

Result from truss :
3313/19:         3.9623 read(25, 0xFDA7B997, 1)                         Err#131 
ECONNRESET
3313/19:         3.9624 write(11, 0xFDA79938, 126)                      = 126
3313/19:           [ M o n   J u n   2 6   1 4 : 3 7 : 3 6   2 0 0 6 ]   [ d e 
b u
3313/19:           g ]   p r o x y _ u t i l . c ( 2 1 4 1 ) :   p r o x y :   
H T
3313/19:           T P :   c o n n e c t i o n   c o m p l e t e   t o   1 3 
6 . 1
3313/19:           7 3 . 2 1 . 2 4 3 : 8 8 8 8   ( e i c i t e s t v s 0 9 )\n
3313/19:         3.9628 writev(25, 0xFDA7B780, 3)                       Err#32 
EPIPE
3313/19:                iov_base = 0x003F2278  iov_len = 1057
3313/19:           P O S T   / E i c i - A D M / A u t h e n t i c a t i o n / 
u s
...
3313/1:          3.9628     Received signal #13, SIGPIPE, in read() [ignored]
3313/19:         3.9635 write(11, 0xFDA795B8, 123)                      = 123
3313/19:           [ M o n   J u n   2 6   1 4 : 3 7 : 3 6   2 0 0 6 ]   [ i n 
f o
3313/19:           ]   [ c l i e n t   1 3 6 . 1 7 3 . 2 1 . 2 4 3 ]   ( 3 2 ) 
B r
3313/19:           o k e n   p i p e :   c o r e _ o u t p u t _ f i l t e 
r :   w
3313/19:           r i t i n g   d a t a   t o   t h e   n e t w o r k\n
3313/19:         3.9639 read(25, 0x003D6208, 8000)                      = 0
3313/19:         3.9641 write(11, 0xFDA77910, 158)                      = 158
3313/19:           [ M o n   J u n   2 6   1 4 : 3 7 : 3 6   2 0 0 6 ]   [ e r 
r o
3313/19:           r ]   [ c l i e n t   1 3 6 . 1 7 3 . 2 2 . 1 3 7 ]   p r o 
x y
3313/19:           :   e r r o r   r e a d i n g   s t a t u s   l i n e   f r 
o m
3313/19:             r e m o t e   s e r v e r   e i c i t e s t v s 0 9 ,   r 
e f
3313/19:           e r e r :   h t t p : / / e i c i a p p q u a / G E D A /\n
3313/19:         3.9656 write(11, 0xFDA77888, 186)                      = 186
3313/19:           [ M o n   J u n   2 6   1 4 : 3 7 : 3 6   2 0 0 6 ]   [ e r 
r o
3313/19:           r ]   [ c l i e n t   1 3 6 . 1 7 3 . 2 2 . 1 3 7 ]   p r o 
x y
3313/19:           :   E r r o r   r e a d i n g   f r o m   r e m o t e   s e 
r v
3313/19:           e r   r e t u r n e d   b y   / E i c i - A D M / A u t h e 
n t
3313/19:           i c a t i o n / u s e r i n f o s e c . a s p ,   r e f e r 
e r
3313/19:           :   h t t p : / / e i c i a p p q u a / G E D A /\n
3313/19:         3.9658 write(11, 0xFDA798D0, 111)                      = 111
3313/19:           [ M o n   J u n   2 6   1 4 : 3 7 : 3 6   2 0 0 6 ]   [ d e 
b u
3313/19:           g ]   p r o x y _ u t i l . c ( 1 8 1 6 ) :   p r o x y :   
H T
3313/19:           T P :   h a s   r e l e a s e d   c o n n e c t i o n   f o r
3313/19:           ( e i c i t e s t v s 0 9 )\n
3313/19:         3.9661 close(25)                                       = 0
3313/19:         3.9664 read(24, 0x003EE268, 8000)                      Err#11 
EAGAIN
3313/19:         3.9666 writev(24, 0xFDA7BBF0, 2)                       = 729
3313/19:                iov_base = 0x003F4280  iov_len = 188
3313/19:           H T T P / 1 . 1   5 0 2   P r o x y   E r r o r\r\n D a t e :
Comment 28 braini 2006-06-26 15:54:59 UTC
(In reply to comment #24)
> Does anybody still observe segfaults with 2.2.2 or is it "only" the "error 
reading
> status line" that is left?

Same problem here:
No sefaults, but error reading status line
Config: Solaris 8, MPM Worker, HTTPd 2.2.2 mod_proxy to IIS 6 on Windows
No error with MPM Prefork and HTTPd 2.0.54
Comment 29 braini 2006-07-05 16:05:32 UTC
With the following 'Murxaround' it's working now:
- Added 'keeplive Off' in httpd.conf  -> No effect
- Set keep-alive timeout to 1 day in IIS -> No effect
- Added
 'SetEnv force-proxy-request-1.0 1'
 'SetEnv proxy-nokeepalive 1'
 in httpd.conf
It's working now without errors since 30 hours...
Don't know if first steps were necessary, I let them in place

Comment 30 Olivier BOEL 2006-08-07 09:35:11 UTC
As workaround, I made the following modification in modules/proxy/proxy_util.c 
(from source files) and changed line 1972 :
BEFORE : if (APR_STATUS_IS_EOF(socket_status))
AFTER : if (APR_STATUS_IS_EOF(socket_status) || APR_STATUS_IS_ECONNRESET
(socket_status))

configure
make
make install

Restart Apache

No more message "proxy: error reading status line from remote server"

Hope this helps...
Comment 31 James 2006-08-14 11:56:04 UTC
(In reply to comment #30)
> As workaround, I made the following modification in modules/proxy/proxy_util.c 
> (from source files) and changed line 1972 :
> BEFORE : if (APR_STATUS_IS_EOF(socket_status))
> AFTER : if (APR_STATUS_IS_EOF(socket_status) || APR_STATUS_IS_ECONNRESET
> (socket_status))
> Hope this helps...

That works perfectly. Apache no longer reports and error, when IIS uses a RST
packet to close an idle back end connection, and the request is serviced correcly.

Many thanks

James
Comment 32 Nick Kew 2006-08-14 13:23:51 UTC
That fix (Olivier BOEL, comment #30) looks right to me, and works for two of 
you.  I'm applying it to /trunk/.

Others, if you're in a position to recompile mod_proxy, please apply the patch 
and report back.
Comment 33 Danny 2006-08-17 07:58:37 UTC
Apache 2.2.3 compiled with mod_proxy used as rev.proxy for Owa.

Works like a charm. I no longer see the '502 Proxy error' when clients try to
connect, while before this error showed 20 times per hour. A big improvement. I
still have some minor error logging issues with ActiveSync but these may be
config issues as well.

Is this going to be adressed in an official patch?

Many thanks,

Danny 
Comment 34 anneb 2006-09-01 16:00:25 UTC
The error message from this bug also appears when:
a large and therefore slow incoming request over mobile GPRS line takes more
than about 15 secs to complete.

The following workaround works well:
add the following two lines to file httpd.conf:
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
Comment 35 Cliff Woolley 2006-10-31 18:06:45 UTC
(In reply to comment #32)
> That fix (Olivier BOEL, comment #30) looks right to me, and works for two of 
> you.  I'm applying it to /trunk/.
> Others, if you're in a position to recompile mod_proxy, please apply the patch 
> and report back.

I can repro what I believe is this problem as well on a 2.2.3 server on linux
doing a reverse proxy to IIS using the prefork MPM.  I'm going to apply the
patch locally and see if it clears up.  Will report back tomorrow.
Comment 36 Cliff Woolley 2006-11-01 05:56:10 UTC
> I can repro what I believe is this problem as well on a 2.2.3 server on linux
> doing a reverse proxy to IIS using the prefork MPM.  I'm going to apply the
> patch locally and see if it clears up.  Will report back tomorrow.

So far so good.  :)

Will keep watching it throughout the day.
Comment 37 Cliff Woolley 2006-11-01 09:48:57 UTC
(In reply to comment #36)
> So far so good.  :)

I've seen absolutely zero instances of this problem since applying the patch. 
I'm going to go ahead and declare it fixed.  Furthermore, the patch does appear
correct to me, especially given the details given in comment #17.

+1 (emeritus) for backport to the 2.2 branch.
Comment 38 junkymail50 2006-11-03 03:18:50 UTC
Just to add, the SetEnv workaround also seems to work for httpd 2.0.50 which I'm
still using.  I did not try compiling the code fix since we need to use official
releases only.  However, if the SetEnv works, then the code fix likely works, in
case it needs to be backported as well.

Thanks for all the great info and the workaround.  Hope the fix gets into the
next release.
Comment 39 Cedric Anes 2006-11-08 01:11:08 UTC
Fix unfortunately does'nt work with my configuration.

httpd-2.2.3
configure option:
--enable-proxy
--enable-proxy-balancer
--enable-proxy-http
--with-mpm=worker

Found expected code at modules/proxy/proxy_util.c:1972 and patch it 
(configure/make/make install/restart)

Still have :
proxy: error reading status line from remote server www.myserver.com
proxy: Error reading from remote server returned by /myurl.htm

I've tried to add the following 2 directives in httpd.conf:
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

Restarted apache. same errors occur.

I've tried debug Loglevel, but I don't get any other messages when it comes to 
that error.

Any ideas ?
Comment 40 Jim Jagielski 2006-11-08 09:40:19 UTC
In 2.2.4-dev and trunk
Comment 41 Nick Kew 2007-01-11 08:33:30 UTC
*** Bug 41150 has been marked as a duplicate of this bug. ***
Comment 42 Jan P. Sorensen 2007-01-14 05:21:20 UTC
*** Bug 39587 has been marked as a duplicate of this bug. ***
Comment 43 Jan P. Sorensen 2007-01-14 05:24:44 UTC
*** Bug 39499 has been marked as a duplicate of this bug. ***
Comment 44 Mark Round 2007-02-08 05:27:43 UTC
Just like to point out that I'm seeing this all the time still. I have upgraded
to httpd-2.2.4, switched to using prefork instead of worker, added the SetEnv
configuration options mentioned previously, tried various backend server
combinations of KeepAlive etc.

Running httpd-2.2.4, prefork on Solaris 10 Sparc (T1000), caching and proxying
to httpd-2.2.4 prefork on Solaris 10 AMD64.

Thu Feb 08 11:37:44 2007] [error] [client 88.110.18.62] proxy: error reading
status line from remote server www1.mydomain.co.uk,
referer: http://www.mydomain.co.uk/forums/addpost.php
[Thu Feb 08 11:37:44 2007] [error] [client 88.110.18.62] proxy: Error reading
from remote server returned by /forums/postlist.php, referer:
http://www.mydomain.co.uk/forums/addpost.php

Comment 45 Nick Kew 2007-02-08 06:52:26 UTC
Did you definitely upgrade mod_proxy?

If you ran your 2.2.4 configure without *explicitly* compiling mod_proxy, then
it wont have compiled it, so it'll still be using your previous mod_proxy.  Just
setting, for example --enable-mods-shared=all is not sufficient.
Comment 46 Andy Weston 2007-02-09 14:22:45 UTC
I have to say that I am still seeing these errors as well.
I'm using RedHat and I built and installed a new rpm for http-2.2.4 so I'm 
sure that I compiled in the new mod_proxy.  I verified that the source fix is 
there in the .c file as well.

[Fri Feb 09 10:18:21 2007] [error] [client xxx] proxy: error reading status 
line from remote server 127.0.0.1
[Fri Feb 09 10:18:21 2007] [error] [client xxx] proxy: Error reading from 
remote server returned by /search/guide/query

I am running a cluster of Ruby/Mongrel servers behind a proxy balancer.
--Andy
Comment 47 David Wallgren 2007-03-01 20:46:27 UTC
I saw this error in my new environment.  The cause of my error was by the
ExceptionNotifier.exception_recipients = $exception_recipients line in the
environment.rb
which calls the smtp.rb.  This was hanging due to environment restrictions.  I added
ActionMailer::Base.server_settings = {
  :address  => 'somemachine',
  :domain  => 'somedoamin.com',
  :port  => 25,
  :user_address => 'someuser@somedomain.com',
  :user_name  => 'someuser',
  :password  => 'somepassword',
  :authentication=>:login
    }
to the production.rb and the problem was resolve.  Didn't happen much since we
write such good code. ;)  
Comment 48 Klavs Klavsen 2007-04-18 07:09:35 UTC
I have this problem with apache-2.2.4 (and 2.0.59) on Windows 2003 SP1. It
proxies (via a rewriterule - which btw. means I can't set the ttl etc. timings
enabled for ProxyPass in v2.2 :( )

I reproduce it, via a stresstest setup.

I have tried to set: 
    SetEnv force-proxy-request-1.0 1


    SetEnv proxy-nokeepalive 1
in the virtualhost - but it has no effect.
I have also tried to disable keepalive on the local IIS backend (and keeping the
SetEnv's ) but to no effect either.

in LogLevel debug I get this:
[Wed Apr 18 16:03:12 2007] [error] [client x.x.x.x] proxy: error reading status
line from remote server www.sitename.dk

[Wed Apr 18 16:03:12 2007] [error] [client x.x.x.x] proxy: Error reading from
remote server returned by /apps/file.dll/forside

[Wed Apr 18 16:03:12 2007] [debug] proxy_util.c(1816): proxy: HTTP: has released
connection for (*)

Anything I can try - I'm out of options :(

p.s. I have set max ephemeral ports to 5000 and set tcptimedwaitdelay to 30
secs, to avoid running into the 10048 OS error (every so often) - in apache 2.2
this has the unfortunate side effect of disabling the site for the standard
timeout seconds, since mod_proxy has this new "feature" - and I can't set the
disable-time, as I'm using mod_proxy via mod_rewrite :(
Comment 49 Klavs Klavsen 2007-04-18 07:10:58 UTC
I have set max ephemeral ports to 65534 ofcourse - 5000 is the default :)
Comment 50 Ruediger Pluem 2007-04-18 13:00:23 UTC
(In reply to comment #48)
> I have this problem with apache-2.2.4 (and 2.0.59) on Windows 2003 SP1. It
> proxies (via a rewriterule - which btw. means I can't set the ttl etc. timings
> enabled for ProxyPass in v2.2 :( )
> 
> I reproduce it, via a stresstest setup.
> 
> I have tried to set: 
>     SetEnv force-proxy-request-1.0 1
> 
> 
>     SetEnv proxy-nokeepalive 1
> in the virtualhost - but it has no effect.

I don't think that your problem is the same problem as described by many others
here. It is also no surprise to me that force-proxy-request-1.0 and
proxy-nokeepalive show no effect in your situation as you are using mod_rewrite
to do reverse proxying to a single server (a balancer would be a different
story). In this case httpd NEVER reuses the backend connections. For each
request to the backend a new one will be opened.


> [Wed Apr 18 16:03:12 2007] [debug] proxy_util.c(1816): proxy: HTTP: has released
> connection for (*)

Above (the star) proves that you are using the 'reverse worker' which never
reuses backend connections. 
So it looks like your fresh backend connections get broken for some other reasons.
Comment 51 Grant Kelly 2007-05-02 13:35:35 UTC
I'm seeing this pretty frequently with Apache 2.2.4 on Solaris 10 using preform
MPM. The proxy is forwarding through a firewall, so I have disabled KeepAlive on
both internal and external servers. The internal runs Apache 2.0.55. I cannot
replicate the error myself, so testing solutions involves trying something and
then waiting to see if the error still pops up.

Not sure if this is an issue, but I actually have a server name in "...status
line from remote server (null)".
Comment 52 Grant Kelly 2007-05-02 16:21:55 UTC
(In reply to comment #51)
> I'm seeing this pretty frequently with Apache 2.2.4 on Solaris 10 using preform
> MPM. The proxy is forwarding through a firewall, so I have disabled KeepAlive on
> both internal and external servers. The internal runs Apache 2.0.55. I cannot
> replicate the error myself, so testing solutions involves trying something and
> then waiting to see if the error still pops up.
> 
> Not sure if this is an issue, but I actually have a server name in "...status
> line from remote server (null)".

Nevermind, it is the firewall inbetween that is dropping the connections. This
is not an Apache bug for me. Thanks.

Comment 53 Jo 2007-05-14 11:47:33 UTC
I also have this bug running Apache 2.2.4 on Solaris 9, worker MPM.
The SetEnv settings do not make it go away for me.

My setup is Apache doing reverse proxy + caching in front of a Tomcat server (no
firewalls in between, running on the same machine in fact).

Problem seems to happen under high load. An interesting fact is that it happens
around the time the pages should expire (I control this from Tomcat by setting
the "Expires" header). So maybe something is wrong with some header processing?

Running the same setup but using mod_jk instead of mod_proxy works fine for me.
Comment 54 Paul Kmiec 2007-05-21 00:53:06 UTC
I believe I'm running into this problem as well. My situation is similar to post
#46 except I'm on apache 2.2.3 (64bit). I noticed the problem immediately after
deploying my rails app to a new server; my mongrels completely freeze after
little bit of inactivity. Some information about my setup:

Linux web001 2.6.16.27-0.9-smp #1 SMP Tue Feb 13 09:35:18 UTC 2007 x86_64 x86_64
x86_64 GNU/Linux

<VirtualHost *:80>
  ServerName demo
  DocumentRoot '/demo/public'

  <Location />
    Order deny,allow
    Allow from all
  </Location>

  <Directory '/demo/public'>
    Options FollowSymLinks
    AllowOverride None
  </Directory>

  RewriteEngine On

  # Check for maintenance file and redirect all requests
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance_demo.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance_demo.html
  RewriteRule ^.*$ /system/maintenance_demo.html [PT]

  # Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://demo_balancer%{REQUEST_URI} [P,QSA,L]
</VirtualHost>

<Proxy balancer://demo_balancer>
  BalancerMember http://localhost:3528
  BalancerMember http://localhost:3529
</Proxy>

May 21 07:46:25 web001 httpd2-prefork[12751]: [error] [client x.x.x.x] proxy:
error reading status line from remote server localhost, referer: https://x.x.x.x/
May 21 07:46:25 web001 httpd2-prefork[12751]: [error] [client x.x.x.x] proxy:
Error reading from remote server returned by /, referer: https://x.x.x.x/

Using the SetEnv solution in the virtual host did not work.
Paul
Comment 55 Ruediger Pluem 2007-05-21 13:53:38 UTC
(In reply to comment #54)
> I believe I'm running into this problem as well. My situation is similar to post
> #46 except I'm on apache 2.2.3 (64bit). I noticed the problem immediately 

First please try with 2.2.4 as it contains a fix. Second, if things do not work
with

SetEnv proxy-nokeepalive 1

(regardles of 2.2.3 / 2.2.4) then this looks similar to #48 for me and it looks
like your fresh backend connections gets broken for some other reasons that are
not caused by the bug of this PR.
Comment 56 jfclere 2007-06-12 23:50:20 UTC
*** Bug 39629 has been marked as a duplicate of this bug. ***
Comment 57 Scott Campbell 2007-06-21 13:50:29 UTC
I too am having this issue.
Running apache 2.2.4 on linux 64-bit, reverse proxy to IIS 6 backend running on 
Windows 2003.

Apache Errors:
[error] [client x.x.x.x] proxy: error reading status line from remote server 
y.y.y.y, referer: https://foo.bar.
[error] [client x.x.x.x] proxy: Error reading from remote server returned 
by /URL/blah

We are using mod_proxy.  Errors seem very random, and tough to pin-point.  When 
we use AJP, problem is gone.

We have very high volume with this setup.

Both env's are set:
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

We have tried no keepalives on the IIS side, large keepalives, but no 
difference.

There is a PIX firewall in between, but we are not losing packets with that.

Any ideas are indeed welcome.
Comment 58 David Svejda 2007-07-01 13:37:57 UTC
Hi all,

I have the same problem. I've just installed Apache 2.2.4 on the frontend
(reverse proxy) server, on the backend server there is Apache 2.2.3 version
(Debian Etch package).

I still do have these problems:

[Sun Jul 01 22:21:26 2007] [error] [client xx.xx.xx.xx] proxy: error reading
status line from remote server backend.xxxx.com
[Sun Jul 01 22:21:26 2007] [error] [client xx.xx.xx.xx] proxy: Error reading
from remote server returned by /articles/

According to Changelog of 2.2.4 this should be fixed if I'm not mistaken.
Comment 59 Tapan 2007-07-11 18:05:37 UTC
Hi all
We are using apache 2.2.4 and proxying to jboss 4.0.2 server. 

Our users reported a problem of 2 trade failures. When we checked apache logs, 
we noticed a slowdown in response time (5 sec and 7 sec respectively)
Seems like request was 'waiting' 'in' apache because it hit our jboss server 
after 5 and 7 second delay ( usual delay time is in millisecond level).

On checking in error logs we saw this error ->

[Tue Jun 05 14:14:16 2007] [error] [client xxx.xxx.19.41] proxy: error reading 
status line from remote server 10.10.xxx.xx
[Tue Jun 05 14:14:16 2007] [error] [client xxx.xxx.19.41] proxy: Error reading 
from remote server returned by /isCxxx/Heartbeat.do
[Tue Jun 05 14:14:16 2007] [error] [client xxx.xxx.19.41] File does not 
exist: /xxxx/opt/apache2/htdocs/maintenance.html

On deeper analysis, we found that this error message comes 2-3 times an hour in 
our logs consistently. But we were not noticing it because it was not causing 
any problems.
However, during the time when trades were rejected, we have 20-25 instances of 
this error. 
So it looks like due to this error response time gets increased.

I am ready to provide more information if required.
Tapan
Comment 60 Jeff Johnson 2007-08-23 16:31:29 UTC
I have created a very simple test for this.

a Fresh Ubuntu Install with the latest Gutsy Apache2.2.4 on it

I have done it as a prefork and as a worker both fail

here is the error log

[Thu Aug 23 19:20:09 2007] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving
URL http://192.168.0.110:2054/
[Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(1798): proxy: HTTP: has acquired
connection for (192.168.0.110)
[Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(1859): proxy: connecting
http://192.168.0.110:2054/ to 192.168.0.110:2054
[Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(1955): proxy: connected / to
192.168.0.110:2054
[Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(2050): proxy: HTTP: fam 2 socket
created to connect to 192.168.0.110
[Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(2146): proxy: HTTP: connection
complete to 192.168.0.110:2054 (192.168.0.110)
[Thu Aug 23 19:22:38 2007] [error] [client 192.168.0.1] proxy: error reading
status line from remote server 192.168.0.110
[Thu Aug 23 19:22:38 2007] [error] [client 192.168.0.1] proxy: Error reading
from remote server returned by /ceservice.cetix
[Thu Aug 23 19:22:38 2007] [debug] proxy_util.c(1816): proxy: HTTP: has released
connection for (192.168.0.110)
[Thu Aug 23 19:22:38 2007] [debug] mod_proxy_balancer.c(521):
proxy_balancer_post_request for (balancer://demo_balancer)


Here is the sites-available file...

Pretty straight forward.  

NameVirtualHost *
<Proxy balancer://demo_balancer>
  BalancerMember http://192.168.0.110:2054
#  BalancerMember http://192.168.0.110:7054
</Proxy>
<VirtualHost *>
        ServerAdmin webmaster@localhost
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]
RewriteLog "/var/log/apache2/rewrite.log"
RewriteLogLevel 9
RewriteRule ^/(.*.cetix) balancer://demo_balancer [P,QSA,L]


        DocumentRoot /var/www/
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
        <Directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
                # This directive allows us to have apache2's default start page
                # in /apache2-default/, but still have / go to the right place
                #RedirectMatch ^/$ /apache2-default/
        </Directory>

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory "/usr/lib/cgi-bin">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        </Directory>

        ErrorLog /var/log/apache2/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel debug

        CustomLog /var/log/apache2/access.log combined
        ServerSignature On

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

</VirtualHost>
~


This is forwarding to another web service on a local network.  I can talk
directly to the web service just fine.

I need to get this resolved any advice is great.

Jeff

Comment 61 Nick Kew 2007-09-08 06:18:23 UTC
*** Bug 32731 has been marked as a duplicate of this bug. ***
Comment 62 Andrew Robinson 2007-09-10 15:12:08 UTC
I have compiled the latest httpd 2.2.6 with static modules i.e. 

./configure --prefix=/usr/local/apache --enable-modules="asis alias cache 
deflate disk_cache expires file_cache mem_cache headers mime mimemagic include 
log_config proxy proxy_http rewrite setenvif status vhost_alias" --with-
mpm=worker

We have IIS 6.0 as backend webservers and a BigIP F5 for load balancing in 
front of them. The Apache sits in a different geographical location and 
connects to the IIS webservers thru a VPN connection. The virtual host config 
is as follows:

<VirtualHost xx.xx.xx.xx>
        ServerName www.abc.com
        ServerAlias abc.com


        ProxyRequests Off
        ProxyBadHeader Ignore
        SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1


        ProxyPass / http://www.abc.com/  smax=5 max=20 ttl=120 retry=1 
keepalive=Off
        #ProxyPass / http://www.abc.com/
        ProxyPassReverse / http://www.abc.com/
</VirtualHost>

[Mon Sep 10 17:36:59 2007] [error] ap_proxy_connect_backend disabling worker 
for (www.ABC.com)
[Mon Sep 10 17:36:59 2007] [error] (110)Connection timed out: proxy: HTTP: 
attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed
[Mon Sep 10 17:37:00 2007] [error] (110)Connection timed out: proxy: HTTP: 
attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed
[Mon Sep 10 17:37:00 2007] [error] ap_proxy_connect_backend disabling worker 
for (www.ABC.com)
[Mon Sep 10 17:37:00 2007] [error] (110)Connection timed out: proxy: HTTP: 
attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed
[Mon Sep 10 17:37:01 2007] [error] (110)Connection timed out: proxy: HTTP: 
attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed
[Mon Sep 10 17:37:01 2007] [error] ap_proxy_connect_backend disabling worker 
for (www.ABC.com)
[Mon Sep 10 17:38:21 2007] [error] [client 200.124.142.132] (70007)The timeout 
specified has expired: proxy: error reading status line from remote server 
www.ABC.com, referer: http://www.ABC.com/
[Mon Sep 10 17:38:21 2007] [error] [client 200.124.142.132] proxy: Error 
reading from remote server returned by /welcome/welcome.aspx, referer: 
http://www.ABC.com/

These errors are appearing randomly and I was monitoring a ping between Apache 
to the F5 and there were no hiccups in the VPN connection. The problem gets 
exaggerated when I use multiple Apache's in front and load balance between 
them.

Please let me know if there is any fix available to the problem or If I should 
not be using Apache to reverse proxy a website being loadbalanced by F5.

Thanks in anticipation:

AS

Comment 63 Aleksey Midenkov 2007-12-03 03:29:19 UTC
Error reading status line can appear due to network timeout and is often 
observed when the 'Timeout' setting is set to low value. This is a normal 
situation.

Comment 64 Jerry Baker 2007-12-03 07:48:50 UTC
If it is a normal situation, perhaps it shouldn't be logged to the error log?
Comment 65 Aleksey Midenkov 2007-12-06 02:03:12 UTC
It is a normal situation in terms of httpd functioning. The causes of 'error 
reading status line' are:

(70007)The timeout specified has expired -- indicates network timeout, 
configured by Timeout directive
(104)Connection reset by peer -- TCP RST was sent from server
(70014)End of file found -- TCP FIN was sent from server

All these messages appearing in error.log indicate some problem with network on 
server side or with server itself and do not mean errors in proxy software. I 
think the bug should be closed.
Comment 66 anneb 2007-12-06 02:52:39 UTC
I think this bug should NOT be closed because the proxy server should be
transparent and it is not as long this problem remains. Due to this bug, the
system behaves differently when the proxy server is in between the client and
the server. Direct connections to the server do not produce this error message
in clients. When the workarounds proposed are in place, the error is not
reproduced either.

Now the proxy module produces an error message, thus moving problems on the TCP
network layer to the http layer. I think it is not practical to start proxying
the TCP layer messages, so instead the http proxy should react to TCP layer
problems in the same way a http client would.
Comment 67 Jerry Baker 2007-12-06 08:20:33 UTC
(In reply to comment #65)
> All these messages appearing in error.log indicate some problem with network on 
> server side or with server itself and do not mean errors in proxy software. I 
> think the bug should be closed.

What data are you privy to that causes you to believe that there isn't a bug
causing the proxy module to believe one of those conditions is true when it, in
fact, is false?
Comment 68 Aleksey Midenkov 2007-12-06 08:59:50 UTC
(In reply to comment #67)
And what data can be previed to the belief that there is a bug? Do you think 
there can be a bug in apr functions that are called from ap_rgetline_core or do 
you think that there can be a bug in ap_rgetline_core itself? If this could be 
the case, then we would see the bug not only in "reading status line" but 
everywhere!
Comment 69 Jerry Baker 2007-12-06 09:12:13 UTC
I'm not familiar with the intracacies of APR, but what I do know is that I
experience problems with Apache acting as a proxy server that I do not
experience when it is not the proxy server. I may be ignorant of the code, but
to me that points to something about the proxy. I haven't seen anything reported
in this bug that points to another cause other than what appears to be blind
faith that there cannot be anything wrong with the proxy code.
Comment 70 Ruediger Pluem 2007-12-06 13:21:58 UTC
(In reply to comment #67)
> (In reply to comment #65)
> > All these messages appearing in error.log indicate some problem with network on 
> > server side or with server itself and do not mean errors in proxy software. I 
> > think the bug should be closed.
> 
> What data are you privy to that causes you to believe that there isn't a bug
> causing the proxy module to believe one of those conditions is true when it, in
> fact, is false?

He is correct in several cases:

(70007)The timeout specified has expired -- indicates network timeout.

This can happen if your backend takes a long time preparing the response and
this is a wanted behaviour. You can adjust the timeout either via TimeOut,
ProxyTimeout or a specific worker timeout.

(104)Connection reset by peer -- TCP RST was sent from server
(70014)End of file found -- TCP FIN was sent from server

These can happen either because of network problems or problems on the backend
server or because of a race condition in the proxy code.
Currently there is a race condition that the backend server can close the
keepalive connection right after httpd checked the state of this TCP connection
and httpd starts sending a request and waiting on a response on a connection
that is in fact dead. But this should happen rarely and browsers try to resent
the request in such situations provided the request was idempotent (so no POST
request or GET request with parameters).
All other cases (except for the race condition) also lead to error messages in
the browser (maybe a different one that the one sent by the proxy, but an error
message).
So the race condition is the only reason why I think that it is justified to
keep this PR open.

Comment 71 Aleksey Midenkov 2007-12-07 07:31:36 UTC
Created attachment 21243 [details]
backend connection race condition fix (untested)

This should fix such race condition.
Comment 72 Nick Kew 2007-12-07 10:12:30 UTC
(In reply to comment #71)
> Created an attachment (id=21243) [edit]
> backend connection race condition fix (untested)
> 
> This should fix such race condition.

I think it's missing a cleanup before the retry in the APR_ECONNRESET/APR_EOF
case.  But otherwise looks reasonable, if this race condition is indeed the culprit.

It would be good to get some feedback from folks affected by this problem.  Does
the patch fix it for you?
Comment 73 Ruediger Pluem 2007-12-07 14:29:58 UTC
(In reply to comment #71)
> Created an attachment (id=21243) [edit]
> backend connection race condition fix (untested)
> 
> This should fix such race condition.

As explained before this patch is not correct:

1. Once you sent the request body it is no longer available.
2. The RFC does not allow to resend non idempotent requests.

The only fix I can think of is to send a test request over the connection first
that does not trigger (a significant action) on backend side.

OPTIONS *

would be a candidate for this. If retrieve an 200 OK here we can be pretty sure
that the race will not occur
Comment 74 Aleksey Midenkov 2007-12-09 01:07:59 UTC
(In reply to comment #73)
> (In reply to comment #71)
> > Created an attachment (id=21243) [edit] [edit]
> > backend connection race condition fix (untested)
> > 
> > This should fix such race condition.
> 
> As explained before this patch is not correct:
> 
> 1. Once you sent the request body it is no longer available.

This can be fixed.

> 2. The RFC does not allow to resend non idempotent requests.

Why does this applies to our situation when the request does not reach backend 
server?
Comment 75 Ruediger Pluem 2007-12-09 02:57:07 UTC
(In reply to comment #74)
> (In reply to comment #73)
> > (In reply to comment #71)
> > > Created an attachment (id=21243) [edit] [edit] [edit]
> > > backend connection race condition fix (untested)
> > > 
> > > This should fix such race condition.
> > 
> > As explained before this patch is not correct:
> > 
> > 1. Once you sent the request body it is no longer available.
> 
> This can be fixed.

Not easily as buffering large request entities is a pain.

> 
> > 2. The RFC does not allow to resend non idempotent requests.
> 
> Why does this applies to our situation when the request does not reach backend 
> server?

With your loop you don't know. Once you sent a single byte of the request body
or even only the GET request with parameters you do not really now what the
backend server did. Didn't it receive it or did it receive it, processed it and
just in the moment as it wanted to send the reply the network broke down or the
process that processed the request died.


Comment 76 Aleksey Midenkov 2007-12-10 02:32:36 UTC
(In reply to comment #75)
> With your loop you don't know. Once you sent a single byte of the request 
body
> or even only the GET request with parameters you do not really now what the
> backend server did. Didn't it receive it or did it receive it, processed it 
and
> just in the moment as it wanted to send the reply the network broke down or 
the
> process that processed the request died.
> 

If after FIN was recieved RST is following couldn't we assume that our request 
is sure rejected by remote server's TCP/IP stack?

Besides, I suppose that any client will suffer from such Byzantine problem. 
Where is the difference between mod_proxy and browser behaviour addressing this 
situation?
Comment 77 Bill McGonigle 2007-12-10 21:03:50 UTC
(In reply to comment #63)
> Error reading status line can appear due to network timeout and is often 
> observed when the 'Timeout' setting is set to low value. This is a normal 
> situation.

Ummmm, yeah - thanks Aleksey!  Big autodopeslap here.  I've been watching this
bug for a while and  it never occurred to me to 'de-optimize' the global
Timeout.  I guess I assumed the ProxyTimeout would have superceded Timeout in a
VirtualHost (if I'm reading the code correctly, the recent commit for bug 11540
does just that, but I'm still on 2.2.6).  Anyway, a new Timeout in the affected
VirtualHost fixed this for me (finally - yay!)

I filed bug 44051 asking for a more specific error message.
Comment 78 Nick Kew 2007-12-14 12:29:41 UTC
*** Bug 44079 has been marked as a duplicate of this bug. ***
Comment 79 Marcos Marado 2008-04-28 10:32:25 UTC
It also happens with us. Any thoughts on when the patch existing in Comment #71 will be commited in SVN?
Comment 80 Aleksey Midenkov 2008-04-28 23:23:14 UTC
(In reply to comment #79)
> It also happens with us. Any thoughts on when the patch existing in Comment #71
> will be commited in SVN?
> 

It is really wrong. Don't use it.
Comment 81 Marcos Marado 2008-04-29 01:33:50 UTC
Any thoughts on how to fix it then?
Comment 82 Ruediger Pluem 2008-04-29 04:03:11 UTC
Try to use the patch from r645813 (http://svn.apache.org/viewvc?view=rev&revision=645813). Patch:
svn diff -r645812:645813 https://svn.apache.org/repos/asf/httpd/httpd/trunk
Comment 83 Adam 2008-05-09 12:58:35 UTC
(In reply to comment #82)
> Try to use the patch from r645813
> (http://svn.apache.org/viewvc?view=rev&revision=645813). Patch:
> svn diff -r645812:645813 https://svn.apache.org/repos/asf/httpd/httpd/trunk
> 

So as I understand it, there is a bug regarding a race condition in mod_proxy that is still causing this error.  And the only patch in place is the one above in comment #82, which I noticed is also in trunk.  However, this patch, while perhaps a good one to have in place, doesn't solve the race condition.  And the race condition is the real issue here.

This is a serious problem that is affecting the use of Apache 2.2.8 on many customers.  The frequency of the error with some sites is high enough that it puts Apache mod_proxy in jeopardy of being unusable, especially for non-idempotent requests.  It may be less than 5% of requests at most but that is a lot for a major traffic site.

Are there any new developments on getting this race condition fixed?

Can someone please point me to the area in the code where this race condition exists and describe the complications around it that have prevented it from being fixed thus far?
Comment 84 Ruediger Pluem 2008-05-17 12:56:26 UTC
(In reply to comment #82)
> Try to use the patch from r645813
> (http://svn.apache.org/viewvc?view=rev&revision=645813). Patch:
> svn diff -r645812:645813 https://svn.apache.org/repos/asf/httpd/httpd/trunk
> 

Backported to 2.2.x as r657443 (http://svn.apache.org/viewvc?rev=657443&view=rev).
Comment 85 Johnathan Conley 2008-07-21 12:47:47 UTC
Any chance of getting this fix ported to 2.0?

I'm having the same/similar issue when using mod_proxy to an https target. (2.0.63/worker/linux)
Using the SetEnv hack seems to resolve the issue... but this is not a desirable workaround.
Comment 86 Ruediger Pluem 2008-07-21 12:56:07 UTC
(In reply to comment #85)
> Any chance of getting this fix ported to 2.0?

Should never say never but the probability is very low. There is low developer interest in 2.0.x these days and so expect only security fixes for 2.0.x.
Upgrade to 2.2.x. It has many interesting new features in the proxy module.

Comment 87 Ruediger Pluem 2008-07-31 13:14:59 UTC
*** Bug 45510 has been marked as a duplicate of this bug. ***
Comment 88 Ruediger Pluem 2008-07-31 13:23:28 UTC
Created attachment 22341 [details]
Patch against trunk

Can you please try this patch and add
setenv proxy-initial-not-pooled 1
to your virtualhost configuration.
Note that it comes with a performance penalty.
Comment 89 Mao Morimoto 2008-07-31 18:17:10 UTC
(In reply to comment #87)
> *** Bug 45510 has been marked as a duplicate of this bug. ***

At first, I introduced r657443.
The problem has been improved, but not perfect. 
I was watching "netstat", but sometimes there were "ESTABLISHED" connection ghosts after backend closed them, and in this case "Proxy error" occurred at 100%.
Comment 90 Mao Morimoto 2008-07-31 18:47:28 UTC
(In reply to comment #88)
> Created an attachment (id=22341) [details]
> Patch against trunk
> Can you please try this patch and add
> setenv proxy-initial-not-pooled 1
> to your virtualhost configuration.
> Note that it comes with a performance penalty.

I tried this just now.
It looks working well perfectly. But it has really very large performance problem. Especially, it is very serious with clients which does not have HTTP/1.1 keep-alive like IE on https.
Comment 91 Mao Morimoto 2008-07-31 19:48:49 UTC
(In reply to comment #89)

Here is the debug log related to the "Proxy error" on r657443. It occurred after the backend closes all connection by keep-alive timeout and there were connection ghost.

[Fri Aug 01 11:33:07 2008] [info] Initial (No.1) HTTPS request received for child 2 (server example.com:443)
[Fri Aug 01 11:33:07 2008] [debug] mod_proxy_http.c(55): proxy: HTTP: canonicalising URL //192.168.1.100/example.gif
[Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(1488): [client 192.168.3.125] proxy: http: found worker http://192.168.1.100/ for http://192.168.1.100/example.gif
[Fri Aug 01 11:33:07 2008] [debug] mod_proxy.c(966): Running scheme http handler (attempt 0)
[Fri Aug 01 11:33:07 2008] [debug] mod_proxy_http.c(1874): proxy: HTTP: serving URL http://192.168.1.100/example.gif
[Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2044): proxy: HTTP: has acquired connection for (192.168.1.100)
[Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2102): proxy: connecting http://192.168.1.100/example.gif to 192.168.1.100:80
[Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2200): proxy: connected /example.gif to 192.168.1.100:80
[Fri Aug 01 11:33:07 2008] [error] [client 192.168.3.125] (104)Connection reset by peer: proxy: error reading status line from remote server 192.168.1.100
[Fri Aug 01 11:33:07 2008] [debug] mod_proxy_http.c(1395): [client 192.168.3.125] proxy: NOT Closing connection to client although reading from backend server 192.168.1.100 failed.
[Fri Aug 01 11:33:07 2008] [error] [client 192.168.3.125] proxy: Error reading from remote server returned by /apps/example.gif
[Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2062): proxy: HTTP: has released connection for (192.168.1.100)
[Fri Aug 01 11:33:07 2008] [info] [client 192.168.3.125] Connection closed to child 2 with unclean shutdown (server example.com:443)
Comment 92 Ruediger Pluem 2008-08-07 11:49:27 UTC
*** Bug 45588 has been marked as a duplicate of this bug. ***
Comment 93 Ruediger Pluem 2008-10-15 12:57:01 UTC
With the latest fixes in 2.2.10 this is now fixed and reduces itself to a configuration problem.
Comment 94 Nick Kew 2009-01-09 11:20:21 UTC
*** Bug 46504 has been marked as a duplicate of this bug. ***
Comment 95 Mao Morimoto 2009-05-17 22:35:13 UTC
Was this problem really fixed?
I updated my system to 2.2.11, and add new conf lines;

<VirtualHost *:80>
    ServerName   www.example.com
    DocumentRoot /www
    <Location /sub/>
        ProxyPass        http://192.168.60.34/sub/
        ProxyPassReverse http://192.168.60.34/sub/
        SetEnv           force-proxy-request-1.0 1
        SetEnv           proxy-nokeepalive       1
    </Location>
</VirtualHost>

But, I still have same problem, and the error log is;

[Mon May 18 09:30:48 2009] [error] [client ***.***.***.***] (104)Connection reset by peer: proxy: error reading status line from remote server 192.168.60.34
[Mon May 18 09:30:48 2009] [error] [client ***.***.***.***] proxy: Error reading from remote server returned by /sub/index.html

Does anyone know how to fix this?
Comment 96 Mao Morimoto 2009-05-17 23:52:31 UTC
Sorry!
I just forgot to add "SetEnv proxy-initial-not-pooled 1".
Now it seems to working well, but really too slow....
Comment 97 kirant400 2010-05-12 10:12:43 UTC
I was using apache 2.2.3 and internal web server is IIS 6 hosted on windows 2003.
Initially i have tried
SetEnv           force-proxy-request-1.0 1
SetEnv           proxy-nokeepalive       1
SetEnv proxy-initial-not-pooled 1
With this change the frequency of getting the proxy error is reduced.
And now upgraded to 2.2.15 and got the same result after adding "SetEnv" hack.

I have configured the proxy time out to 900
Timeout 900
httpd keepalive option on and off 

IIS Keep alive time is also 900.

The error is consistent if the request made is time consuming. and if approx 10 simultaneous request has send to the server.

I was behind this problem for more than 8 months and still no hope :(

Appreciate your help!

--kirant400
Comment 98 ibram 2010-09-20 08:50:42 UTC
Our hoster company is using an Apache 2.2.9 as Reverse-Proxy. The web server is IIS 6 on windows 2003.
We tried also
SetEnv           force-proxy-request-1.0 1
SetEnv           proxy-nokeepalive       1

but we get the error like kirant400 if the request is time consuming.
Now our hoster and I changed some settings and we have NOT this issue anymore.

1. Unchanged "force-proxy-request-1.0 1"
2. Deactivated "proxy-nokeepalive" on apache
3. Apache Keep Alive set to 900
4. IIS Keep Alive set to 900

Can someone try and confirm this.

Thanks,
ibram
Comment 99 Eric Covener 2010-09-20 09:46:17 UTC
ibram, please use the users support list to discuss your issue with verbatim details.
Comment 100 Hytham 2013-05-27 11:49:47 UTC
Hi,

The SetEnv Parameters worked well with my case and I am no longer getting this error but I faced another issue which is HTTP/1.1 417 Expectation Failed.

This is due to the apache client application is using HTTP/1.1 and the Proxy was forced to use HTTP/1.0 as per the setEnv Parameters.

I found a solution for this problem in this URL: https://lists.gnu.org/archive/html/sks-devel/2013-03/msg00017.html

I applied this solution and now my httpd.conf in the proxy server is having the below lines:

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
RequestHeader unset Expect early


Thank you for the support.
Comment 101 Yann Ylavic 2015-02-05 12:30:13 UTC
*** Bug 57520 has been marked as a duplicate of this bug. ***
Comment 102 vinay 2015-10-27 07:35:49 UTC
Hi ,
 Error message shown below:
 Proxy Error
 The proxy server received an invalid response from an upstream server.
 The proxy server could not handle the request GET /.
 Reason: Error reading from remote server
 Apache/2.2.15 (CentOS) Server at calmacprod.cigniti.com Port 86
NameVirtualHost *:85

<VirtualHost *:85>
   ServerName calmacprod.cigniti.com
ProxyRequests Off
ProxyPreserveHost On
<Proxy *>
  Order allow,deny
  Allow from all
</Proxy>
ProxyPass /rm http://159.122.202.173:3060/rm
ProxyPassReverse /rm http://159.122.202.173:3060/rm
ProxyPass / http://159.122.202.173:8080/
ProxyPassReverse / http://159.122.202.173:8080/
<Location /jira>
     Order allow,deny
    Allow from all
</Location>
#SSLProxyEngine on
SSLEngine on
SSLCertificateFile /etc/httpd/ssl/131fd0633495ecf7.crt
SSLCertificateKeyFile /etc/httpd/ssl/calmacprod.cigniti.com.key
ErrorLog logs/jira.cigniti
</VirtualHost>
NameVirtualHost *:86
<VirtualHost *:86>
   ServerName calmacprod.cigniti.com
ProxyRequests Off
ProxyPreserveHost On
<Proxy *>
  Order deny,allow
  Allow from all
</Proxy>
ProxyPass / http://159.122.202.173:8090/
ProxyPassReverse / http://159.122.202.173:8090/
<Location /confluence>
     Order allow,deny
    Allow from all
</Location>
SSLEngine on
SSLCertificateFile /etc/httpd/ssl/131fd0633495ecf7.crt
SSLCertificateKeyFile /etc/httpd/ssl/calmacprod.cigniti.com.key
ErrorLog logs/confluence.cigniti
</VirtualHost>
ReWriteEngine On
ReWriteCond %{HTPS} off
Ratlassian/installation/confluence/jre//bin/java -Djava.util.log                                                        ging.config.file=/atlassian/installation/confluence/conf/logging.properties -Djava.util.logging.manager=org.apac                                                        he.juli.ClassLoaderLogManager -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -Djava.awt.headless=tru                                                        e -Xloggc:/atlassian/installation/confluence/logs/gc-2015-10-27_01-21-59.log -XX:+UseGCLogFileRotation -XX:Numbe                                                        rOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-PrintTenuringDistribution -                                                        Djava.endorsed.dirs=/atlassian/installation/confluence/endorsed -classpath /atlassian/installation/confluence/bi                                                        n/bootstrap.jar:/atlassian/installation/confluence/bin/tomcat-juli.jar -Dcatalina.base=/atlassian/installation/c                                                        onfluence -Dcatalina.home=/atlassian/installation/confluence -Djava.io.tmpdir=/atlassian/installation/confluence                                                        /temp org.apache.catalina.startup.Bootstrap start
postgres 13603 30198  0 01:23 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(47601) idle
postgres 14165 30198  0 01:46 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(48008) idle
postgres 20868 30198  0 04:18 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50767) idle
postgres 20956 30198  0 04:19 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50796) idle
postgres 20957 30198  0 04:19 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50797) idle
postgres 21001 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50817) idle
postgres 21002 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50818) idle
postgres 21011 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50823) idle
postgres 21012 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50824) idle
postgres 21013 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50825) idle
postgres 21014 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50830) idle
postgres 21015 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50831) idle
postgres 21016 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50832) idle
postgres 21017 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50833) idle
postgres 21018 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50834) idle
postgres 21019 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50835) idle
postgres 21020 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50836) idle
postgres 21021 30198  0 04:20 ?        00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50837) idle
root     21040 20801  0 04:21 pts/0    00:00:00 grep confluence
root     28153     1  1 Aug11 ?        1-02:47:44 /atlassian/installation/confluence/jre//bin/java -Djava.util.l                                                        ogging.config.file=/atlassian/installation/confluence/conf/logging.properties -Djava.util.logging.manager=org.ap                                                        ache.juli.ClassLoaderLogManager -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -Djava.awt.headless=t                                                        rue -Xloggc:/atlassian/installation/confluence/logs/gc-2015-08-11_08-37-19.log -XX:+UseGCLogFileRotation -XX:Num                                                        berOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-PrintTenuringDistribution                                                         -Djava.endorsed.dirs=/atlassian/installation/confluence/endorsed -classpath /atlassian/installation/confluence/                                                        bin/bootstrap.jar:/atlassian/installation/confluence/bin/tomcat-juli.jar -Dcatalina.base=/atlassian/installation                                                        /confluence -Dcatalina.home=/atlassian/installation/confluence -Djava.io.tmpdir=/atlassian/installation/confluen                                                        ce/temp org.apache.catalina.startup.Bootstrap start
[root@production ~]# cd /atlassian/installation/confluence/bin/
[root@production bin]#
[root@production bin]# ./stop-confluence.sh
executing using dedicated user
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian                                                        .com/display/DOC/Confluence+Installation+Guide
[root@production bin]# ./startup.sh
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

[root@production logs]# tail -f -n 1000 catalina.out
"I/O dispatcher 9" #90 daemon prio=5 os_prio=0 tid=0x00007f698c093000 nid=0x6e8a runnable [0x00007f69879f8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb0bc1a8> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb0bc1b8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb0bc160> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 8" #89 daemon prio=5 os_prio=0 tid=0x00007f698c009000 nid=0x6e89 runnable [0x00007f6987af9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb15b550> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb15b560> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb15b508> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 7" #88 daemon prio=5 os_prio=0 tid=0x00007f698c007800 nid=0x6e88 runnable [0x00007f6987bfa000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb0dca28> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb0dca38> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb0dc9e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 6" #87 daemon prio=5 os_prio=0 tid=0x00007f698c006000 nid=0x6e87 runnable [0x00007f6967ffe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb0ff338> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb0ff348> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb0ff2f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 5" #86 daemon prio=5 os_prio=0 tid=0x00007f698c00f800 nid=0x6e86 runnable [0x00007f6987cfb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb0bc438> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb0bc448> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb0bc3f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 4" #85 daemon prio=5 os_prio=0 tid=0x00007f698c00e000 nid=0x6e85 runnable [0x00007f6987dfc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb0ff5d8> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb0ff5e8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb0ff590> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 3" #84 daemon prio=5 os_prio=0 tid=0x00007f698c00c000 nid=0x6e84 runnable [0x00007f6987efd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb1dbc08> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb1dbc18> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb1dbbc0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 2" #83 daemon prio=5 os_prio=0 tid=0x00007f698c002000 nid=0x6e83 runnable [0x00007f697fefd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb0dccb8> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb0dccc8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb0dcc70> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 1" #82 daemon prio=5 os_prio=0 tid=0x00007f698c001000 nid=0x6e82 runnable [0x00007f6987ffe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000cb15b7e0> (a sun.nio.ch.Util$2)
        - locked <0x00000000cb15b7f0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000cb15b798> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
        at java.lang.Thread.run(Thread.java:745)

"httpclient-io:thread-1" #81 daemon prio=5 os_prio=0 tid=0x00007f69ec224800 nid=0x6e81 runnable [0x00007f6a08113000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ca3cece0> (a sun.nio.ch.Util$2)
        - locked <0x00000000ca3cecf0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ca3cec98> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)
        at java.lang.Thread.run(Thread.java:745)

"Timer-3" #54 daemon prio=5 os_prio=0 tid=0x00007f69f00b3000 nid=0x6e63 in Object.wait() [0x00007f6a5818f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000c8453ee0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Timer-2" #53 daemon prio=5 os_prio=0 tid=0x00007f69f0029800 nid=0x6e62 in Object.wait() [0x00007f6a0b7fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000c80fb128> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"FelixStartLevel" #52 daemon prio=5 os_prio=0 tid=0x00007f69f4439000 nid=0x6e61 in Object.wait() [0x00007f6a0b5fc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:279)
        - locked <0x00000000c809bda8> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:745)

"FelixDispatchQueue" #51 daemon prio=5 os_prio=0 tid=0x00007f69f400c000 nid=0x6e60 in Object.wait() [0x00007f6a0b6fd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
        - locked <0x00000000c82ffca0> (a java.util.ArrayList)
        at org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
        at org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
        at java.lang.Thread.run(Thread.java:745)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" #44 daemon prio=5 os_prio=0 tid=0x00007f6a63788800 nid=0x6e57 runnable [0x00007f6a58290000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
        at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
        at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1700)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        - locked <0x00000000d5180430> (a org.postgresql.core.v3.QueryExecutorImpl)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLQuery(AbstractJdbc2Connection.java:347)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLQuery(AbstractJdbc2Connection.java:339)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.getTransactionIsolation(AbstractJdbc2Connection.java:868)
        at com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:107)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:198)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
        at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
        at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" #43 daemon prio=5 os_prio=0 tid=0x00007f6a6b55b800 nid=0x6e56 runnable [0x00007f6a58390000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.ClassLoader.findLoadedClass0(Native Method)
        at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1035)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
        - locked <0x00000000c00bf930> (a java.lang.Object)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
        - locked <0x00000000c0010630> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:503)
        at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2640)
        at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:890)
        at sun.util.resources.LocaleData$1.run(LocaleData.java:164)
        at sun.util.resources.LocaleData$1.run(LocaleData.java:160)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.util.resources.LocaleData.getBundle(LocaleData.java:160)
        at sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:124)
        at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:697)
        at java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:146)
        at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
        at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:359)
        at java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:349)
        at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
        at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580)
        at org.postgresql.core.Logger.<init>(Logger.java:26)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:105)
        - locked <0x00000000c74afe48> (a java.lang.Class for org.postgresql.jdbc2.AbstractJdbc2Connection)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
        at org.postgresql.Driver.makeConnection(Driver.java:393)
        at org.postgresql.Driver.connect(Driver.java:267)
        at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
        at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
        at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" #42 daemon prio=5 os_prio=0 tid=0x00007f6a54206800 nid=0x6e55 waiting for monitor entry [0x00007f6a58492000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:104)
        - waiting to lock <0x00000000c74afe48> (a java.lang.Class for org.postgresql.jdbc2.AbstractJdbc2Connection)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
        at org.postgresql.Driver.makeConnection(Driver.java:393)
        at org.postgresql.Driver.connect(Driver.java:267)
        at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
        at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
        at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

[root@production logs]#



Please Advise,
Vinay.
Comment 103 vin01 2016-10-09 06:38:18 UTC
Is it still supposed to be open? I see the same behaviour in 2.4.20, 
Using the "proxy-nokeepalive" to disable keepalives definitely is not the best way to solve this and causes other issues.
I am seeing this with  Jetty(9.3.3.v20150827) as backend.
Comment 104 Eric Covener 2016-10-09 10:57:23 UTC
(In reply to vin01 from comment #103)
> Is it still supposed to be open? I see the same behaviour in 2.4.20, 
> Using the "proxy-nokeepalive" to disable keepalives definitely is not the
> best way to solve this and causes other issues.
> I am seeing this with  Jetty(9.3.3.v20150827) as backend.

If you're hitting a keepalive race, you can also set smax=0 and a ttl lower then your backend keepalive timeout.  There's also proxy-initial-not-pooled.  Either way, a lot more detail is needed in a report.
Comment 105 Francesco 2017-09-29 07:09:24 UTC
I have the same error in this configuration: apache 2.4.6 on a centos 7, configured as https reverse proxy on a filesender server (http://filesender.org/), on a Redhat 5.3 server. Filesender is a software for sending big files with email links.
The error occurs clicking on the download.php link (for downloading files), and only for very big files (several GB I guess). Direct links to filesender server work.
All suggested solutions don't work.
Finally I found the problem, but don't know why it happens: I tought the problem could be in the headers sent by download.php, so I tryied to comment them one by one, so I found that the one which gives error is the one which sends file dimension:

header('Content-Length: '.$functions->getFileSize($file));

Commenting out this one eliminates the error.
I don't know if it occurs only with certain type of file or in https only, I could try and let you know.
Do you think is this a bug related to other ones or a new one?
Comment 106 Eric Covener 2017-09-29 12:18:28 UTC
Any further work on this requires a new bug on contemporary software
levels.