Hi all, I have configured httpd-2.2.0 (release) with mod_proxy and virtual hosting and when i stress it, even low (10 simultaneous users with Mercury Load Runner) i get some errors messages and status code 502 on the client browser. The problem occurs only when i do some load tests, when i request from single client i have no problem... My error.log : [Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: error reading status line from remote server (null), referer: http://portail.polen.xxxxx.fr/ [Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: Error reading from remote server returned by /xxxx.gif, referer: http://portail.polen.xxxxx.fr/ [Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: error reading status line from remote server (null), referer: http://portail.polen.xxxxx.fr/polen/ [Sat Dec 03 13:16:25 2005] [error] [client 10.133.2.63] proxy: Error reading from remote server returned by /polen/styles/polen.css, referer: http://portail.polen.xxxxx.fr/polen/ [Sat Dec 03 13:16:26 2005] [error] [client 10.133.2.63] proxy: error reading status line from remote server (null), referer: http://portail.polen.xxxx.fr/polen/ [Sat Dec 03 13:16:26 2005] [error] [client 10.133.2.63] proxy: Error reading from remote server returned by /polen/images/droite2.gif, referer: http://portail.polen.sofinco.fr/polen/ [Sat Dec 03 13:16:28 2005] [error] [client 10.133.2.63] (9)Bad file descriptor: proxy: error reading response My config : <VirtualHost *> ServerName portail.polen.xxxxxxx.fr CustomLog /var/log/apache21/access-portail.log combined <Location /> Order deny,allow Allow from all </Location> RewriteEngine On RewriteRule "(.*)" "http://uxdev14.xxxxx.fr:8080%{REQUEST_URI}" [P,L] </VirtualHost> I have tried with mod_proxy_balancer, mod_rewrite [P] (which call mod_proxy) and of course mod_proxy... The problem seems coming from mod_proxy.
Merge from Christophe's mail to the dev list (http://mail-archives.apache.org/mod_mbox/httpd-dev/200511.mbox/ajax/%3c4388AC09.2000907@nbux.com%3e): Hi all, i have just tried with the last 2.1.10 and i have the same problem : 1. in error logs : [Sat Nov 26 19:29:55 2005] [error] [client 10.133.2.63] proxy: error reading status line from remote server (null), referer: http://portail.polen.sofinco.fr/polen/ [Sat Nov 26 19:29:55 2005] [error] [client 10.133.2.63] proxy: Error reading from remote server returned by /polen/images/passe.gif, referer: http://portail.polen.sofinco.fr/polen/ [Sat Nov 26 19:29:56 2005] [notice] child pid 25976 exit signal Segmentation fault (11), possible coredump in /tmp 2. i used gdb to get a trace : gdb /opt/apache21/bin/httpd /tmp/core.25976 GNU gdb 6.3-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Core was generated by `/opt/apache21/bin/httpd -k start'. Program terminated with signal 11, Segmentation fault. warning: current_sos: Can't read pathname for load map: Input/output error Reading symbols from /lib/tls/libm.so.6...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /opt/apache21/lib/libaprutil-1.so.0...done. Loaded symbols for /opt/apache21/lib/libaprutil-1.so.0 Reading symbols from /usr/lib/libdb-4.3.so...done. Loaded symbols for /usr/lib/libdb-4.3.so Reading symbols from /usr/lib/libexpat.so.1...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /opt/apache21/lib/libapr-1.so.0...done. Loaded symbols for /opt/apache21/lib/libapr-1.so.0 Reading symbols from /lib/tls/librt.so.1...done. Loaded symbols for /lib/tls/librt.so.1 Reading symbols from /lib/tls/libcrypt.so.1...done. Loaded symbols for /lib/tls/libcrypt.so.1 Reading symbols from /lib/tls/libpthread.so.0...done. Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /lib/tls/libdl.so.2...done. Loaded symbols for /lib/tls/libdl.so.2 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/tls/libnss_compat.so.2...done. Loaded symbols for /lib/tls/libnss_compat.so.2 Reading symbols from /lib/tls/libnsl.so.1...done. Loaded symbols for /lib/tls/libnsl.so.1 Reading symbols from /lib/tls/libnss_nis.so.2...done. Loaded symbols for /lib/tls/libnss_nis.so.2 Reading symbols from /lib/tls/libnss_files.so.2...done. Loaded symbols for /lib/tls/libnss_files.so.2 Reading symbols from /opt/apache21/modules/mod_authz_host.so...done. Loaded symbols for /opt/apache21/modules/mod_authz_host.so Reading symbols from /opt/apache21/modules/mod_dumpio.so...done. Loaded symbols for /opt/apache21/modules/mod_dumpio.so Reading symbols from /opt/apache21/modules/mod_include.so...done. Loaded symbols for /opt/apache21/modules/mod_include.so Reading symbols from /opt/apache21/modules/mod_log_config.so...done. Loaded symbols for /opt/apache21/modules/mod_log_config.so Reading symbols from /opt/apache21/modules/mod_logio.so...done. Loaded symbols for /opt/apache21/modules/mod_logio.so Reading symbols from /opt/apache21/modules/mod_env.so...done. Loaded symbols for /opt/apache21/modules/mod_env.so Reading symbols from /opt/apache21/modules/mod_expires.so...done. Loaded symbols for /opt/apache21/modules/mod_expires.so Reading symbols from /opt/apache21/modules/mod_headers.so...done. Loaded symbols for /opt/apache21/modules/mod_headers.so Reading symbols from /opt/apache21/modules/mod_ident.so...done. Loaded symbols for /opt/apache21/modules/mod_ident.so Reading symbols from /opt/apache21/modules/mod_usertrack.so...done. Loaded symbols for /opt/apache21/modules/mod_usertrack.so Reading symbols from /opt/apache21/modules/mod_unique_id.so...done. Loaded symbols for /opt/apache21/modules/mod_unique_id.so Reading symbols from /opt/apache21/modules/mod_setenvif.so...done. Loaded symbols for /opt/apache21/modules/mod_setenvif.so Reading symbols from /opt/apache21/modules/mod_version.so...done. Loaded symbols for /opt/apache21/modules/mod_version.so Reading symbols from /opt/apache21/modules/mod_proxy.so...done. Loaded symbols for /opt/apache21/modules/mod_proxy.so Reading symbols from /opt/apache21/modules/mod_proxy_connect.so...done. Loaded symbols for /opt/apache21/modules/mod_proxy_connect.so Reading symbols from /opt/apache21/modules/mod_proxy_http.so...done. Loaded symbols for /opt/apache21/modules/mod_proxy_http.so Reading symbols from /opt/apache21/modules/mod_proxy_balancer.so...done. Loaded symbols for /opt/apache21/modules/mod_proxy_balancer.so Reading symbols from /opt/apache21/modules/mod_mime.so...done. Loaded symbols for /opt/apache21/modules/mod_mime.so Reading symbols from /opt/apache21/modules/mod_status.so...done. Loaded symbols for /opt/apache21/modules/mod_status.so Reading symbols from /opt/apache21/modules/mod_autoindex.so...done. Loaded symbols for /opt/apache21/modules/mod_autoindex.so Reading symbols from /opt/apache21/modules/mod_asis.so...done. Loaded symbols for /opt/apache21/modules/mod_asis.so Reading symbols from /opt/apache21/modules/mod_info.so...done. Loaded symbols for /opt/apache21/modules/mod_info.so Reading symbols from /opt/apache21/modules/mod_vhost_alias.so...done. Loaded symbols for /opt/apache21/modules/mod_vhost_alias.so Reading symbols from /opt/apache21/modules/mod_dir.so...done. Loaded symbols for /opt/apache21/modules/mod_dir.so Reading symbols from /opt/apache21/modules/mod_alias.so...done. Loaded symbols for /opt/apache21/modules/mod_alias.so Reading symbols from /opt/apache21/modules/mod_rewrite.so...done. Loaded symbols for /opt/apache21/modules/mod_rewrite.so Reading symbols from /opt/apache21/modules/mod_cache.so...done. Loaded symbols for /opt/apache21/modules/mod_cache.so Reading symbols from /opt/apache21/modules/mod_mem_cache.so...done. Loaded symbols for /opt/apache21/modules/mod_mem_cache.so Reading symbols from /opt/apache21/modules/mod_ssl.so...done. Loaded symbols for /opt/apache21/modules/mod_ssl.so Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done. Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/tls/libnss_dns.so.2...done. Loaded symbols for /lib/tls/libnss_dns.so.2 Reading symbols from /lib/tls/libresolv.so.2...done. Loaded symbols for /lib/tls/libresolv.so.2 #0 apr_brigade_cleanup (data=0x8398980) at buckets/apr_brigade.c:44 44 apr_bucket_delete(e); (gdb) bt full #0 apr_brigade_cleanup (data=0x8398980) at buckets/apr_brigade.c:44 No locals. #1 0xb7f24e7d in apr_brigade_destroy (b=0x8398980) at buckets/apr_brigade.c:53 No locals. #2 0x080726f4 in ap_core_output_filter (f=0xaa083538, b=0x8398980) at core_filters.c:876 hdtr = {headers = 0xb41c3ff8, numheaders = 134740504, trailers = 0x8392910, numtrailers = 0} bytes_sent = 0 flags = 0 nbytes = 1429 nvec_trailers = 0 vec = {{iov_base = 0x838d208, iov_len = 206}, {iov_base = 0x8394a09, iov_len = 1223}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0xaa083aa0, iov_len = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217623}, {iov_base = 0x0, iov_len = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217643}, {iov_base = 0x8392910, iov_len = 0}, {iov_base = 0xb41c3f30, iov_len = 0}, {iov_base = 0x838e938, iov_len = 0}, {iov_base = 0xb41c3f68, iov_len = 134646597}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x8393960, iov_len = 137967992}, {iov_base = 0xb41c3ff8, iov_len = 134740481}, {iov_base = 0x8393960, iov_len = 137967992}} flen = Variable "flen" is not available. (gdb) thread apply all bt full Thread 27 (process 25976): #0 0x00000293 in ?? () No symbol table info available. #1 0x00000000 in ?? () No symbol table info available. #2 0x00000000 in ?? () No symbol table info available. #3 0x00000000 in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. #5 0x00000000 in ?? () No symbol table info available. #6 0x00000000 in ?? () No symbol table info available. #7 0x00000000 in ?? () No symbol table info available. #8 0x00000000 in ?? () No symbol table info available. #9 0x00000000 in ?? () ---Type <return> to continue, or q <return> to quit---
(In reply to comment #1) [..cut..] > 1. in error logs : > [Sat Nov 26 19:29:55 2005] [error] [client 10.133.2.63] proxy: error reading > status line from remote server (null), referer: > http://portail.polen.sofinco.fr/polen/ Strange that the remote server is null. [..cut..] > (gdb) bt full > #0 apr_brigade_cleanup (data=0x8398980) at buckets/apr_brigade.c:44 > No locals. > #1 0xb7f24e7d in apr_brigade_destroy (b=0x8398980) at buckets/apr_brigade.c:53 > No locals. > #2 0x080726f4 in ap_core_output_filter (f=0xaa083538, b=0x8398980) at > core_filters.c:876 > hdtr = {headers = 0xb41c3ff8, numheaders = 134740504, trailers = > 0x8392910, numtrailers = 0} > bytes_sent = 0 > flags = 0 > nbytes = 1429 > nvec_trailers = 0 > vec = {{iov_base = 0x838d208, iov_len = 206}, {iov_base = 0x8394a09, > iov_len = 1223}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0xaa083aa0, iov_len > = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217623}, {iov_base = 0x0, > iov_len = 2852666000}, {iov_base = 0xb41c3f48, iov_len = 3083217643}, > {iov_base = 0x8392910, iov_len = 0}, {iov_base = 0xb41c3f30, iov_len = 0}, > {iov_base = 0x838e938, iov_len = 0}, {iov_base = 0xb41c3f68, iov_len = > 134646597}, {iov_base = 0x0, > iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x8393960, iov_len > = 137967992}, {iov_base = 0xb41c3ff8, iov_len = 134740481}, {iov_base = > 0x8393960, iov_len = 137967992}} > flen = Variable "flen" is not available. > (gdb) thread apply all bt full Possibly a corrupted brigade. I am a little bit astonished that the stacktrace isn't longer. So could you please do the following: 1. Add the contents of .gdbinit from the root directory of the 2.1.10 tar ball to your .gdbinit (see http://httpd.apache.org/dev/debugging.html#gdb) 2. Issue the following commands: gdb /opt/apache21/bin/httpd /tmp/core.25976 bt frame 2 dump_brigade b dump_filters f [..cut..]
Please set your LogLevel also to debug and let us know the contents of the error log once the problem happens again.
Hum, very very strange, when i have LogLevel debug, problem NEVER occur !, but i have any other LogLevel (info, notice, warn, crit, ...) the problem occur again (but no more detailed log information than info - which i posted). I have retry 10 times, with much of stress (500 simultaneous users) but no luck ... no more error message like my bug report post ! Sorry
Hi again, Here is more informations about the problem, sometimes i even get a segmentation fault (11) : I have done some differents stress tests : 1. with a debug LogLevel : no problem (see my last post), i have the problem only with any other level (warn, notice, info, crit, ...) 2. with others backends servers (simple .html on an apache 2.0.54 instead of a .jsp on a Tomcat-5.5.12) : no problem. Of course, no error on Tomcat backend server, and when i bypass reverse proxy, or when i use apache 2.0.55 (instead of 2.2.0 with mod_proxy), i don't get any error. 3. sometimes i get a segmentation fault and here a trace (i have used .gdbinit of httpd-2.2.0 directory) : ------------------ gdb /opt/apache22/bin/httpd /tmp/core.7637 GNU gdb 6.3-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Core was generated by `/opt/apache22/bin/httpd -k start'. Program terminated with signal 11, Segmentation fault. warning: current_sos: Can't read pathname for load map: Input/output error Reading symbols from /lib/tls/libm.so.6...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /opt/apache22/lib/libaprutil-1.so.0...done. Loaded symbols for /opt/apache22/lib/libaprutil-1.so.0 Reading symbols from /usr/lib/libdb-4.3.so...done. Loaded symbols for /usr/lib/libdb-4.3.so Reading symbols from /usr/lib/libexpat.so.1...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /opt/apache22/lib/libapr-1.so.0...done. Loaded symbols for /opt/apache22/lib/libapr-1.so.0 Reading symbols from /lib/tls/librt.so.1...done. Loaded symbols for /lib/tls/librt.so.1 Reading symbols from /lib/tls/libcrypt.so.1...done. Loaded symbols for /lib/tls/libcrypt.so.1 Reading symbols from /lib/tls/libpthread.so.0...done. Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /lib/tls/libdl.so.2...done. Loaded symbols for /lib/tls/libdl.so.2 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/tls/libnss_compat.so.2...done. Loaded symbols for /lib/tls/libnss_compat.so.2 Reading symbols from /lib/tls/libnsl.so.1...done. Loaded symbols for /lib/tls/libnsl.so.1 Reading symbols from /lib/tls/libnss_nis.so.2...done. Loaded symbols for /lib/tls/libnss_nis.so.2 Reading symbols from /lib/tls/libnss_files.so.2...done. Loaded symbols for /lib/tls/libnss_files.so.2 Reading symbols from /opt/apache22/modules/mod_authz_host.so...done. Loaded symbols for /opt/apache22/modules/mod_authz_host.so Reading symbols from /opt/apache22/modules/mod_dumpio.so...done. Loaded symbols for /opt/apache22/modules/mod_dumpio.so Reading symbols from /opt/apache22/modules/mod_include.so...done. Loaded symbols for /opt/apache22/modules/mod_include.so Reading symbols from /opt/apache22/modules/mod_log_config.so...done. Loaded symbols for /opt/apache22/modules/mod_log_config.so Reading symbols from /opt/apache22/modules/mod_logio.so...done. Loaded symbols for /opt/apache22/modules/mod_logio.so Reading symbols from /opt/apache22/modules/mod_env.so...done. Loaded symbols for /opt/apache22/modules/mod_env.so Reading symbols from /opt/apache22/modules/mod_expires.so...done. Loaded symbols for /opt/apache22/modules/mod_expires.so Reading symbols from /opt/apache22/modules/mod_headers.so...done. Loaded symbols for /opt/apache22/modules/mod_headers.so Reading symbols from /opt/apache22/modules/mod_ident.so...done. Loaded symbols for /opt/apache22/modules/mod_ident.so Reading symbols from /opt/apache22/modules/mod_usertrack.so...done. Loaded symbols for /opt/apache22/modules/mod_usertrack.so Reading symbols from /opt/apache22/modules/mod_unique_id.so...done. Loaded symbols for /opt/apache22/modules/mod_unique_id.so Reading symbols from /opt/apache22/modules/mod_setenvif.so...done. Loaded symbols for /opt/apache22/modules/mod_setenvif.so Reading symbols from /opt/apache22/modules/mod_version.so...done. Loaded symbols for /opt/apache22/modules/mod_version.so Reading symbols from /opt/apache22/modules/mod_proxy.so...done. Loaded symbols for /opt/apache22/modules/mod_proxy.so Reading symbols from /opt/apache22/modules/mod_proxy_connect.so...done. Loaded symbols for /opt/apache22/modules/mod_proxy_connect.so Reading symbols from /opt/apache22/modules/mod_proxy_http.so...done. Loaded symbols for /opt/apache22/modules/mod_proxy_http.so Reading symbols from /opt/apache22/modules/mod_proxy_balancer.so...done. Loaded symbols for /opt/apache22/modules/mod_proxy_balancer.so Reading symbols from /opt/apache22/modules/mod_mime.so...done. Loaded symbols for /opt/apache22/modules/mod_mime.so Reading symbols from /opt/apache22/modules/mod_status.so...done. Loaded symbols for /opt/apache22/modules/mod_status.so Reading symbols from /opt/apache22/modules/mod_autoindex.so...done. Loaded symbols for /opt/apache22/modules/mod_autoindex.so Reading symbols from /opt/apache22/modules/mod_asis.so...done. Loaded symbols for /opt/apache22/modules/mod_asis.so Reading symbols from /opt/apache22/modules/mod_info.so...done. Loaded symbols for /opt/apache22/modules/mod_info.so Reading symbols from /opt/apache22/modules/mod_vhost_alias.so...done. Loaded symbols for /opt/apache22/modules/mod_vhost_alias.so Reading symbols from /opt/apache22/modules/mod_dir.so...done. Loaded symbols for /opt/apache22/modules/mod_dir.so Reading symbols from /opt/apache22/modules/mod_alias.so...done. Loaded symbols for /opt/apache22/modules/mod_alias.so Reading symbols from /opt/apache22/modules/mod_rewrite.so...done. Loaded symbols for /opt/apache22/modules/mod_rewrite.so Reading symbols from /opt/apache22/modules/mod_cache.so...done. Loaded symbols for /opt/apache22/modules/mod_cache.so Reading symbols from /opt/apache22/modules/mod_mem_cache.so...done. Loaded symbols for /opt/apache22/modules/mod_mem_cache.so Reading symbols from /opt/apache22/modules/mod_ssl.so...done. Loaded symbols for /opt/apache22/modules/mod_ssl.so Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done. Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/tls/libnss_dns.so.2...done. Loaded symbols for /lib/tls/libnss_dns.so.2 Reading symbols from /lib/tls/libresolv.so.2...done. Loaded symbols for /lib/tls/libresolv.so.2 #0 0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0 (gdb) bt #0 0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0 #1 0x08085ea7 in ap_mpm_pod_check (pod=0xfffffe00) at pod.c:54 #2 0x08083f69 in child_main (child_num_arg=Variable "child_num_arg" is not available. ) at worker.c:1233 #3 0x080840d0 in make_child (s=0x80a7d88, slot=0) at worker.c:1316 #4 0x0808417b in startup_children (number_to_start=2) at worker.c:1350 #5 0x08084ce1 in ap_mpm_run (_pconf=0x80a60a8, plog=0x80d4160, s=0x80a7d88) at worker.c:1700 #6 0x08061bef in main (argc=3, argv=0xbfe76ca4) at main.c:712 (gdb) frame 2 #2 0x08083f69 in child_main (child_num_arg=Variable "child_num_arg" is not available. ) at worker.c:1233 1233 rv = ap_mpm_pod_check(pod); (gdb) dump_brigade b No symbol "b" in current context. (gdb) dump_filters f No symbol "f" in current context. --------------------------------
Please do the following things: 1. Apply the steps (dump_brigade and co) I had described to the old core dump (core.25976). 2. Apply the command the following commands to your latest core dump (/tmp/core.7637): info threads thread apply all bt
I have just tried with prefork mpm method instead of worker, and no more problem... I appears that it is worker mpm with is the origin of the problem... Should i open a new bug report ?
1. sorry i have deleted it ... 2. (gdb) info threads 27 process 7640 0x00000212 in ?? () 26 process 7641 0x00000212 in ?? () 25 process 7642 0x00000212 in ?? () 24 process 7643 0x00000212 in ?? () 23 process 7644 0x00000212 in ?? () 22 process 7645 0x00000212 in ?? () 21 process 7646 0x00000246 in ?? () 20 process 7647 0x00000246 in ?? () 19 process 7648 0x00000212 in ?? () 18 process 7649 0x00000212 in ?? () 17 process 7650 0x00000212 in ?? () 16 process 7651 0x00000212 in ?? () 15 process 7652 0x00000212 in ?? () 14 process 7653 0x00000212 in ?? () 13 process 7654 0x00000212 in ?? () 12 process 7655 0x00000212 in ?? () 11 process 7656 0x00000212 in ?? () 10 process 7657 0x00000246 in ?? () 9 process 7659 0x00000212 in ?? () 8 process 7660 0x00000246 in ?? () 7 process 7661 0x00010292 in ?? () 6 process 7662 0x00000212 in ?? () 5 process 7663 0x00000212 in ?? () 4 process 7664 0x00000212 in ?? () 3 process 7665 0x00000246 in ?? () 2 process 7666 0x00000293 in ?? () * 1 process 7637 0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0 (gdb) thread apply all bt Thread 27 (process 7640): #0 0x00000212 in ?? () Cannot access memory at address 0x0 #0 0xb7dc43f1 in __read_nocancel () from /lib/tls/libpthread.so.0 see my last post, when i use mpm prefork, no more problem, it appears that it come from worker... should i open a new bug report ?
No, you do not need to create a new bug report. We keep this one open.
*** Bug 37876 has been marked as a duplicate of this bug. ***
OS -> All since I see this on WinXP (even the crash).
Apologies for the bug spam, but I thought I should note that this isn't happening to me when load testing. This is happening in real-world usage. The crashes are coming about every two hours or so. Apache 2.2.0's proxy is unusable in a production environment.
Christophe said that he cannot reproduce the error with LogLevel debug. Are you able to reproduce the error with LogLevel debug? If yes, it would be very helpful if you could attach the error log here.
[Mon Dec 12 13:49:44 2005] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //shserver2.ssl.berkeley.edu/ [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1382): [client 127.0.0.1] proxy: *: found forward proxy worker for http://shserver2.ssl.berkeley.edu/ [Mon Dec 12 13:49:44 2005] [debug] mod_proxy.c(736): Running scheme http handler (attempt 0) [Mon Dec 12 13:49:44 2005] [debug] mod_proxy_http.c(1661): proxy: HTTP: serving URL http://shserver2.ssl.berkeley.edu/ [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1754): proxy: HTTP: has acquired connection for (*) [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1811): proxy: connecting http://shserver2.ssl.berkeley.edu/ to shserver2.ssl.berkeley.edu:80 [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1911): proxy: connected / to shserver2.ssl.berkeley.edu:80 [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(2005): proxy: HTTP: fam 2 socket created to connect to * [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(2101): proxy: HTTP: connection complete to 66.28.250.122:80 (shserver2.ssl.berkeley.edu) [Mon Dec 12 13:49:44 2005] [debug] proxy_util.c(1769): proxy: (null): has released connection for (*) [Mon Dec 12 13:49:44 2005] [error] [client 127.0.0.1] proxy: error reading status line from remote server (null) [Mon Dec 12 13:49:44 2005] [error] [client 127.0.0.1] proxy: Error reading from remote server returned by http://shserver2.ssl.berkeley.edu/
httpd 2.2.2 contains many proxy fixes. Could you please check if the problem is still there with 2.2.2?
We have tried(In reply to comment #15) > httpd 2.2.2 contains many proxy fixes. Could you please check if the problem is > still there with 2.2.2? We upgraded 6 of our proxy servers to 2.2.2 and are still experiencing problems. Is anyone else able to reproduce the problem in 2.2.2??? Thanks.
I to have seen this with 2.2.2 and also the latest cvs. Everything works ok with prefork mpm. The problem also only accours for us when we use an IIS backend device, with an apache its ok. From a tcpdump the only diff I can see between the two sessions is that the IIS closes the connection when the keep-alive time out is reached with a TCP RST packet, while the apache does a Fin/Ack handshake. When IIS sends this RST packet the next request to the mod_proxy device thats hits the worker with the RST connection returns the 502 error msg. Our fix (read hack) was to set the timeout to 24hours on all the backend devices.
hi all, i have just tried with 2.2.2 and apache/mod_proxy_ajp+balancer as backend (mod_proxy_balancer -> mod_proxy_balancer/ajp -> tomcat ajp) with the same load test in my first post, and i have only status line error message when i get a timeout from backend which is normal in my case... i used apache 2.2.2 with worker mpm... we are trying another load test and production switch if it ok thursday night... i return you friday, i hope it will be ok...
does anyone encounter segmentation fault with mpm worker ?
We are experiencing the same problem using Apache 2.2 on Windows 2000 Server. The backend is a combination of Tomcat / JBoss and IIS. Yet the problem only seems to affect IIS. C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe -l Compiled in modules: core.c mod_win32.c mpm_winnt.c http_core.c mod_so.c C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe -v Server version: Apache/2.2.2 Server built: Apr 29 2006 18:32:31 This is an extract from the log file: [Tue May 16 15:55:50 2006] [notice] Parent: Received restart signal -- Restarting the server. [Tue May 16 15:55:50 2006] [notice] Apache/2.2.2 (Win32) configured -- resuming normal operations [Tue May 16 15:55:50 2006] [notice] Server built: Apr 29 2006 18:32:31 [Tue May 16 15:55:50 2006] [notice] Child 2624: Exit event signaled. Child process is ending. [Tue May 16 15:55:50 2006] [notice] Parent: Created child process 1864 [Tue May 16 15:55:50 2006] [notice] Child 1864: Child process is running [Tue May 16 15:55:51 2006] [notice] Child 2624: Released the start mutex [Tue May 16 15:55:51 2006] [notice] Child 1864: Acquired the start mutex. [Tue May 16 15:55:51 2006] [notice] Child 1864: Starting 250 worker threads. [Tue May 16 15:55:51 2006] [notice] Child 1864: Starting thread to listen on port 8080. [Tue May 16 15:55:52 2006] [notice] Child 2624: Waiting for 250 worker threads to exit. [Tue May 16 15:55:52 2006] [notice] Child 2624: All worker threads have exited. [Tue May 16 15:55:52 2006] [notice] Child 2624: Child process is exiting [Tue May 16 16:54:10 2006] [error] [client 81.187.164.45] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 16:54:10 2006] [error] [client 81.187.164.45] proxy: Error reading from remote server returned by /support/ [Tue May 16 18:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 18:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Tue May 16 19:48:42 2006] [error] [client 207.46.98.44] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 19:48:42 2006] [error] [client 207.46.98.44] proxy: Error reading from remote server returned by /robots.txt [Tue May 16 19:59:10 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 19:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Tue May 16 20:56:43 2006] [error] [client 80.176.89.29] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 20:56:43 2006] [error] [client 80.176.89.29] proxy: Error reading from remote server returned by /support/helpdesk/viewrequest.asp [Tue May 16 20:58:22 2006] [error] [client 207.46.98.44] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 20:58:22 2006] [error] [client 207.46.98.44] proxy: Error reading from remote server returned by /default.asp [Tue May 16 21:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 21:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Tue May 16 22:59:10 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 22:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Tue May 16 23:59:10 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Tue May 16 23:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 00:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 00:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 01:33:59 2006] [error] [client 207.46.98.44] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 01:33:59 2006] [error] [client 207.46.98.44] proxy: Error reading from remote server returned by /robots.txt [Wed May 17 01:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 01:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 02:11:08 2006] [error] [client 68.142.212.239] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 02:11:08 2006] [error] [client 68.142.212.239] proxy: Error reading from remote server returned by /robots.txt [Wed May 17 02:59:08 2006] [error] [client 68.142.212.239] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 02:59:08 2006] [error] [client 68.142.212.239] proxy: Error reading from remote server returned by /img/front/fnt_utilities.jpg [Wed May 17 02:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 02:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 03:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 03:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 04:48:46 2006] [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed. [Wed May 17 04:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 04:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 05:59:10 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 05:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 06:59:10 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 06:59:10 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 07:59:11 2006] [error] [client 81.29.74.130] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 07:59:11 2006] [error] [client 81.29.74.130] proxy: Error reading from remote server returned by /support/huns/post.asp [Wed May 17 08:29:25 2006] [error] [client 81.187.164.45] proxy: error reading status line from remote server 192.168.99.88 [Wed May 17 08:29:25 2006] [error] [client 81.187.164.45] proxy: Error reading from remote server returned by /support/helpdesk/viewrequest.asp [Wed May 17 09:01:38 2006] [notice] Parent: Received restart signal -- Restarting the server. [Wed May 17 09:01:38 2006] [notice] Apache/2.2.2 (Win32) configured -- resuming normal operations
We installed 2.2.2 and turned debugging on but it does not appear to give us any further information. Here is a portion of the error log that contains the errors: [Fri May 19 04:02:28 2006] [error] [client 69.60.122.139] proxy: error reading status line from remote server 192.198.54.130 [Fri May 19 04:02:28 2006] [error] [client 69.60.122.139] proxy: Error reading from remote server returned by /community/ruralmeded/model.htm [Fri May 19 05:02:27 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/calm.htm [Fri May 19 05:02:27 2006] [debug] proxy_util.c(1378): [client 72.30.133.212] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/calm.htm [Fri May 19 05:02:27 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/calm.htm [Fri May 19 05:02:27 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/calm.htm to 192.198.54.130:80 [Fri May 19 05:02:27 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/calm.htm to 192.198.54.130:80 [Fri May 19 05:23:12 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/security.htm [Fri May 19 05:23:12 2006] [debug] proxy_util.c(1378): [client 72.30.101.219] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/security.htm [Fri May 19 05:23:12 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/security.htm [Fri May 19 05:23:12 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/security.htm to 192.198.54.130:80 [Fri May 19 05:23:12 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/security.htm to 192.198.54.130:80 [Fri May 19 05:59:07 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/ [Fri May 19 05:59:07 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/, referer: http://www.ianr.unl.edu/ [Fri May 19 05:59:07 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/ [Fri May 19 05:59:07 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/ to 192.198.54.130:80 [Fri May 19 05:59:07 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/ to 192.198.54.130:80 [Fri May 19 06:06:19 2006] [debug] proxy_util.c(1378): [client 65.214.39.180] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/5/unmc.css, referer: http://www.unmc.edu/bioterrorism/ommrslist.htm [Fri May 19 06:06:19 2006] [debug] proxy_util.c(1378): [client 65.214.39.180] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/template04/footer.js, referer: http://www.unmc.edu/bioterrorism/ommrslist.htm [Fri May 19 06:37:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/nphl.htm [Fri May 19 06:37:19 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/nphl.htm [Fri May 19 06:37:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/nphl.htm [Fri May 19 06:37:19 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/nphl.htm to 192.198.54.130:80 [Fri May 19 06:37:19 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/nphl.htm to 192.198.54.130:80 [Fri May 19 06:47:07 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/rural/mederrors/faq.htm [Fri May 19 06:47:07 2006] [debug] proxy_util.c(1378): [client 68.142.249.93] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/mederrors/faq.htm [Fri May 19 06:47:07 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/rural/mederrors/faq.htm [Fri May 19 06:47:07 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/rural/mederrors/faq.htm to 192.198.54.130:80 [Fri May 19 06:47:07 2006] [debug] proxy_util.c(1951): proxy: connected /rural/mederrors/faq.htm to 192.198.54.130:80 [Fri May 19 06:48:37 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/rural/mederrors/ [Fri May 19 06:48:37 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/mederrors/ [Fri May 19 06:48:37 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/rural/mederrors/ [Fri May 19 06:48:37 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/rural/mederrors/ to 192.198.54.130:80 [Fri May 19 06:48:37 2006] [debug] proxy_util.c(1951): proxy: connected /rural/mederrors/ to 192.198.54.130:80 [Fri May 19 06:49:29 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/nphl.htm [Fri May 19 06:49:29 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/nphl.htm, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 06:49:29 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/nphl.htm [Fri May 19 06:49:29 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/nphl.htm to 192.198.54.130:80 [Fri May 19 06:49:29 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/nphl.htm to 192.198.54.130:80 [Fri May 19 06:49:32 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/sambol.htm [Fri May 19 06:49:32 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/sambol.htm, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 06:49:32 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/sambol.htm [Fri May 19 06:49:32 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/sambol.htm to 192.198.54.130:80 [Fri May 19 06:49:32 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/sambol.htm to 192.198.54.130:80 [Fri May 19 06:49:42 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/inthenews.htm [Fri May 19 06:49:42 2006] [debug] proxy_util.c(1378): [client 24.96.199.46] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/inthenews.htm, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 06:49:42 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/inthenews.htm [Fri May 19 06:49:42 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/inthenews.htm to 192.198.54.130:80 [Fri May 19 06:49:42 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/inthenews.htm to 192.198.54.130:80 [Fri May 19 06:52:30 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/sambol.htm [Fri May 19 06:52:30 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/sambol.htm [Fri May 19 06:52:30 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/sambol.htm [Fri May 19 06:52:30 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/sambol.htm to 192.198.54.130:80 [Fri May 19 06:52:30 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/sambol.htm to 192.198.54.130:80 [Fri May 19 06:52:33 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/newsarchive.htm [Fri May 19 06:52:33 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/newsarchive.htm [Fri May 19 06:52:33 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/newsarchive.htm [Fri May 19 06:52:33 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/newsarchive.htm to 192.198.54.130:80 [Fri May 19 06:52:33 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/newsarchive.htm to 192.198.54.130:80 [Fri May 19 06:59:06 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/rural/mederrors/partners.htm [Fri May 19 06:59:06 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/mederrors/partners.htm [Fri May 19 06:59:06 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/rural/mederrors/partners.htm [Fri May 19 06:59:06 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/rural/mederrors/partners.htm to 192.198.54.130:80 [Fri May 19 06:59:06 2006] [debug] proxy_util.c(1951): proxy: connected /rural/mederrors/partners.htm to 192.198.54.130:80 [Fri May 19 07:51:38 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_erro rs.htm [Fri May 19 07:51:38 2006] [debug] proxy_util.c(1378): [client 66.249.72.77] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err ors.htm [Fri May 19 07:51:38 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err ors.htm [Fri May 19 07:51:38 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/Community/ruralmeded/admissions/admissions_and_medical_err ors.htm to 192.198.54.130:80 [Fri May 19 07:51:38 2006] [debug] proxy_util.c(1951): proxy: connected /Community/ruralmeded/admissions/admissions_and_medical_errors.htm to 192.198.54.130:80 [Fri May 19 08:00:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/coping.htm [Fri May 19 08:00:19 2006] [debug] proxy_util.c(1378): [client 65.38.102.153] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/coping.htm [Fri May 19 08:00:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/coping.htm [Fri May 19 08:00:19 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/coping.htm to 192.198.54.130:80 [Fri May 19 08:00:19 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/coping.htm to 192.198.54.130:80 [Fri May 19 08:00:19 2006] [debug] proxy_util.c(1378): [client 65.38.102.153] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/template04/footer.js, referer: http://www.unmc.edu/bioterrorism/coping.htm [Fri May 19 08:14:21 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/calm.htm [Fri May 19 08:14:21 2006] [debug] proxy_util.c(1378): [client 65.214.44.82] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/calm.htm [Fri May 19 08:14:21 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/calm.htm [Fri May 19 08:14:21 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/calm.htm to 192.198.54.130:80 [Fri May 19 08:14:21 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/calm.htm to 192.198.54.130:80 [Fri May 19 08:20:15 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/minimed/smallpox.htm, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 08:20:15 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/system.htm [Fri May 19 08:20:15 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/system.htm, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 08:20:15 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/system.htm [Fri May 19 08:20:15 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/system.htm to 192.198.54.130:80 [Fri May 19 08:20:15 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/system.htm to 192.198.54.130:80 [Fri May 19 08:20:16 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/newsarchive.htm [Fri May 19 08:20:16 2006] [debug] proxy_util.c(1378): [client 69.86.74.229] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/newsarchive.htm, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 08:20:16 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/newsarchive.htm [Fri May 19 08:20:16 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/newsarchive.htm to 192.198.54.130:80 [Fri May 19 08:20:16 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/newsarchive.htm to 192.198.54.130:80 [Fri May 19 08:27:19 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/bioterrorism/ [Fri May 19 08:27:19 2006] [debug] proxy_util.c(1378): [client 64.34.145.194] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/bioterrorism/ [Fri May 19 08:27:19 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/bioterrorism/ [Fri May 19 08:27:19 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/bioterrorism/ to 192.198.54.130:80 [Fri May 19 08:27:19 2006] [debug] proxy_util.c(1951): proxy: connected /bioterrorism/ to 192.198.54.130:80 [Fri May 19 08:44:53 2006] [debug] proxy_util.c(1378): [client 204.248.22.161] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/template04/headerbar.js, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/rural/mederrors/ [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/mederrors/, referer: http://images.google.com/imgres? imgurl=http://www.unmc.edu/rural/mederrors/images/logo- graphics.jpg&imgrefurl=http://www.unmc.edu/rural/mederrors/&h=181&w=209&sz=11&tb nid=vKuIcgUlFtYkKM:&tbnh=86&tbnw=100&hl=en&start=19&prev=/images%3Fq% 3Dmedication%2Berrors%26svnum%3D10%26hl%3Den%26lr%3D [Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/rural/mederrors/ [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/rural/mederrors/ to 192.198.54.130:80 [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: connected /rural/mederrors/ to 192.198.54.130:80 [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/department.js, referer: http://www.unmc.edu/rural/mederrors/ [Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/rural/mederrors/images/logo-partners.jpg [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/mederrors/images/logo-partners.jpg, referer: http://www.unmc.edu/rural/mederrors/ [Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/rural/mederrors/images/logo-partners.jpg [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/rural/mederrors/images/logo-partners.jpg to 192.198.54.130:80 [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: connected /rural/mederrors/images/logo-partners.jpg to 192.198.54.130:80 [Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //192.198.54.130/rural/mederrors/images/logo-contact.jpg [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/mederrors/images/logo-contact.jpg, referer: http://www.unmc.edu/rural/mederrors/ [Fri May 19 08:50:25 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.198.54.130/rural/mederrors/images/logo-contact.jpg [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1858): proxy: connecting http://192.198.54.130/rural/mederrors/images/logo-contact.jpg to 192.198.54.130:80 [Fri May 19 08:50:25 2006] [debug] proxy_util.c(1951): proxy: connected /rural/mederrors/images/logo-contact.jpg to 192.198.54.130:80 [Fri May 19 08:50:26 2006] [debug] proxy_util.c(1378): [client 170.163.32.2] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/rural/images/getacro.gif, referer: http://www.unmc.edu/rural/mederrors/ [Fri May 19 08:54:32 2006] [error] [client 24.252.13.39] request failed: error reading the headers [Fri May 19 09:16:43 2006] [debug] proxy_util.c(1378): [client 137.197.145.122] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/5/unmc.css, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 09:16:43 2006] [debug] proxy_util.c(1378): [client 137.197.145.122] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/template04/footer.js, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 09:32:06 2006] [debug] proxy_util.c(1378): [client 138.163.0.38] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/template04/headerbar.js, referer: http://www.unmc.edu/bioterrorism/ [Fri May 19 09:32:19 2006] [debug] proxy_util.c(1378): [client 138.163.0.44] proxy: http: found worker http://192.198.54.130/ for http://192.198.54.130/5/unmc.css, referer: http://www.unmc.edu/bioterrorism/default.htm
oops... sorry, didn't mean to post twice.
Does anybody still observe segfaults with 2.2.2 or is it "only" the "error reading status line" that is left?
not me...
nope... no segfaults.
Same problem, here : Configuration : Apache 2.2.2 (with worker MPM) on Solaris 9 proxying to an IIS 6.0 back-end server. No core dump produced. As I understand, truss (see below) shows that the process uses a file (25, in the example below) handle to communicate with the back-end server but does not check whether it is still usable or not. This file handle seems to be used from the connection pool ("As of Apache 2.1, the ability to use pooled connections to a backend server is available"). Result from truss : 3313/19: 3.9623 read(25, 0xFDA7B997, 1) Err#131 ECONNRESET 3313/19: 3.9624 write(11, 0xFDA79938, 126) = 126 3313/19: [ M o n J u n 2 6 1 4 : 3 7 : 3 6 2 0 0 6 ] [ d e b u 3313/19: g ] p r o x y _ u t i l . c ( 2 1 4 1 ) : p r o x y : H T 3313/19: T P : c o n n e c t i o n c o m p l e t e t o 1 3 6 . 1 3313/19: 7 3 . 2 1 . 2 4 3 : 8 8 8 8 ( e i c i t e s t v s 0 9 )\n 3313/19: 3.9628 writev(25, 0xFDA7B780, 3) Err#32 EPIPE 3313/19: iov_base = 0x003F2278 iov_len = 1057 3313/19: P O S T / E i c i - A D M / A u t h e n t i c a t i o n / u s ... 3313/1: 3.9628 Received signal #13, SIGPIPE, in read() [ignored] 3313/19: 3.9635 write(11, 0xFDA795B8, 123) = 123 3313/19: [ M o n J u n 2 6 1 4 : 3 7 : 3 6 2 0 0 6 ] [ i n f o 3313/19: ] [ c l i e n t 1 3 6 . 1 7 3 . 2 1 . 2 4 3 ] ( 3 2 ) B r 3313/19: o k e n p i p e : c o r e _ o u t p u t _ f i l t e r : w 3313/19: r i t i n g d a t a t o t h e n e t w o r k\n 3313/19: 3.9639 read(25, 0x003D6208, 8000) = 0 3313/19: 3.9641 write(11, 0xFDA77910, 158) = 158 3313/19: [ M o n J u n 2 6 1 4 : 3 7 : 3 6 2 0 0 6 ] [ e r r o 3313/19: r ] [ c l i e n t 1 3 6 . 1 7 3 . 2 2 . 1 3 7 ] p r o x y 3313/19: : e r r o r r e a d i n g s t a t u s l i n e f r o m 3313/19: r e m o t e s e r v e r e i c i t e s t v s 0 9 , r e f 3313/19: e r e r : h t t p : / / e i c i a p p q u a / G E D A /\n 3313/19: 3.9656 write(11, 0xFDA77888, 186) = 186 3313/19: [ M o n J u n 2 6 1 4 : 3 7 : 3 6 2 0 0 6 ] [ e r r o 3313/19: r ] [ c l i e n t 1 3 6 . 1 7 3 . 2 2 . 1 3 7 ] p r o x y 3313/19: : E r r o r r e a d i n g f r o m r e m o t e s e r v 3313/19: e r r e t u r n e d b y / E i c i - A D M / A u t h e n t 3313/19: i c a t i o n / u s e r i n f o s e c . a s p , r e f e r e r 3313/19: : h t t p : / / e i c i a p p q u a / G E D A /\n 3313/19: 3.9658 write(11, 0xFDA798D0, 111) = 111 3313/19: [ M o n J u n 2 6 1 4 : 3 7 : 3 6 2 0 0 6 ] [ d e b u 3313/19: g ] p r o x y _ u t i l . c ( 1 8 1 6 ) : p r o x y : H T 3313/19: T P : h a s r e l e a s e d c o n n e c t i o n f o r 3313/19: ( e i c i t e s t v s 0 9 )\n 3313/19: 3.9661 close(25) = 0 3313/19: 3.9664 read(24, 0x003EE268, 8000) Err#11 EAGAIN 3313/19: 3.9666 writev(24, 0xFDA7BBF0, 2) = 729 3313/19: iov_base = 0x003F4280 iov_len = 188 3313/19: H T T P / 1 . 1 5 0 2 P r o x y E r r o r\r\n D a t e :
(In reply to comment #24) > Does anybody still observe segfaults with 2.2.2 or is it "only" the "error reading > status line" that is left? Same problem here: No sefaults, but error reading status line Config: Solaris 8, MPM Worker, HTTPd 2.2.2 mod_proxy to IIS 6 on Windows No error with MPM Prefork and HTTPd 2.0.54
With the following 'Murxaround' it's working now: - Added 'keeplive Off' in httpd.conf -> No effect - Set keep-alive timeout to 1 day in IIS -> No effect - Added 'SetEnv force-proxy-request-1.0 1' 'SetEnv proxy-nokeepalive 1' in httpd.conf It's working now without errors since 30 hours... Don't know if first steps were necessary, I let them in place
As workaround, I made the following modification in modules/proxy/proxy_util.c (from source files) and changed line 1972 : BEFORE : if (APR_STATUS_IS_EOF(socket_status)) AFTER : if (APR_STATUS_IS_EOF(socket_status) || APR_STATUS_IS_ECONNRESET (socket_status)) configure make make install Restart Apache No more message "proxy: error reading status line from remote server" Hope this helps...
(In reply to comment #30) > As workaround, I made the following modification in modules/proxy/proxy_util.c > (from source files) and changed line 1972 : > BEFORE : if (APR_STATUS_IS_EOF(socket_status)) > AFTER : if (APR_STATUS_IS_EOF(socket_status) || APR_STATUS_IS_ECONNRESET > (socket_status)) > Hope this helps... That works perfectly. Apache no longer reports and error, when IIS uses a RST packet to close an idle back end connection, and the request is serviced correcly. Many thanks James
That fix (Olivier BOEL, comment #30) looks right to me, and works for two of you. I'm applying it to /trunk/. Others, if you're in a position to recompile mod_proxy, please apply the patch and report back.
Apache 2.2.3 compiled with mod_proxy used as rev.proxy for Owa. Works like a charm. I no longer see the '502 Proxy error' when clients try to connect, while before this error showed 20 times per hour. A big improvement. I still have some minor error logging issues with ActiveSync but these may be config issues as well. Is this going to be adressed in an official patch? Many thanks, Danny
The error message from this bug also appears when: a large and therefore slow incoming request over mobile GPRS line takes more than about 15 secs to complete. The following workaround works well: add the following two lines to file httpd.conf: SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1
(In reply to comment #32) > That fix (Olivier BOEL, comment #30) looks right to me, and works for two of > you. I'm applying it to /trunk/. > Others, if you're in a position to recompile mod_proxy, please apply the patch > and report back. I can repro what I believe is this problem as well on a 2.2.3 server on linux doing a reverse proxy to IIS using the prefork MPM. I'm going to apply the patch locally and see if it clears up. Will report back tomorrow.
> I can repro what I believe is this problem as well on a 2.2.3 server on linux > doing a reverse proxy to IIS using the prefork MPM. I'm going to apply the > patch locally and see if it clears up. Will report back tomorrow. So far so good. :) Will keep watching it throughout the day.
(In reply to comment #36) > So far so good. :) I've seen absolutely zero instances of this problem since applying the patch. I'm going to go ahead and declare it fixed. Furthermore, the patch does appear correct to me, especially given the details given in comment #17. +1 (emeritus) for backport to the 2.2 branch.
Just to add, the SetEnv workaround also seems to work for httpd 2.0.50 which I'm still using. I did not try compiling the code fix since we need to use official releases only. However, if the SetEnv works, then the code fix likely works, in case it needs to be backported as well. Thanks for all the great info and the workaround. Hope the fix gets into the next release.
Fix unfortunately does'nt work with my configuration. httpd-2.2.3 configure option: --enable-proxy --enable-proxy-balancer --enable-proxy-http --with-mpm=worker Found expected code at modules/proxy/proxy_util.c:1972 and patch it (configure/make/make install/restart) Still have : proxy: error reading status line from remote server www.myserver.com proxy: Error reading from remote server returned by /myurl.htm I've tried to add the following 2 directives in httpd.conf: SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 Restarted apache. same errors occur. I've tried debug Loglevel, but I don't get any other messages when it comes to that error. Any ideas ?
In 2.2.4-dev and trunk
*** Bug 41150 has been marked as a duplicate of this bug. ***
*** Bug 39587 has been marked as a duplicate of this bug. ***
*** Bug 39499 has been marked as a duplicate of this bug. ***
Just like to point out that I'm seeing this all the time still. I have upgraded to httpd-2.2.4, switched to using prefork instead of worker, added the SetEnv configuration options mentioned previously, tried various backend server combinations of KeepAlive etc. Running httpd-2.2.4, prefork on Solaris 10 Sparc (T1000), caching and proxying to httpd-2.2.4 prefork on Solaris 10 AMD64. Thu Feb 08 11:37:44 2007] [error] [client 88.110.18.62] proxy: error reading status line from remote server www1.mydomain.co.uk, referer: http://www.mydomain.co.uk/forums/addpost.php [Thu Feb 08 11:37:44 2007] [error] [client 88.110.18.62] proxy: Error reading from remote server returned by /forums/postlist.php, referer: http://www.mydomain.co.uk/forums/addpost.php
Did you definitely upgrade mod_proxy? If you ran your 2.2.4 configure without *explicitly* compiling mod_proxy, then it wont have compiled it, so it'll still be using your previous mod_proxy. Just setting, for example --enable-mods-shared=all is not sufficient.
I have to say that I am still seeing these errors as well. I'm using RedHat and I built and installed a new rpm for http-2.2.4 so I'm sure that I compiled in the new mod_proxy. I verified that the source fix is there in the .c file as well. [Fri Feb 09 10:18:21 2007] [error] [client xxx] proxy: error reading status line from remote server 127.0.0.1 [Fri Feb 09 10:18:21 2007] [error] [client xxx] proxy: Error reading from remote server returned by /search/guide/query I am running a cluster of Ruby/Mongrel servers behind a proxy balancer. --Andy
I saw this error in my new environment. The cause of my error was by the ExceptionNotifier.exception_recipients = $exception_recipients line in the environment.rb which calls the smtp.rb. This was hanging due to environment restrictions. I added ActionMailer::Base.server_settings = { :address => 'somemachine', :domain => 'somedoamin.com', :port => 25, :user_address => 'someuser@somedomain.com', :user_name => 'someuser', :password => 'somepassword', :authentication=>:login } to the production.rb and the problem was resolve. Didn't happen much since we write such good code. ;)
I have this problem with apache-2.2.4 (and 2.0.59) on Windows 2003 SP1. It proxies (via a rewriterule - which btw. means I can't set the ttl etc. timings enabled for ProxyPass in v2.2 :( ) I reproduce it, via a stresstest setup. I have tried to set: SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 in the virtualhost - but it has no effect. I have also tried to disable keepalive on the local IIS backend (and keeping the SetEnv's ) but to no effect either. in LogLevel debug I get this: [Wed Apr 18 16:03:12 2007] [error] [client x.x.x.x] proxy: error reading status line from remote server www.sitename.dk [Wed Apr 18 16:03:12 2007] [error] [client x.x.x.x] proxy: Error reading from remote server returned by /apps/file.dll/forside [Wed Apr 18 16:03:12 2007] [debug] proxy_util.c(1816): proxy: HTTP: has released connection for (*) Anything I can try - I'm out of options :( p.s. I have set max ephemeral ports to 5000 and set tcptimedwaitdelay to 30 secs, to avoid running into the 10048 OS error (every so often) - in apache 2.2 this has the unfortunate side effect of disabling the site for the standard timeout seconds, since mod_proxy has this new "feature" - and I can't set the disable-time, as I'm using mod_proxy via mod_rewrite :(
I have set max ephemeral ports to 65534 ofcourse - 5000 is the default :)
(In reply to comment #48) > I have this problem with apache-2.2.4 (and 2.0.59) on Windows 2003 SP1. It > proxies (via a rewriterule - which btw. means I can't set the ttl etc. timings > enabled for ProxyPass in v2.2 :( ) > > I reproduce it, via a stresstest setup. > > I have tried to set: > SetEnv force-proxy-request-1.0 1 > > > SetEnv proxy-nokeepalive 1 > in the virtualhost - but it has no effect. I don't think that your problem is the same problem as described by many others here. It is also no surprise to me that force-proxy-request-1.0 and proxy-nokeepalive show no effect in your situation as you are using mod_rewrite to do reverse proxying to a single server (a balancer would be a different story). In this case httpd NEVER reuses the backend connections. For each request to the backend a new one will be opened. > [Wed Apr 18 16:03:12 2007] [debug] proxy_util.c(1816): proxy: HTTP: has released > connection for (*) Above (the star) proves that you are using the 'reverse worker' which never reuses backend connections. So it looks like your fresh backend connections get broken for some other reasons.
I'm seeing this pretty frequently with Apache 2.2.4 on Solaris 10 using preform MPM. The proxy is forwarding through a firewall, so I have disabled KeepAlive on both internal and external servers. The internal runs Apache 2.0.55. I cannot replicate the error myself, so testing solutions involves trying something and then waiting to see if the error still pops up. Not sure if this is an issue, but I actually have a server name in "...status line from remote server (null)".
(In reply to comment #51) > I'm seeing this pretty frequently with Apache 2.2.4 on Solaris 10 using preform > MPM. The proxy is forwarding through a firewall, so I have disabled KeepAlive on > both internal and external servers. The internal runs Apache 2.0.55. I cannot > replicate the error myself, so testing solutions involves trying something and > then waiting to see if the error still pops up. > > Not sure if this is an issue, but I actually have a server name in "...status > line from remote server (null)". Nevermind, it is the firewall inbetween that is dropping the connections. This is not an Apache bug for me. Thanks.
I also have this bug running Apache 2.2.4 on Solaris 9, worker MPM. The SetEnv settings do not make it go away for me. My setup is Apache doing reverse proxy + caching in front of a Tomcat server (no firewalls in between, running on the same machine in fact). Problem seems to happen under high load. An interesting fact is that it happens around the time the pages should expire (I control this from Tomcat by setting the "Expires" header). So maybe something is wrong with some header processing? Running the same setup but using mod_jk instead of mod_proxy works fine for me.
I believe I'm running into this problem as well. My situation is similar to post #46 except I'm on apache 2.2.3 (64bit). I noticed the problem immediately after deploying my rails app to a new server; my mongrels completely freeze after little bit of inactivity. Some information about my setup: Linux web001 2.6.16.27-0.9-smp #1 SMP Tue Feb 13 09:35:18 UTC 2007 x86_64 x86_64 x86_64 GNU/Linux <VirtualHost *:80> ServerName demo DocumentRoot '/demo/public' <Location /> Order deny,allow Allow from all </Location> <Directory '/demo/public'> Options FollowSymLinks AllowOverride None </Directory> RewriteEngine On # Check for maintenance file and redirect all requests RewriteCond %{DOCUMENT_ROOT}/system/maintenance_demo.html -f RewriteCond %{SCRIPT_FILENAME} !maintenance_demo.html RewriteRule ^.*$ /system/maintenance_demo.html [PT] # Redirect all non-static requests to cluster RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f RewriteRule ^/(.*)$ balancer://demo_balancer%{REQUEST_URI} [P,QSA,L] </VirtualHost> <Proxy balancer://demo_balancer> BalancerMember http://localhost:3528 BalancerMember http://localhost:3529 </Proxy> May 21 07:46:25 web001 httpd2-prefork[12751]: [error] [client x.x.x.x] proxy: error reading status line from remote server localhost, referer: https://x.x.x.x/ May 21 07:46:25 web001 httpd2-prefork[12751]: [error] [client x.x.x.x] proxy: Error reading from remote server returned by /, referer: https://x.x.x.x/ Using the SetEnv solution in the virtual host did not work. Paul
(In reply to comment #54) > I believe I'm running into this problem as well. My situation is similar to post > #46 except I'm on apache 2.2.3 (64bit). I noticed the problem immediately First please try with 2.2.4 as it contains a fix. Second, if things do not work with SetEnv proxy-nokeepalive 1 (regardles of 2.2.3 / 2.2.4) then this looks similar to #48 for me and it looks like your fresh backend connections gets broken for some other reasons that are not caused by the bug of this PR.
*** Bug 39629 has been marked as a duplicate of this bug. ***
I too am having this issue. Running apache 2.2.4 on linux 64-bit, reverse proxy to IIS 6 backend running on Windows 2003. Apache Errors: [error] [client x.x.x.x] proxy: error reading status line from remote server y.y.y.y, referer: https://foo.bar. [error] [client x.x.x.x] proxy: Error reading from remote server returned by /URL/blah We are using mod_proxy. Errors seem very random, and tough to pin-point. When we use AJP, problem is gone. We have very high volume with this setup. Both env's are set: SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 We have tried no keepalives on the IIS side, large keepalives, but no difference. There is a PIX firewall in between, but we are not losing packets with that. Any ideas are indeed welcome.
Hi all, I have the same problem. I've just installed Apache 2.2.4 on the frontend (reverse proxy) server, on the backend server there is Apache 2.2.3 version (Debian Etch package). I still do have these problems: [Sun Jul 01 22:21:26 2007] [error] [client xx.xx.xx.xx] proxy: error reading status line from remote server backend.xxxx.com [Sun Jul 01 22:21:26 2007] [error] [client xx.xx.xx.xx] proxy: Error reading from remote server returned by /articles/ According to Changelog of 2.2.4 this should be fixed if I'm not mistaken.
Hi all We are using apache 2.2.4 and proxying to jboss 4.0.2 server. Our users reported a problem of 2 trade failures. When we checked apache logs, we noticed a slowdown in response time (5 sec and 7 sec respectively) Seems like request was 'waiting' 'in' apache because it hit our jboss server after 5 and 7 second delay ( usual delay time is in millisecond level). On checking in error logs we saw this error -> [Tue Jun 05 14:14:16 2007] [error] [client xxx.xxx.19.41] proxy: error reading status line from remote server 10.10.xxx.xx [Tue Jun 05 14:14:16 2007] [error] [client xxx.xxx.19.41] proxy: Error reading from remote server returned by /isCxxx/Heartbeat.do [Tue Jun 05 14:14:16 2007] [error] [client xxx.xxx.19.41] File does not exist: /xxxx/opt/apache2/htdocs/maintenance.html On deeper analysis, we found that this error message comes 2-3 times an hour in our logs consistently. But we were not noticing it because it was not causing any problems. However, during the time when trades were rejected, we have 20-25 instances of this error. So it looks like due to this error response time gets increased. I am ready to provide more information if required. Tapan
I have created a very simple test for this. a Fresh Ubuntu Install with the latest Gutsy Apache2.2.4 on it I have done it as a prefork and as a worker both fail here is the error log [Thu Aug 23 19:20:09 2007] [debug] mod_proxy_http.c(1662): proxy: HTTP: serving URL http://192.168.0.110:2054/ [Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(1798): proxy: HTTP: has acquired connection for (192.168.0.110) [Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(1859): proxy: connecting http://192.168.0.110:2054/ to 192.168.0.110:2054 [Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(1955): proxy: connected / to 192.168.0.110:2054 [Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(2050): proxy: HTTP: fam 2 socket created to connect to 192.168.0.110 [Thu Aug 23 19:20:09 2007] [debug] proxy_util.c(2146): proxy: HTTP: connection complete to 192.168.0.110:2054 (192.168.0.110) [Thu Aug 23 19:22:38 2007] [error] [client 192.168.0.1] proxy: error reading status line from remote server 192.168.0.110 [Thu Aug 23 19:22:38 2007] [error] [client 192.168.0.1] proxy: Error reading from remote server returned by /ceservice.cetix [Thu Aug 23 19:22:38 2007] [debug] proxy_util.c(1816): proxy: HTTP: has released connection for (192.168.0.110) [Thu Aug 23 19:22:38 2007] [debug] mod_proxy_balancer.c(521): proxy_balancer_post_request for (balancer://demo_balancer) Here is the sites-available file... Pretty straight forward. NameVirtualHost * <Proxy balancer://demo_balancer> BalancerMember http://192.168.0.110:2054 # BalancerMember http://192.168.0.110:7054 </Proxy> <VirtualHost *> ServerAdmin webmaster@localhost SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 RewriteEngine on RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) RewriteRule .* - [F] RewriteLog "/var/log/apache2/rewrite.log" RewriteLogLevel 9 RewriteRule ^/(.*.cetix) balancer://demo_balancer [P,QSA,L] DocumentRoot /var/www/ <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory /var/www/> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all # This directive allows us to have apache2's default start page # in /apache2-default/, but still have / go to the right place #RedirectMatch ^/$ /apache2-default/ </Directory> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all </Directory> ErrorLog /var/log/apache2/error.log # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. LogLevel debug CustomLog /var/log/apache2/access.log combined ServerSignature On Alias /doc/ "/usr/share/doc/" <Directory "/usr/share/doc/"> Options Indexes MultiViews FollowSymLinks AllowOverride None Order deny,allow Deny from all Allow from 127.0.0.0/255.0.0.0 ::1/128 </Directory> </VirtualHost> ~ This is forwarding to another web service on a local network. I can talk directly to the web service just fine. I need to get this resolved any advice is great. Jeff
*** Bug 32731 has been marked as a duplicate of this bug. ***
I have compiled the latest httpd 2.2.6 with static modules i.e. ./configure --prefix=/usr/local/apache --enable-modules="asis alias cache deflate disk_cache expires file_cache mem_cache headers mime mimemagic include log_config proxy proxy_http rewrite setenvif status vhost_alias" --with- mpm=worker We have IIS 6.0 as backend webservers and a BigIP F5 for load balancing in front of them. The Apache sits in a different geographical location and connects to the IIS webservers thru a VPN connection. The virtual host config is as follows: <VirtualHost xx.xx.xx.xx> ServerName www.abc.com ServerAlias abc.com ProxyRequests Off ProxyBadHeader Ignore SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 ProxyPass / http://www.abc.com/ smax=5 max=20 ttl=120 retry=1 keepalive=Off #ProxyPass / http://www.abc.com/ ProxyPassReverse / http://www.abc.com/ </VirtualHost> [Mon Sep 10 17:36:59 2007] [error] ap_proxy_connect_backend disabling worker for (www.ABC.com) [Mon Sep 10 17:36:59 2007] [error] (110)Connection timed out: proxy: HTTP: attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed [Mon Sep 10 17:37:00 2007] [error] (110)Connection timed out: proxy: HTTP: attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed [Mon Sep 10 17:37:00 2007] [error] ap_proxy_connect_backend disabling worker for (www.ABC.com) [Mon Sep 10 17:37:00 2007] [error] (110)Connection timed out: proxy: HTTP: attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed [Mon Sep 10 17:37:01 2007] [error] (110)Connection timed out: proxy: HTTP: attempt to connect to xx.xx.xx.xx:80 (www.ABC.com) failed [Mon Sep 10 17:37:01 2007] [error] ap_proxy_connect_backend disabling worker for (www.ABC.com) [Mon Sep 10 17:38:21 2007] [error] [client 200.124.142.132] (70007)The timeout specified has expired: proxy: error reading status line from remote server www.ABC.com, referer: http://www.ABC.com/ [Mon Sep 10 17:38:21 2007] [error] [client 200.124.142.132] proxy: Error reading from remote server returned by /welcome/welcome.aspx, referer: http://www.ABC.com/ These errors are appearing randomly and I was monitoring a ping between Apache to the F5 and there were no hiccups in the VPN connection. The problem gets exaggerated when I use multiple Apache's in front and load balance between them. Please let me know if there is any fix available to the problem or If I should not be using Apache to reverse proxy a website being loadbalanced by F5. Thanks in anticipation: AS
Error reading status line can appear due to network timeout and is often observed when the 'Timeout' setting is set to low value. This is a normal situation.
If it is a normal situation, perhaps it shouldn't be logged to the error log?
It is a normal situation in terms of httpd functioning. The causes of 'error reading status line' are: (70007)The timeout specified has expired -- indicates network timeout, configured by Timeout directive (104)Connection reset by peer -- TCP RST was sent from server (70014)End of file found -- TCP FIN was sent from server All these messages appearing in error.log indicate some problem with network on server side or with server itself and do not mean errors in proxy software. I think the bug should be closed.
I think this bug should NOT be closed because the proxy server should be transparent and it is not as long this problem remains. Due to this bug, the system behaves differently when the proxy server is in between the client and the server. Direct connections to the server do not produce this error message in clients. When the workarounds proposed are in place, the error is not reproduced either. Now the proxy module produces an error message, thus moving problems on the TCP network layer to the http layer. I think it is not practical to start proxying the TCP layer messages, so instead the http proxy should react to TCP layer problems in the same way a http client would.
(In reply to comment #65) > All these messages appearing in error.log indicate some problem with network on > server side or with server itself and do not mean errors in proxy software. I > think the bug should be closed. What data are you privy to that causes you to believe that there isn't a bug causing the proxy module to believe one of those conditions is true when it, in fact, is false?
(In reply to comment #67) And what data can be previed to the belief that there is a bug? Do you think there can be a bug in apr functions that are called from ap_rgetline_core or do you think that there can be a bug in ap_rgetline_core itself? If this could be the case, then we would see the bug not only in "reading status line" but everywhere!
I'm not familiar with the intracacies of APR, but what I do know is that I experience problems with Apache acting as a proxy server that I do not experience when it is not the proxy server. I may be ignorant of the code, but to me that points to something about the proxy. I haven't seen anything reported in this bug that points to another cause other than what appears to be blind faith that there cannot be anything wrong with the proxy code.
(In reply to comment #67) > (In reply to comment #65) > > All these messages appearing in error.log indicate some problem with network on > > server side or with server itself and do not mean errors in proxy software. I > > think the bug should be closed. > > What data are you privy to that causes you to believe that there isn't a bug > causing the proxy module to believe one of those conditions is true when it, in > fact, is false? He is correct in several cases: (70007)The timeout specified has expired -- indicates network timeout. This can happen if your backend takes a long time preparing the response and this is a wanted behaviour. You can adjust the timeout either via TimeOut, ProxyTimeout or a specific worker timeout. (104)Connection reset by peer -- TCP RST was sent from server (70014)End of file found -- TCP FIN was sent from server These can happen either because of network problems or problems on the backend server or because of a race condition in the proxy code. Currently there is a race condition that the backend server can close the keepalive connection right after httpd checked the state of this TCP connection and httpd starts sending a request and waiting on a response on a connection that is in fact dead. But this should happen rarely and browsers try to resent the request in such situations provided the request was idempotent (so no POST request or GET request with parameters). All other cases (except for the race condition) also lead to error messages in the browser (maybe a different one that the one sent by the proxy, but an error message). So the race condition is the only reason why I think that it is justified to keep this PR open.
Created attachment 21243 [details] backend connection race condition fix (untested) This should fix such race condition.
(In reply to comment #71) > Created an attachment (id=21243) [edit] > backend connection race condition fix (untested) > > This should fix such race condition. I think it's missing a cleanup before the retry in the APR_ECONNRESET/APR_EOF case. But otherwise looks reasonable, if this race condition is indeed the culprit. It would be good to get some feedback from folks affected by this problem. Does the patch fix it for you?
(In reply to comment #71) > Created an attachment (id=21243) [edit] > backend connection race condition fix (untested) > > This should fix such race condition. As explained before this patch is not correct: 1. Once you sent the request body it is no longer available. 2. The RFC does not allow to resend non idempotent requests. The only fix I can think of is to send a test request over the connection first that does not trigger (a significant action) on backend side. OPTIONS * would be a candidate for this. If retrieve an 200 OK here we can be pretty sure that the race will not occur
(In reply to comment #73) > (In reply to comment #71) > > Created an attachment (id=21243) [edit] [edit] > > backend connection race condition fix (untested) > > > > This should fix such race condition. > > As explained before this patch is not correct: > > 1. Once you sent the request body it is no longer available. This can be fixed. > 2. The RFC does not allow to resend non idempotent requests. Why does this applies to our situation when the request does not reach backend server?
(In reply to comment #74) > (In reply to comment #73) > > (In reply to comment #71) > > > Created an attachment (id=21243) [edit] [edit] [edit] > > > backend connection race condition fix (untested) > > > > > > This should fix such race condition. > > > > As explained before this patch is not correct: > > > > 1. Once you sent the request body it is no longer available. > > This can be fixed. Not easily as buffering large request entities is a pain. > > > 2. The RFC does not allow to resend non idempotent requests. > > Why does this applies to our situation when the request does not reach backend > server? With your loop you don't know. Once you sent a single byte of the request body or even only the GET request with parameters you do not really now what the backend server did. Didn't it receive it or did it receive it, processed it and just in the moment as it wanted to send the reply the network broke down or the process that processed the request died.
(In reply to comment #75) > With your loop you don't know. Once you sent a single byte of the request body > or even only the GET request with parameters you do not really now what the > backend server did. Didn't it receive it or did it receive it, processed it and > just in the moment as it wanted to send the reply the network broke down or the > process that processed the request died. > If after FIN was recieved RST is following couldn't we assume that our request is sure rejected by remote server's TCP/IP stack? Besides, I suppose that any client will suffer from such Byzantine problem. Where is the difference between mod_proxy and browser behaviour addressing this situation?
(In reply to comment #63) > Error reading status line can appear due to network timeout and is often > observed when the 'Timeout' setting is set to low value. This is a normal > situation. Ummmm, yeah - thanks Aleksey! Big autodopeslap here. I've been watching this bug for a while and it never occurred to me to 'de-optimize' the global Timeout. I guess I assumed the ProxyTimeout would have superceded Timeout in a VirtualHost (if I'm reading the code correctly, the recent commit for bug 11540 does just that, but I'm still on 2.2.6). Anyway, a new Timeout in the affected VirtualHost fixed this for me (finally - yay!) I filed bug 44051 asking for a more specific error message.
*** Bug 44079 has been marked as a duplicate of this bug. ***
It also happens with us. Any thoughts on when the patch existing in Comment #71 will be commited in SVN?
(In reply to comment #79) > It also happens with us. Any thoughts on when the patch existing in Comment #71 > will be commited in SVN? > It is really wrong. Don't use it.
Any thoughts on how to fix it then?
Try to use the patch from r645813 (http://svn.apache.org/viewvc?view=rev&revision=645813). Patch: svn diff -r645812:645813 https://svn.apache.org/repos/asf/httpd/httpd/trunk
(In reply to comment #82) > Try to use the patch from r645813 > (http://svn.apache.org/viewvc?view=rev&revision=645813). Patch: > svn diff -r645812:645813 https://svn.apache.org/repos/asf/httpd/httpd/trunk > So as I understand it, there is a bug regarding a race condition in mod_proxy that is still causing this error. And the only patch in place is the one above in comment #82, which I noticed is also in trunk. However, this patch, while perhaps a good one to have in place, doesn't solve the race condition. And the race condition is the real issue here. This is a serious problem that is affecting the use of Apache 2.2.8 on many customers. The frequency of the error with some sites is high enough that it puts Apache mod_proxy in jeopardy of being unusable, especially for non-idempotent requests. It may be less than 5% of requests at most but that is a lot for a major traffic site. Are there any new developments on getting this race condition fixed? Can someone please point me to the area in the code where this race condition exists and describe the complications around it that have prevented it from being fixed thus far?
(In reply to comment #82) > Try to use the patch from r645813 > (http://svn.apache.org/viewvc?view=rev&revision=645813). Patch: > svn diff -r645812:645813 https://svn.apache.org/repos/asf/httpd/httpd/trunk > Backported to 2.2.x as r657443 (http://svn.apache.org/viewvc?rev=657443&view=rev).
Any chance of getting this fix ported to 2.0? I'm having the same/similar issue when using mod_proxy to an https target. (2.0.63/worker/linux) Using the SetEnv hack seems to resolve the issue... but this is not a desirable workaround.
(In reply to comment #85) > Any chance of getting this fix ported to 2.0? Should never say never but the probability is very low. There is low developer interest in 2.0.x these days and so expect only security fixes for 2.0.x. Upgrade to 2.2.x. It has many interesting new features in the proxy module.
*** Bug 45510 has been marked as a duplicate of this bug. ***
Created attachment 22341 [details] Patch against trunk Can you please try this patch and add setenv proxy-initial-not-pooled 1 to your virtualhost configuration. Note that it comes with a performance penalty.
(In reply to comment #87) > *** Bug 45510 has been marked as a duplicate of this bug. *** At first, I introduced r657443. The problem has been improved, but not perfect. I was watching "netstat", but sometimes there were "ESTABLISHED" connection ghosts after backend closed them, and in this case "Proxy error" occurred at 100%.
(In reply to comment #88) > Created an attachment (id=22341) [details] > Patch against trunk > Can you please try this patch and add > setenv proxy-initial-not-pooled 1 > to your virtualhost configuration. > Note that it comes with a performance penalty. I tried this just now. It looks working well perfectly. But it has really very large performance problem. Especially, it is very serious with clients which does not have HTTP/1.1 keep-alive like IE on https.
(In reply to comment #89) Here is the debug log related to the "Proxy error" on r657443. It occurred after the backend closes all connection by keep-alive timeout and there were connection ghost. [Fri Aug 01 11:33:07 2008] [info] Initial (No.1) HTTPS request received for child 2 (server example.com:443) [Fri Aug 01 11:33:07 2008] [debug] mod_proxy_http.c(55): proxy: HTTP: canonicalising URL //192.168.1.100/example.gif [Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(1488): [client 192.168.3.125] proxy: http: found worker http://192.168.1.100/ for http://192.168.1.100/example.gif [Fri Aug 01 11:33:07 2008] [debug] mod_proxy.c(966): Running scheme http handler (attempt 0) [Fri Aug 01 11:33:07 2008] [debug] mod_proxy_http.c(1874): proxy: HTTP: serving URL http://192.168.1.100/example.gif [Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2044): proxy: HTTP: has acquired connection for (192.168.1.100) [Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2102): proxy: connecting http://192.168.1.100/example.gif to 192.168.1.100:80 [Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2200): proxy: connected /example.gif to 192.168.1.100:80 [Fri Aug 01 11:33:07 2008] [error] [client 192.168.3.125] (104)Connection reset by peer: proxy: error reading status line from remote server 192.168.1.100 [Fri Aug 01 11:33:07 2008] [debug] mod_proxy_http.c(1395): [client 192.168.3.125] proxy: NOT Closing connection to client although reading from backend server 192.168.1.100 failed. [Fri Aug 01 11:33:07 2008] [error] [client 192.168.3.125] proxy: Error reading from remote server returned by /apps/example.gif [Fri Aug 01 11:33:07 2008] [debug] proxy_util.c(2062): proxy: HTTP: has released connection for (192.168.1.100) [Fri Aug 01 11:33:07 2008] [info] [client 192.168.3.125] Connection closed to child 2 with unclean shutdown (server example.com:443)
*** Bug 45588 has been marked as a duplicate of this bug. ***
With the latest fixes in 2.2.10 this is now fixed and reduces itself to a configuration problem.
*** Bug 46504 has been marked as a duplicate of this bug. ***
Was this problem really fixed? I updated my system to 2.2.11, and add new conf lines; <VirtualHost *:80> ServerName www.example.com DocumentRoot /www <Location /sub/> ProxyPass http://192.168.60.34/sub/ ProxyPassReverse http://192.168.60.34/sub/ SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 </Location> </VirtualHost> But, I still have same problem, and the error log is; [Mon May 18 09:30:48 2009] [error] [client ***.***.***.***] (104)Connection reset by peer: proxy: error reading status line from remote server 192.168.60.34 [Mon May 18 09:30:48 2009] [error] [client ***.***.***.***] proxy: Error reading from remote server returned by /sub/index.html Does anyone know how to fix this?
Sorry! I just forgot to add "SetEnv proxy-initial-not-pooled 1". Now it seems to working well, but really too slow....
I was using apache 2.2.3 and internal web server is IIS 6 hosted on windows 2003. Initially i have tried SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 SetEnv proxy-initial-not-pooled 1 With this change the frequency of getting the proxy error is reduced. And now upgraded to 2.2.15 and got the same result after adding "SetEnv" hack. I have configured the proxy time out to 900 Timeout 900 httpd keepalive option on and off IIS Keep alive time is also 900. The error is consistent if the request made is time consuming. and if approx 10 simultaneous request has send to the server. I was behind this problem for more than 8 months and still no hope :( Appreciate your help! --kirant400
Our hoster company is using an Apache 2.2.9 as Reverse-Proxy. The web server is IIS 6 on windows 2003. We tried also SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 but we get the error like kirant400 if the request is time consuming. Now our hoster and I changed some settings and we have NOT this issue anymore. 1. Unchanged "force-proxy-request-1.0 1" 2. Deactivated "proxy-nokeepalive" on apache 3. Apache Keep Alive set to 900 4. IIS Keep Alive set to 900 Can someone try and confirm this. Thanks, ibram
ibram, please use the users support list to discuss your issue with verbatim details.
Hi, The SetEnv Parameters worked well with my case and I am no longer getting this error but I faced another issue which is HTTP/1.1 417 Expectation Failed. This is due to the apache client application is using HTTP/1.1 and the Proxy was forced to use HTTP/1.0 as per the setEnv Parameters. I found a solution for this problem in this URL: https://lists.gnu.org/archive/html/sks-devel/2013-03/msg00017.html I applied this solution and now my httpd.conf in the proxy server is having the below lines: SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 RequestHeader unset Expect early Thank you for the support.
*** Bug 57520 has been marked as a duplicate of this bug. ***
Hi , Error message shown below: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /. Reason: Error reading from remote server Apache/2.2.15 (CentOS) Server at calmacprod.cigniti.com Port 86 NameVirtualHost *:85 <VirtualHost *:85> ServerName calmacprod.cigniti.com ProxyRequests Off ProxyPreserveHost On <Proxy *> Order allow,deny Allow from all </Proxy> ProxyPass /rm http://159.122.202.173:3060/rm ProxyPassReverse /rm http://159.122.202.173:3060/rm ProxyPass / http://159.122.202.173:8080/ ProxyPassReverse / http://159.122.202.173:8080/ <Location /jira> Order allow,deny Allow from all </Location> #SSLProxyEngine on SSLEngine on SSLCertificateFile /etc/httpd/ssl/131fd0633495ecf7.crt SSLCertificateKeyFile /etc/httpd/ssl/calmacprod.cigniti.com.key ErrorLog logs/jira.cigniti </VirtualHost> NameVirtualHost *:86 <VirtualHost *:86> ServerName calmacprod.cigniti.com ProxyRequests Off ProxyPreserveHost On <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass / http://159.122.202.173:8090/ ProxyPassReverse / http://159.122.202.173:8090/ <Location /confluence> Order allow,deny Allow from all </Location> SSLEngine on SSLCertificateFile /etc/httpd/ssl/131fd0633495ecf7.crt SSLCertificateKeyFile /etc/httpd/ssl/calmacprod.cigniti.com.key ErrorLog logs/confluence.cigniti </VirtualHost> ReWriteEngine On ReWriteCond %{HTPS} off Ratlassian/installation/confluence/jre//bin/java -Djava.util.log ging.config.file=/atlassian/installation/confluence/conf/logging.properties -Djava.util.logging.manager=org.apac he.juli.ClassLoaderLogManager -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -Djava.awt.headless=tru e -Xloggc:/atlassian/installation/confluence/logs/gc-2015-10-27_01-21-59.log -XX:+UseGCLogFileRotation -XX:Numbe rOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-PrintTenuringDistribution - Djava.endorsed.dirs=/atlassian/installation/confluence/endorsed -classpath /atlassian/installation/confluence/bi n/bootstrap.jar:/atlassian/installation/confluence/bin/tomcat-juli.jar -Dcatalina.base=/atlassian/installation/c onfluence -Dcatalina.home=/atlassian/installation/confluence -Djava.io.tmpdir=/atlassian/installation/confluence /temp org.apache.catalina.startup.Bootstrap start postgres 13603 30198 0 01:23 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(47601) idle postgres 14165 30198 0 01:46 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(48008) idle postgres 20868 30198 0 04:18 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50767) idle postgres 20956 30198 0 04:19 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50796) idle postgres 20957 30198 0 04:19 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50797) idle postgres 21001 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50817) idle postgres 21002 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50818) idle postgres 21011 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50823) idle postgres 21012 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50824) idle postgres 21013 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50825) idle postgres 21014 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50830) idle postgres 21015 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50831) idle postgres 21016 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50832) idle postgres 21017 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50833) idle postgres 21018 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50834) idle postgres 21019 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50835) idle postgres 21020 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50836) idle postgres 21021 30198 0 04:20 ? 00:00:00 postgres: confluence1 confluencedb 127.0.0.1(50837) idle root 21040 20801 0 04:21 pts/0 00:00:00 grep confluence root 28153 1 1 Aug11 ? 1-02:47:44 /atlassian/installation/confluence/jre//bin/java -Djava.util.l ogging.config.file=/atlassian/installation/confluence/conf/logging.properties -Djava.util.logging.manager=org.ap ache.juli.ClassLoaderLogManager -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -Djava.awt.headless=t rue -Xloggc:/atlassian/installation/confluence/logs/gc-2015-08-11_08-37-19.log -XX:+UseGCLogFileRotation -XX:Num berOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-PrintTenuringDistribution -Djava.endorsed.dirs=/atlassian/installation/confluence/endorsed -classpath /atlassian/installation/confluence/ bin/bootstrap.jar:/atlassian/installation/confluence/bin/tomcat-juli.jar -Dcatalina.base=/atlassian/installation /confluence -Dcatalina.home=/atlassian/installation/confluence -Djava.io.tmpdir=/atlassian/installation/confluen ce/temp org.apache.catalina.startup.Bootstrap start [root@production ~]# cd /atlassian/installation/confluence/bin/ [root@production bin]# [root@production bin]# ./stop-confluence.sh executing using dedicated user If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian .com/display/DOC/Confluence+Installation+Guide [root@production bin]# ./startup.sh If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide [root@production logs]# tail -f -n 1000 catalina.out "I/O dispatcher 9" #90 daemon prio=5 os_prio=0 tid=0x00007f698c093000 nid=0x6e8a runnable [0x00007f69879f8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb0bc1a8> (a sun.nio.ch.Util$2) - locked <0x00000000cb0bc1b8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb0bc160> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 8" #89 daemon prio=5 os_prio=0 tid=0x00007f698c009000 nid=0x6e89 runnable [0x00007f6987af9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb15b550> (a sun.nio.ch.Util$2) - locked <0x00000000cb15b560> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb15b508> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 7" #88 daemon prio=5 os_prio=0 tid=0x00007f698c007800 nid=0x6e88 runnable [0x00007f6987bfa000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb0dca28> (a sun.nio.ch.Util$2) - locked <0x00000000cb0dca38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb0dc9e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 6" #87 daemon prio=5 os_prio=0 tid=0x00007f698c006000 nid=0x6e87 runnable [0x00007f6967ffe000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb0ff338> (a sun.nio.ch.Util$2) - locked <0x00000000cb0ff348> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb0ff2f0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 5" #86 daemon prio=5 os_prio=0 tid=0x00007f698c00f800 nid=0x6e86 runnable [0x00007f6987cfb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb0bc438> (a sun.nio.ch.Util$2) - locked <0x00000000cb0bc448> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb0bc3f0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 4" #85 daemon prio=5 os_prio=0 tid=0x00007f698c00e000 nid=0x6e85 runnable [0x00007f6987dfc000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb0ff5d8> (a sun.nio.ch.Util$2) - locked <0x00000000cb0ff5e8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb0ff590> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 3" #84 daemon prio=5 os_prio=0 tid=0x00007f698c00c000 nid=0x6e84 runnable [0x00007f6987efd000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb1dbc08> (a sun.nio.ch.Util$2) - locked <0x00000000cb1dbc18> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb1dbbc0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 2" #83 daemon prio=5 os_prio=0 tid=0x00007f698c002000 nid=0x6e83 runnable [0x00007f697fefd000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb0dccb8> (a sun.nio.ch.Util$2) - locked <0x00000000cb0dccc8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb0dcc70> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "I/O dispatcher 1" #82 daemon prio=5 os_prio=0 tid=0x00007f698c001000 nid=0x6e82 runnable [0x00007f6987ffe000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000cb15b7e0> (a sun.nio.ch.Util$2) - locked <0x00000000cb15b7f0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000cb15b798> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) "httpclient-io:thread-1" #81 daemon prio=5 os_prio=0 tid=0x00007f69ec224800 nid=0x6e81 runnable [0x00007f6a08113000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000ca3cece0> (a sun.nio.ch.Util$2) - locked <0x00000000ca3cecf0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000ca3cec98> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57) at java.lang.Thread.run(Thread.java:745) "Timer-3" #54 daemon prio=5 os_prio=0 tid=0x00007f69f00b3000 nid=0x6e63 in Object.wait() [0x00007f6a5818f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.util.TimerThread.mainLoop(Timer.java:526) - locked <0x00000000c8453ee0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "Timer-2" #53 daemon prio=5 os_prio=0 tid=0x00007f69f0029800 nid=0x6e62 in Object.wait() [0x00007f6a0b7fe000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.util.TimerThread.mainLoop(Timer.java:526) - locked <0x00000000c80fb128> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "FelixStartLevel" #52 daemon prio=5 os_prio=0 tid=0x00007f69f4439000 nid=0x6e61 in Object.wait() [0x00007f6a0b5fc000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:279) - locked <0x00000000c809bda8> (a java.util.ArrayList) at java.lang.Thread.run(Thread.java:745) "FelixDispatchQueue" #51 daemon prio=5 os_prio=0 tid=0x00007f69f400c000 nid=0x6e60 in Object.wait() [0x00007f6a0b6fd000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063) - locked <0x00000000c82ffca0> (a java.util.ArrayList) at org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54) at org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101) at java.lang.Thread.run(Thread.java:745) "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" #44 daemon prio=5 os_prio=0 tid=0x00007f6a63788800 nid=0x6e57 runnable [0x00007f6a58290000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1700) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) - locked <0x00000000d5180430> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403) at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLQuery(AbstractJdbc2Connection.java:347) at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLQuery(AbstractJdbc2Connection.java:339) at org.postgresql.jdbc2.AbstractJdbc2Connection.getTransactionIsolation(AbstractJdbc2Connection.java:868) at com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:107) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:198) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" #43 daemon prio=5 os_prio=0 tid=0x00007f6a6b55b800 nid=0x6e56 runnable [0x00007f6a58390000] java.lang.Thread.State: RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method) at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1035) at java.lang.ClassLoader.loadClass(ClassLoader.java:406) - locked <0x00000000c00bf930> (a java.lang.Object) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) - locked <0x00000000c0010630> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:503) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2640) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361) at java.util.ResourceBundle.getBundle(ResourceBundle.java:890) at sun.util.resources.LocaleData$1.run(LocaleData.java:164) at sun.util.resources.LocaleData$1.run(LocaleData.java:160) at java.security.AccessController.doPrivileged(Native Method) at sun.util.resources.LocaleData.getBundle(LocaleData.java:160) at sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:124) at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:697) at java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:146) at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85) at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:359) at java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:349) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580) at org.postgresql.core.Logger.<init>(Logger.java:26) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:105) - locked <0x00000000c74afe48> (a java.lang.Class for org.postgresql.jdbc2.AbstractJdbc2Connection) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:393) at org.postgresql.Driver.connect(Driver.java:267) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" #42 daemon prio=5 os_prio=0 tid=0x00007f6a54206800 nid=0x6e55 waiting for monitor entry [0x00007f6a58492000] java.lang.Thread.State: BLOCKED (on object monitor) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:104) - waiting to lock <0x00000000c74afe48> (a java.lang.Class for org.postgresql.jdbc2.AbstractJdbc2Connection) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:393) at org.postgresql.Driver.connect(Driver.java:267) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) [root@production logs]# Please Advise, Vinay.
Is it still supposed to be open? I see the same behaviour in 2.4.20, Using the "proxy-nokeepalive" to disable keepalives definitely is not the best way to solve this and causes other issues. I am seeing this with Jetty(9.3.3.v20150827) as backend.
(In reply to vin01 from comment #103) > Is it still supposed to be open? I see the same behaviour in 2.4.20, > Using the "proxy-nokeepalive" to disable keepalives definitely is not the > best way to solve this and causes other issues. > I am seeing this with Jetty(9.3.3.v20150827) as backend. If you're hitting a keepalive race, you can also set smax=0 and a ttl lower then your backend keepalive timeout. There's also proxy-initial-not-pooled. Either way, a lot more detail is needed in a report.
I have the same error in this configuration: apache 2.4.6 on a centos 7, configured as https reverse proxy on a filesender server (http://filesender.org/), on a Redhat 5.3 server. Filesender is a software for sending big files with email links. The error occurs clicking on the download.php link (for downloading files), and only for very big files (several GB I guess). Direct links to filesender server work. All suggested solutions don't work. Finally I found the problem, but don't know why it happens: I tought the problem could be in the headers sent by download.php, so I tryied to comment them one by one, so I found that the one which gives error is the one which sends file dimension: header('Content-Length: '.$functions->getFileSize($file)); Commenting out this one eliminates the error. I don't know if it occurs only with certain type of file or in https only, I could try and let you know. Do you think is this a bug related to other ones or a new one?
Any further work on this requires a new bug on contemporary software levels.