|Summary:||Apache server probably falls into endless cpu-consuming loop.|
|Product:||Apache httpd-2||Reporter:||Yannis Tsakiris <gtsakiris>|
|Component:||All||Assignee:||Apache HTTPD Bugs Mailing List <bugs>|
Description Yannis Tsakiris 2005-03-29 14:03:00 UTC
I have two identical servers (2 cpus, pIII @ 1000MHz, 1 GB each) running SuSE Linux 9.1 (kernel 2.4) with 2.0.52 apache installed and php 4.3.11 (apache module). The computers are used as web servers with no other software running (like database servers, etc). At rush hour the two machines serve about 300,000 requests both (the load is balanced equally). At rush hour the load average of each computer reaches a maximum of 0.5-0.6. Ar random time one of the two computers stops serving pages, all opened httpd processes are busy (running) trying to consume cpu. The load average increases quickly and in few seconds it approaches the number of the maximum server processes (500 processes). At this conditions the machine responds very slowly, you can hardly type commands in the shell, and no respond at all to web requests. When this happens I reboot the computer and everything is coming back to normal. I checked the logs both apache's and system's but there is nothing suspicious, just normal messages ending at the time of the stuck. This can happen at any time, and at any of the two computers. It has occur after a few hours of uptime or after 20 days since the last reboot. Anyway, as an average we can accept that it happens once in a week. I discussed this in the apache web server user's forums and many people said that have the same problem, even when using apache without any modules (like php). The do not have a decent explanation and the only thing they suggest is to downgrade to apache 1.3 (where the problem exists no more). I looked at the changelog of 2.0.53 but I can't see something relevant. The symptoms make me think that all of the httpd processes fall into an cpu consuming infinite-loop.
Comment 1 Joe Orton 2005-03-29 14:08:12 UTC
To diagnose this further you need to get a backtrace out of one of the httpd processes which is consuming CPU, by picking a pid (e.g. out of top) and running: $ gdb /path/to/httpd <pid> then enter "backtrace" at the gdb prompt; then attach the output to this bug report.
Comment 2 Paul Querna 2005-04-06 22:59:59 UTC
Ping. Looking for feedback from the reporter.
Comment 3 Yannis Tsakiris 2005-04-07 17:17:43 UTC
This is the output from the backtrace, I was this helps. (gdb) backtrace #0 0x4023ab36 in poll () from /lib/i686/libc.so.6 #1 0x4008f4c4 in apr_poll (aprset=0xbffff880, num=1, nsds=0xbffff87c, timeout=10000) at poll.c:129 #2 0x4008fb9f in apr_wait_for_io_or_timeout (f=0x2710, s=0x1, for_read=1) at waitio.c:53 #3 0x40085831 in apr_socket_recv (sock=0x81eb1a8, buf=0x82386a0 "GET /assets/diakopes_st_winter_prosfores.gif HTTP/1.1 \r\nAccept: */*\r\nReferer: http://ads.in.gr/custom/topbanners.php\r\nAccept- Language: el\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: Mozilla/4.0 (compa"..., len=0xbffff958) at sendrecv.c:86 #4 0x4001f65e in socket_bucket_read (a=0x81ed450, str=0xbffff954, len=0xbffff958, block=APR_BLOCK_READ) at apr_buckets_socket.c:35 #5 0x40020275 in apr_brigade_split_line (bbOut=0x81f3f30, bbIn=0x81eb630, block=APR_BLOCK_READ, maxbytes=8192) at apr_brigade.c:288 #6 0x080920b8 in core_input_filter (f=0x81eb5f8, b=0x81f3f30, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3741 #7 0x0808a92a in ap_get_brigade (next=0x2710, bb=0x81f3f30, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:474 #8 0x0808a92a in ap_get_brigade (next=0x2710, bb=0x81f3f30, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:474 #9 0x0808b87a in ap_rgetline_core (s=0x81f32b8, n=8192, read=0xbffffaac, r=0x81f32a0, fold=0, bb=0x81f3f30) at protocol.c:214 #10 0x0808bdde in read_request_line (r=0x81f32a0, bb=0x81f3f30) at protocol.c:592 #11 0x0808c4ca in ap_read_request (conn=0x81eb280) at protocol.c:886 #12 0x0806916a in ap_process_http_connection (c=0x81eb280) at http_core.c:268 #13 0x0808860b in ap_run_process_connection (c=0x81eb280) at connection.c:42 #14 0x0807d31d in child_main (child_num_arg=-4) at prefork.c:609 #15 0x0807d461 in make_child (s=0x80f6808, slot=252) at prefork.c:703 #16 0x0807d65c in perform_idle_server_maintenance (p=0x80b9ff0) at prefork.c:838 #17 0x0807db73 in ap_mpm_run (_pconf=0x120, plog=0x80f20d0, s=0x0) at prefork.c:1039 #18 0x08082fba in main (argc=3, argv=0xbffffd84) at main.c:617
Comment 4 Paul Querna 2005-06-02 23:46:24 UTC
If all the HTTPD Processes are waiting in Poll, it sounds like a Denial of Service Attack that is sending requests very slowly.
Comment 5 Joshua Slive 2005-12-08 19:50:11 UTC
Probable DoS attack.