Bug 34221 - Apache server probably falls into endless cpu-consuming loop.
Summary: Apache server probably falls into endless cpu-consuming loop.
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.0.52
Hardware: Other Linux
: P1 major (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Depends on:
Reported: 2005-03-29 14:03 UTC by Yannis Tsakiris
Modified: 2005-12-08 10:50 UTC (History)
1 user (show)


Note You need to log in before you can comment on or make changes to this bug.
Description Yannis Tsakiris 2005-03-29 14:03:00 UTC
I have two identical servers (2 cpus, pIII @ 1000MHz, 1 GB each) running SuSE 
Linux 9.1 (kernel 2.4) with 2.0.52 apache installed and php 4.3.11 (apache 
The computers are used as web servers with no other software running (like 
database servers, etc). At rush hour the two machines serve about 300,000 
requests both (the load is balanced equally).
At rush hour the load average of each computer reaches a maximum of 0.5-0.6.
Ar random time one of the two computers stops serving pages, all opened httpd 
processes are busy (running) trying to consume cpu. The load average increases 
quickly and in few seconds it approaches the number of the maximum server 
processes (500 processes). At this conditions the machine responds very 
slowly, you can hardly type commands in the shell, and no respond at all to 
web requests. When this happens I reboot the computer and everything is coming 
back to normal.
I checked the logs both apache's and system's but there is nothing suspicious, 
just normal messages ending at the time of the stuck.
This can happen at any time, and at any of the two computers. It has occur 
after a few hours of uptime or after 20 days since the last reboot. Anyway, as 
an average we can accept that it happens once in a week.
I discussed this in the apache web server user's forums and many people said 
that have the same problem, even when using apache without any modules (like 
php). The do not have a decent explanation and the only thing they suggest is 
to downgrade to apache 1.3 (where the problem exists no more).
I looked at the changelog of 2.0.53 but I can't see something relevant.
The symptoms make me think that all of the httpd processes fall into an cpu 
consuming infinite-loop.
Comment 1 Joe Orton 2005-03-29 14:08:12 UTC
To diagnose this further you need to get a backtrace out of one of the httpd
processes which is consuming CPU, by picking a pid (e.g. out of top) and running:

  $ gdb /path/to/httpd <pid>

then enter "backtrace" at the gdb prompt; then attach the output to this bug report.
Comment 2 Paul Querna 2005-04-06 22:59:59 UTC
Ping. Looking for feedback from the reporter.
Comment 3 Yannis Tsakiris 2005-04-07 17:17:43 UTC
This is the output from the backtrace, I was this helps.

(gdb) backtrace
#0  0x4023ab36 in poll () from /lib/i686/libc.so.6
#1  0x4008f4c4 in apr_poll (aprset=0xbffff880, num=1, nsds=0xbffff87c, 
timeout=10000) at poll.c:129
#2  0x4008fb9f in apr_wait_for_io_or_timeout (f=0x2710, s=0x1, for_read=1) at 
#3  0x40085831 in apr_socket_recv (sock=0x81eb1a8, 
    buf=0x82386a0 "GET /assets/diakopes_st_winter_prosfores.gif HTTP/1.1
\r\nAccept: */*\r\nReferer: http://ads.in.gr/custom/topbanners.php\r\nAccept-
Language: el\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: Mozilla/4.0 
    len=0xbffff958) at sendrecv.c:86
#4  0x4001f65e in socket_bucket_read (a=0x81ed450, str=0xbffff954, 
len=0xbffff958, block=APR_BLOCK_READ)
    at apr_buckets_socket.c:35
#5  0x40020275 in apr_brigade_split_line (bbOut=0x81f3f30, bbIn=0x81eb630, 
block=APR_BLOCK_READ, maxbytes=8192)
    at apr_brigade.c:288
#6  0x080920b8 in core_input_filter (f=0x81eb5f8, b=0x81f3f30, 
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
    at core.c:3741
#7  0x0808a92a in ap_get_brigade (next=0x2710, bb=0x81f3f30, 
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
    at util_filter.c:474
#8  0x0808a92a in ap_get_brigade (next=0x2710, bb=0x81f3f30, 
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
    at util_filter.c:474
#9  0x0808b87a in ap_rgetline_core (s=0x81f32b8, n=8192, read=0xbffffaac, 
r=0x81f32a0, fold=0, bb=0x81f3f30)
    at protocol.c:214
#10 0x0808bdde in read_request_line (r=0x81f32a0, bb=0x81f3f30) at 
#11 0x0808c4ca in ap_read_request (conn=0x81eb280) at protocol.c:886
#12 0x0806916a in ap_process_http_connection (c=0x81eb280) at http_core.c:268
#13 0x0808860b in ap_run_process_connection (c=0x81eb280) at connection.c:42
#14 0x0807d31d in child_main (child_num_arg=-4) at prefork.c:609
#15 0x0807d461 in make_child (s=0x80f6808, slot=252) at prefork.c:703
#16 0x0807d65c in perform_idle_server_maintenance (p=0x80b9ff0) at 
#17 0x0807db73 in ap_mpm_run (_pconf=0x120, plog=0x80f20d0, s=0x0) at 
#18 0x08082fba in main (argc=3, argv=0xbffffd84) at main.c:617

Comment 4 Paul Querna 2005-06-02 23:46:24 UTC
If all the HTTPD Processes are waiting in Poll, it sounds like a Denial of
Service Attack that is sending requests very slowly.
Comment 5 Joshua Slive 2005-12-08 19:50:11 UTC
Probable DoS attack.