httpd-2.0.49. i have no filters or anything like that defined, no mod_deflate, etc. I have a CGI that worked perfectly with 1.3.x that is now causing 2.0.x to run out of memory. The CGI generates an MP3 file (sometimes 100+MB in size) and then prints it to STDOUT. here is the what httpd's mem. usage looks like in top: PID USER PR NI VIRT SWAP RES SHR S %CPU TIME %MEM COMMAND 16495 apache 16 0 277m 133m 144m 133m S 0.0 0:00 14.3 httpd 11813 apache 16 0 263m 133m 130m 133m S 0.0 0:00 13.0 httpd 11812 apache 15 0 361m 232m 129m 133m S 0.0 0:01 12.9 httpd 11815 apache 16 0 333m 208m 124m 133m S 0.0 0:01 12.4 httpd 11809 apache 16 0 316m 227m 89m 133m S 0.0 0:00 8.9 httpd If I then try to restart httpd, it seems that some child processes have become unkillable. If use kill -9 and restart, I try to reproduce the problem, but cannot with my broadband connection. I suspect it is slow clients on dialup modems that is causing httpd to buffer the data??? here is the important part of what the perl CGI does: sub send_file ($) { my $file = shift; $|=1; my $begin = 0; my $range_header = "Accept-Ranges: bytes"; my $size = my $filesize = (stat($file))[7]; if ($ENV{'HTTP_RANGE'} =~ /^bytes=(\d+)/) { $begin = $1; $size = $filesize - $begin; my $tmp = $filesize-1; $range_header .= "\nContent-Range: bytes $begin-$tmp/$filesize"; } print "Content-Type: audio/mpeg Content-Length: $size $range_header "; open( MP3,"< $file"); binmode MP3; seek(MP3, $begin, 0); # from the beginning while (read MP3, my $buf, 32768) { print STDOUT $buf; } close MP3; }
One more thing that I forgot to mention: this is with prefork and mod_cgi, not mod_cgid. -m
Another thing that I forgot, is that the CGI gets called via mod_rewrite, an internal rewrite so that a browser request of something like: /128/20040610.23.00-01.15.mp3 will get mapped to /cgi-bin/archive.cgi?rate=128&range=20040610.23.00-01.15&codec=mp3 Here are the rewrite rules that allows this to happen: RewriteEngine On RewriteRule "^([0-9]+)/([a-z0-9]+)\.([-0-9:\.]+)\.pls$" /cgi-bin/pls.cgi?rate=$1&range=$3&codec=$2 [L,E=QUERY_STRING:rate=$1&range=$3&codec=$2] #RewriteRule "^([0-9]+)/([-0-9:\.]+)\.mp3$" /cgi-bin/mp3.cgi?rate=$1&range=$2 [L,E=QUERY_STRING:rate=$1&range=$2] RewriteRule "^([0-9]+)/([a-z0-9]+)\.([-0-9:\.]+)\.m3u$" /cgi-bin/m3u.cgi?rate=$1&range=$3&codec=$2 [L,E=QUERY_STRING:rate=$1&range=$3&codec=$2] RewriteRule "^([0-9]+)/([-0-9:\.]+)\.test$" /cgi-bin/archive2.cgi?rate=$1&range=$2&codec=mp3 [L,E=QUERY_STRING:rate=$1&range=$2&codec=mp3] RewriteRule "^([0-9]+)/([-0-9:\.]+)\.(...)$" /cgi-bin/archive.cgi?rate=$1&range=$2&codec=$3 [L,E=QUERY_STRING:rate=$1&range=$2&codec=$3]
See also http://issues.apache.org/bugzilla/show_bug.cgi?id=23567
yup, when I use the example cgi script in bug #23567, I can reproduce the problem here. Most likely the same bug. This rules out mod_rewrite and slow clients as well. Interestingly, when it's an already bloated cgi that handles the request, it doesn't start growing until the amount of data sent by the example CGI exceeds the existing size of the httpd process. I've upped the severity to Major. -m
Maybe three bugs separate here: 1) memory leak in brigade and cleanup handling, bug 23567 2) ap_byterange_filter sucks the entire brigade into memory, eek 3) for a CGI script which does its own Range handling, the byterange_filter doesn't get out of the way You could try: http://cvs.apache.org/~jorton/ap_cgirange.diff to solve (3)
Created attachment 11855 [details] notice if a CGI script does its own range handling
So the patch works perfectly, before the patch using Winget I get huge httpd's after the patch the httpd processes do not grow in size at all! this is great, I can finally use apache 2.0.x. I guess that patch can be included into 2.0.50?
Well, we have a voting system, this patch makes perfect sense here.
well. some bad news. It seems that the memory leak is still there. Not as frequent or as easily reproducable, but still there. Probably bug 23567? I've starting logging the memory usage via mrtg, this machine has 1 gig of ram, so 60% is quite a lot. When the usage goes down to ~11%, it's cause I've restarted the server. I was hoping that the logging combined with the PID would allow me to pin the bug to a separate user-agent, but no such luck. Maybe RealMedia's player (RMA), but not sure. Definitely not HTTP_RANGE requests this time though. The mrtg graphs are at: http://secure.ckut.ca/mrtg/httpd.html -m
There are lots of leaks in various places which are being revelead. Once the right approach to fixing these has been decided upon I'll post the set of patches I'm accumulating for 2.0. Are any dynamic content generators like mod_include or PHP in use on this site?
Except for php, there are no dynamic content generators, I purposely commented out mod_include. mod_perl is loaded but not being used. No filters are defined either. I have one php script in use. It's extremely simple and could easily be replaced with a CGI. I'll attach it hear. I'm looking forward to your patches.
Created attachment 11924 [details] the one php script in use on the site.
OK per comment 5, of the three issues here, two are fixed and one is Difficult, but that's covered by bug 23567. The "byterange filter eats all my memory" issue is bug 29962.