Bug 44123

Summary: Apache slowly consumes all system memory while running streaming CGI script.
Product: Apache httpd-2 Reporter: Garrett Gallant <garrett.gallant>
Component: mod_cgiAssignee: Apache HTTPD Bugs Mailing List <bugs>
Severity: major Keywords: FixedInTrunk
Priority: P2    
Version: 2.2.3   
Target Milestone: ---   
Hardware: Other   
OS: Linux   

Description Garrett Gallant 2007-12-21 14:28:28 UTC
Hello, I'm trying to solve a problem we're having where some of our long-
running streaming CGI scripts are causing Apache to grow continuously, 
eventually consuming all system memory and starving Apache and other processes 
(manifested as Apache returning HTTP 500 (internal server error) responses to 
subsequent requests to other resources, and other weirdness). I've reduced the 
test to a really simple CGI script which reproduces the memory growth reliably 
for me (see steps to reproduce below).

I'm wondering if this is normal behaviour (is continuous streaming not 
supported?), or perhaps I'm just mangling my headers or missing something 
really obvious. Any insight you guys have would be great. If it's a bug in 
Apache, my apologies for darkening your door at Christmastime. ;-)

I've observed the behaviour on both of our setups:

Linux: (a custom (i.e. suspect) roll-our-own Linux - gcc 
version 4.0.2 20051125, Red Hat 4.0.2-8).
Apache: 2.0.55

Linux: Ubuntu Fiesty (standard, nothing special)
Apache: 2.2.3

Steps to reproduce:
1. Restart Apache.
2. Point a browser at the CGI script (see offending cgi script (1) below) 
running either runPlain() or runMultipart().
3. Run ps every 10 seconds for 10 minutes (see ps log script (2) below).

My observations:
1. In both the plain/text and multipart case, I observed Apache's memory 
growing linearly at a rate of ~19.6kB/min (typically in 4kB increments 
depending on your Linux/MMU page size).

2. When I stopped the script after 5 minutes, Apache did not appear to free its 
memory. Then, when I restarted the script, Apache ran flat for 5 minutes before 
starting to grab more memory. This suggests that when a CGI script stops, 
Apache frees some of its internal memory pool, but does not release it to 
system memory, instead reusing it for future CGI scripts. I found I had to 
restart Apache after each test to avoid masking the memory growth of subsequent 

3. I tried various data rates in my below scripts, by adjusting the sleep time, 
chars per line and lines per cycle, and found the memory growth rate was 
roughly a linear function of the CGI's data output rate (with the function 
flattening at high data rates probably due to CPU/network saturation).

4. I tried various other stabs in the dark, like flushing stdout on each cycle 
(before the sleep). No change to the growth rate.

1. Offending cgi script:
#include <time.h>
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char *argv[])

Run a text/plain test.
int runPlain()
    int i,j;
    printf("Content-type: text/plain; charset=us-ascii\n\n");
    for (i=0; ; i++)
        for (j=0; j<5; j++)
            printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka\n",i,j);

Run a multipart/mixed-replace test
per http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html section 7.2.1.
int runMultipart()
    int i,j,num;
    printf("Content-type: multipart/mixed-replace; boundary=myboundary\n\n");
    for (i=0; ; i++)
        printf("Content-type: text/plain; charset=us-ascii\n\n");
        for (j=0; j<5; j++)
            printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka\n",i,j);

2. ps log:

while :
  timestamp=`date +%Y.%m.%d-%H.%M.%S`
  ps -o pid,pcpu,mem,cmd | sed "s/^/$timestamp  /"
  sleep $interval
Comment 1 Stefan Fritsch 2011-06-15 22:23:19 UTC
This is fixed in trunk. The relevant commits are


and maybe others.
Comment 2 Stefan Fritsch 2012-02-26 16:47:38 UTC
2.4.1 is released, closing