Bug 44123 - Apache slowly consumes all system memory while running streaming CGI script.
Summary: Apache slowly consumes all system memory while running streaming CGI script.
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_cgi (show other bugs)
Version: 2.2.3
Hardware: Other Linux
: P2 major (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: FixedInTrunk
Depends on:
Blocks:
 
Reported: 2007-12-21 14:28 UTC by Garrett Gallant
Modified: 2012-02-26 16:47 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Garrett Gallant 2007-12-21 14:28:28 UTC
Hello, I'm trying to solve a problem we're having where some of our long-
running streaming CGI scripts are causing Apache to grow continuously, 
eventually consuming all system memory and starving Apache and other processes 
(manifested as Apache returning HTTP 500 (internal server error) responses to 
subsequent requests to other resources, and other weirdness). I've reduced the 
test to a really simple CGI script which reproduces the memory growth reliably 
for me (see steps to reproduce below).

I'm wondering if this is normal behaviour (is continuous streaming not 
supported?), or perhaps I'm just mangling my headers or missing something 
really obvious. Any insight you guys have would be great. If it's a bug in 
Apache, my apologies for darkening your door at Christmastime. ;-)

I've observed the behaviour on both of our setups:

Linux: 2.6.13.4.D400-LEAF (a custom (i.e. suspect) roll-our-own Linux - gcc 
version 4.0.2 20051125, Red Hat 4.0.2-8).
Apache: 2.0.55

Linux: Ubuntu Fiesty (standard, nothing special)
Apache: 2.2.3

Steps to reproduce:
-------------------
1. Restart Apache.
2. Point a browser at the CGI script (see offending cgi script (1) below) 
running either runPlain() or runMultipart().
3. Run ps every 10 seconds for 10 minutes (see ps log script (2) below).

My observations:
-----------------
1. In both the plain/text and multipart case, I observed Apache's memory 
growing linearly at a rate of ~19.6kB/min (typically in 4kB increments 
depending on your Linux/MMU page size).

2. When I stopped the script after 5 minutes, Apache did not appear to free its 
memory. Then, when I restarted the script, Apache ran flat for 5 minutes before 
starting to grab more memory. This suggests that when a CGI script stops, 
Apache frees some of its internal memory pool, but does not release it to 
system memory, instead reusing it for future CGI scripts. I found I had to 
restart Apache after each test to avoid masking the memory growth of subsequent 
tests.

3. I tried various data rates in my below scripts, by adjusting the sleep time, 
chars per line and lines per cycle, and found the memory growth rate was 
roughly a linear function of the CGI's data output rate (with the function 
flattening at high data rates probably due to CPU/network saturation).

4. I tried various other stabs in the dark, like flushing stdout on each cycle 
(before the sleep). No change to the growth rate.

1. Offending cgi script:
------------------------
#include <time.h>
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

/*
main.
*/
int main(int argc, char *argv[])
{
    runPlain();
}

/*
Run a text/plain test.
*/
int runPlain()
{
    int i,j;
    printf("Content-type: text/plain; charset=us-ascii\n\n");
    for (i=0; ; i++)
    {
        for (j=0; j<5; j++)
        {
            printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka\n",i,j);
        }
        printf("\n");
        usleep(5000);
    }
}

/*
Run a multipart/mixed-replace test
per http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html section 7.2.1.
*/
int runMultipart()
{
    int i,j,num;
    printf("Content-type: multipart/mixed-replace; boundary=myboundary\n\n");
    printf("Preamble\n");
    for (i=0; ; i++)
    {
        printf("--myboundary\n");
        printf("Content-type: text/plain; charset=us-ascii\n\n");
        for (j=0; j<5; j++)
        {
            printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka\n",i,j);
        }
        printf("\n");
        usleep(5000);
    }
}

2. ps log:
----------
#!/bin/sh

interval=$1
while :
do
  timestamp=`date +%Y.%m.%d-%H.%M.%S`
  ps -o pid,pcpu,mem,cmd | sed "s/^/$timestamp  /"
  sleep $interval
done
Comment 1 Stefan Fritsch 2011-06-15 22:23:19 UTC
This is fixed in trunk. The relevant commits are

r924452
r910326
r821481
r821477
r821471
r814807

and maybe others.
Comment 2 Stefan Fritsch 2012-02-26 16:47:38 UTC
2.4.1 is released, closing