Bug 23567 - splitting brigades eats memory proportional to number of FLUSH buckets
Summary: splitting brigades eats memory proportional to number of FLUSH buckets
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.0-HEAD
Hardware: PC Linux
: P3 critical with 6 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: FixedInTrunk
: 34589 43223 (view as bug list)
Depends on:
Blocks:
 
Reported: 2003-10-02 13:35 UTC by Jean-François CUINET
Modified: 2012-02-26 16:29 UTC (History)
3 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jean-François CUINET 2003-10-02 13:35:28 UTC
When serving big files (> 150Mo) with PHP or cgi shell scripts, the child 
process memory size grows proportionnaly to the amount of data sent.

Config :
Linux RedHat 9.0, P200 with 256M
Apache is configured to use prefork MPM.
Comment 1 Jeff Trawick 2003-10-10 12:18:45 UTC
It sounds like there is a filter that is holding on to data when it shouldn't.

Could you please provide a scenario to reproduce the problem?  This would
include
+ simple CGI that does whatever is required
+ set of changes to make to default Apache config to see the problem
+ what request should be sent to this simple CGI

Thanks!
Comment 2 André Malo 2003-10-10 14:09:03 UTC
Maybe deflate? There was something fixed recently.
Comment 3 Jeff Trawick 2003-10-10 14:54:53 UTC
Ah, yes!  Jean-François, please see my recent update to PR 23065
(http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23065) pointing to a fix for
a bug that could result in symptoms like you describe.

This assumes, of course, that you have the DEFLATE filter active.

If this does not turn out to be the cause, information requested early on how to
reproduce the problem would be much appreciated.
Comment 4 Jean-François CUINET 2003-10-10 17:21:08 UTC
I think that DEFLATE filter is not active (SetOutputFilter is not present in 
file httpd.conf)

Problem also noted with Apache 2.047 on AIX5L or Linux RedHat 9.0

Example of cgi shell script for reproduce the problem :

#!/bin/sh                                                
                                                         
echo Content-type: text/plain                            
echo                                                     
                                                         
i=150000                                                 
until test $i -eq 0                                      
do                                                       
  # 50 * 20 lines                                        
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  echo 01234567890123456789012345678901234567890123456789
  i=`expr $i - 1`                                        
done                                                     
Comment 5 Jeff Trawick 2003-10-20 10:58:41 UTC
I spent some time last week-end investigating this.  Hopefully this info can
help whoever has time to look for a solution.

The only memory leak I was able to find is caused by an endless number of
cleanups registered against the request pool by content-length filter and then
killed by the core output filter.  A cleanup is registered during
apr_brigade_split(), as the content-length filter separates out the data already
read from the CGI in order to send it to the client.  When the content-length
filter is able to read a bit more from the CGI, the same thing happens again.

This same pattern could occur with most any other filter.  Other filter code
that could do the pipe bucket read is going to have to do the same thing --
brigade-split and pass-brigade -- before waiting for more output from the CGI.
Comment 6 William A. Rowe Jr. 2004-06-16 19:22:25 UTC
  Just a footnote - I'd heard similar responses on another application, a web
  server attempting to feed unlimited streams of data (e.g. audio/video) through
  an HTTP proxy.  One would expect it to reach steady-state, but throughout
  a long connection to mod_proxy the memory footprint continued to grow.
Comment 7 mh 2004-06-17 17:18:39 UTC
an additional comment on this bug that might be helpful: bug 29528 is similar, but
I wasn't getting the increased mem. usage proportional to the amount of data sent. 
29528 had to do with the "Range" header. If download with a client that does not
use the range header, no memory leak occured. This bug happens without any range
header. The one difference between the 2 is that this cgi sends text/plain data
while mine sent audio (audio/mpeg, mp3) data. Obviously something is being
handled differently for the various content types.
Comment 8 Tyler 2004-10-05 19:43:34 UTC
I'm not sure whether bug 29962 and this bug are actually different or not.  I
had a php script that pulled a huge amount of data into memory (800 megabytes),
but Apache 2.0 didn't release the memory afterwards.  None of the data was sent
to the end user.  It was just loaded into memory.  It looks to me like there's a
major problem with Apache 2.0 releasing memory once it's claimed it.

This was eventually causing our server to thrash itself to death.  As more and
more processes grew to consome all available memory.
Comment 9 André Malo 2004-10-05 20:02:28 UTC
Good question: Did apache not release it or mod_php?
Comment 10 Joe Orton 2005-01-12 11:33:40 UTC
Just brain-dumping, the design issue here is, per Jeff's analysis and subsequent
discussion:

1. brigades are allocated out of pools
2. every call to apr_brigade_split allocates a new brigade
3. every time a FLUSH bucket it sent down the filter stack, it causes at least
one call to apr_brigade_split

fixes for this could be either:

fix (1), allocate brigades out of the bucket-allocator so that they can really
be free'd (very intrusive since many filters presume brigades are never really
destroyed)

fix (3), adjust all filters to ensure that they don't allocate a number of
brigades per request (and hence, memory allocated) which is proportional to
number of FLUSH buckets sent.
Comment 11 Joe Orton 2005-09-07 17:31:44 UTC
*** Bug 34589 has been marked as a duplicate of this bug. ***
Comment 12 William A. Rowe Jr. 2007-08-28 02:11:49 UTC
*** Bug 43223 has been marked as a duplicate of this bug. ***
Comment 13 Ruediger Pluem 2007-08-28 12:51:05 UTC
(In reply to comment #10)
> Just brain-dumping, the design issue here is, per Jeff's analysis and subsequent
> discussion:
> 
> 1. brigades are allocated out of pools
> 2. every call to apr_brigade_split allocates a new brigade
> 3. every time a FLUSH bucket it sent down the filter stack, it causes at least
> one call to apr_brigade_split
> 
> fixes for this could be either:
> 
> fix (1), allocate brigades out of the bucket-allocator so that they can really
> be free'd (very intrusive since many filters presume brigades are never really
> destroyed)
> 
> fix (3), adjust all filters to ensure that they don't allocate a number of
> brigades per request (and hence, memory allocated) which is proportional to
> number of FLUSH buckets sent.

What about an apr_brigade_split_ex that takes an additional brigade as
parameter? This brigade could be used instead of creating a new one and could
be stored in the context of the filter for subsequent recyling (like we do in
other places). So something like:

APU_DECLARE(apr_bucket_brigade *) apr_brigade_split_ex(apr_bucket_brigade *b,
                                                       apr_bucket *e
                                                       apr_bucket_brigade *a)
{
    apr_bucket *f;

    if (!a) {
        a = apr_brigade_create(b->p, b->bucket_alloc);
    }
    else if (!APR_BRIGADE_EMPTY(a)) {
        apr_brigade_cleanup(a);
    }
    /* Return an empty brigade if there is nothing left in·
     * the first brigade to split off·
     */
    if (e != APR_BRIGADE_SENTINEL(b)) {
        f = APR_RING_LAST(&b->list);
        APR_RING_UNSPLICE(e, f, link);
        APR_RING_SPLICE_HEAD(&a->list, e, f, apr_bucket, link);
    }

    APR_BRIGADE_CHECK_CONSISTENCY(a);
    APR_BRIGADE_CHECK_CONSISTENCY(b);

    return a;
}

APU_DECLARE(apr_bucket_brigade *) apr_brigade_split(apr_bucket_brigade *b,
                                                       apr_bucket *e)          
                                    
{
   return apr_brigade_split_ex(b, e, NULL);
}
Comment 14 Ruediger Pluem 2009-11-04 09:12:55 UTC
Is this still an issue in trunk with r814807, r821471, r821477 committed?
Comment 15 Stefan Fritsch 2010-05-17 17:24:51 UTC
(In reply to comment #14)
> Is this still an issue in trunk with r814807, r821471, r821477 committed?

No, this is now fixed.

Some other relevant commits are:
r924452 r910326 r821481 r821486
Comment 16 Stefan Fritsch 2012-02-26 16:29:12 UTC
fixed in 2.4.1