Bug 51747 - mod_fcgid loads entire request into memory while passing to FCGI backend
Summary: mod_fcgid loads entire request into memory while passing to FCGI backend
Status: REOPENED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_fcgid (show other bugs)
Version: 2.2.19
Hardware: PC Linux
: P2 blocker (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate, PatchAvailable
: 52282 60353 61028 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-08-31 16:27 UTC by Dominic Benson
Modified: 2018-11-28 08:59 UTC (History)
7 users (show)



Attachments
Patch to prevent memory explosion when fcgid hands off to FCGI backend (1.62 KB, patch)
2011-08-31 16:27 UTC, Dominic Benson
Details | Diff
Updated patch against 2.3.7 (1.51 KB, patch)
2013-02-28 22:11 UTC, Dominic Benson
Details | Diff
Revised patch fixing a bug introduced in the previous update for 2.3.7 (1.62 KB, patch)
2013-03-21 16:32 UTC, Dominic Benson
Details | Diff
Extended patch for windows & unix (3.67 KB, patch)
2015-08-28 12:03 UTC, Jan Stürtz
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dominic Benson 2011-08-31 16:27:20 UTC
Created attachment 27448 [details]
Patch to prevent memory explosion when  fcgid hands off to FCGI backend

While FcgidMaxRequestInMem limits the amount of memory used by the server while the client request is being sent, the process of reading the file (via APR_BUCKET_NEXT) to write via proc_write_ipc causes the entire contents to be in memory in the server process simultaneously.

The problem seems to be that, while data is passed to the FCGI process in small chunks, the apr_bucket_brigade keeps all its buckets as the file is read, and they morph into heap type.

Buckets should be removed from the brigade as the data is delivered, per the guidelines for writing output filters.

Steps to reproduce:
 httpd 2.2.19 on x86 with mod_fcgid 2.3.6
 Create a file of 2.5GB+
 Set FcgidMaxRequestLen to at least the size of this file
 Send the file in a request that will be handled by fcgid

What happens:
 A few seconds after the request body is transferred (depending on server speed), the httpd child handling the request segfaults, as it exceeds the process address limit

What should happen:
 The httpd process should stay at modest memory usage throughout, and the request should be processed by the FCGI backend


The attached patch prevents the end-of-request explosion. I think it probably ought to have some more checks on the added bucket/brigade processing, though.
Comment 1 William A. Rowe Jr. 2012-10-30 21:23:25 UTC
*** Bug 52282 has been marked as a duplicate of this bug. ***
Comment 2 Dominic Benson 2013-02-28 22:11:38 UTC
Created attachment 30006 [details]
Updated patch against 2.3.7

The original patch doesn't apply correctly against 2.3.7, as the addition of:

if (APR_BUCKET_IS_METADATA(e)) {
            continue;
        }

causes an infinite loop (the APR_BUCKET_REMOVE hadn't happened at that point). This patch corrects this; now the APR_BUCKET_REMOVE is the first action in the loop, so it ought to be more robust against future changes.
Comment 3 Dominic Benson 2013-03-21 16:32:05 UTC
Created attachment 30093 [details]
Revised patch fixing a bug introduced in the previous update for 2.3.7

The previous updated patch against 2.3.7 had a bug that it could run into a blocking read that would never return, owing to the bucket not being in the brigade at the point at which the data was read.
This issue seems to only affect SSL at certain file sizes, but those are likely only loosely related as issues.
This updated version restores the order so that the read is prior to the removal from the original brigade, while still avoiding the issue when applying the original patch (written for 2.3.6) against 2.3.7 that metadata buckets would cause an infinite loop.
Comment 4 Dis 2013-10-12 15:06:02 UTC
"Me too" and the Mar 21 patch fixed it. (For future searchers, I hit this on Ubuntu through the owncloud user migration plugin - it can't upload the large dataset successfully. Without this patch, it coredumps but with it applied it worked.)
Comment 5 Jan Stürtz 2015-08-28 12:01:27 UTC
Also occured on Windows with Apache 2.2.27 and mod_fcgid 2.3.9.
The previous patch, patches only unix. Added an new diff patching windows & unix, based on the previous patch, developed on 2.3.9 sources
Comment 6 Jan Stürtz 2015-08-28 12:03:10 UTC
Created attachment 33050 [details]
Extended patch for windows & unix
Comment 7 William A. Rowe Jr. 2018-11-07 21:09:40 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.
Comment 8 Joe Orton 2018-11-28 08:57:52 UTC
Committed in r1847624 - thanks for the patches.
Comment 9 Joe Orton 2018-11-28 08:58:43 UTC
*** Bug 61028 has been marked as a duplicate of this bug. ***
Comment 10 Joe Orton 2018-11-28 08:59:43 UTC
*** Bug 60353 has been marked as a duplicate of this bug. ***