Bug 63098 - Use after free errors when fuzzing against Apache httpd using an APR pool debug build
Summary: Use after free errors when fuzzing against Apache httpd using an APR pool deb...
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.37
Hardware: All All
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-01-22 12:29 UTC by Rainer Jung
Modified: 2019-01-23 10:53 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rainer Jung 2019-01-22 12:29:06 UTC
A reporter uncovered use after free issues in the Apache HTTP Server triggered by a fuzzing tool when APR pool debugging is in use. Pool debugging is not enabled by default, and should not be used in production environments. The bugs do not get triggered in a default build and we would not classify them as a "zero day vulnerability".

The reporter has provided additional info publicly under:
https://github.com/hannob/apache-uaf/.

Daniel Gruno from the web server PMC has looked at the issue extensively, and with tens of millions of tests against httpd versions 2.4.29 through 2.4.38 (he reworked Hanno's code into a threaded python test) with no pool debugging. Those tests did not show any problems. We therefore believe the problem to not occur without pool debugging.

With pool debugging most of the occurrences happen when HTTP/2 is used via mod_http2. Those problems are, at least partially, understood. There are discussion between the web server and the APR project on how to best fix them. As already noted, pool debugging is not meant for production use.

Another type of error also contained in the same original report, an assert in __pthread_tpp_change_priority pointing at a pthread mutex problem, was analyzed and fixed. We are not aware of a successful reproduction of that issue with 2.4.37 (or later).

Unfortunately the OpenBSD platform uses APR pool debugging by default. Due to another bug, these builds would crash when using HTTP/2, so we do currently not expect them to experience the majority of the use after free issues. We are in contact with OpenBSD maintainers on how to best resolve the situation.

The Apache HTTP Server is a volunteer project, where people work on the project as time and ability allows, and as such, we do triage problems based on some simple rules;

- Is the issue reproducible in standard (production) systems, or is it
  solely pertaining to debug/testing scenarios?
- Which version are affected? (old versus new, stable versions)
- Does the potential impact merit an immediate fix as CVE, or is it a bug
  added to the list of things to do at a slower pace?

We examined the report closely, and determined that we needed more information from the reporter on a reproducible vector here, as we were unable to, in any way, reliably reproduce the issue on standard systems. As a response was not received for several months, and as we deemed this did not impact stability or security to a degree that merited concern, we have been in a standard holding pattern, waiting for further feedback from the reporter, while simultaneously looking closer at the issue, seeing if we ourselves could find a potential exploit. We have of course reached out to the APR project to see if they can help address the pool issue, and are working with them to further investigate possible solutions.

Furthermore, we would like to point out that the 11 year old Bugzilla report referenced deals with mod_php in the mix and did not have the requested stack trace needed for further investigation, and as such, cannot be reliably compared to the current issues. In addition the part of the new report that shows the same symptom as the old Bugzilla report, has already been analyzed and fixed.
Comment 1 Rainer Jung 2019-01-22 12:48:26 UTC
Additional current info from Stefan Sperling about the OpenBSD situation:

FYI, I have disabled pool debugging in OpenBSD's port of APR. We are now using Yann's patch to force the default allocator to call free(3) when APR pools are cleared: https://marc.info/?l=openbsd-ports-cvs&m=154815812713288&w=2

This change only affects OpenBSD -current. I do not plan to backport a patch to the OpenBSD 6.4 release. We have had no reports indicating that http2 was crashing on OpenBSD. The likely reason is that nobody is actually running such a setup. If people intend to run such a setup, they should use -current for now, or wait until OpenBSD 6.5 is released.
Comment 2 hanno 2019-01-22 12:50:44 UTC
As "the reporter" I'd like to add two notes:

* The __pthread_tpp_change_priority definitely shows up for me in apache 2.4.37.

* My worries with the pool debugging are not that people may run it in production. (Even though it seems OpenBSD does that.) My worries are primarily that without pool debugging it's very difficult to detect memory safety bugs, as the pool allocator will hide bugs from tools like ASAN.
Comment 3 Ruediger Pluem 2019-01-22 16:06:26 UTC
(In reply to hanno from comment #2)
> As "the reporter" I'd like to add two notes:
> 
> * The __pthread_tpp_change_priority definitely shows up for me in apache
> 2.4.37.
> 
> * My worries with the pool debugging are not that people may run it in
> production. (Even though it seems OpenBSD does that.) My worries are
> primarily that without pool debugging it's very difficult to detect memory
> safety bugs, as the pool allocator will hide bugs from tools like ASAN.

Try running with

MaxMemFree 1

in httpd.conf instead of using pool debugging. This should give you the same malloc / free behaviour as with pool debugging as needed for ASAN, but uses the non debugging code.
Comment 4 hanno 2019-01-22 16:33:05 UTC
(In reply to Ruediger Pluem from comment #3)
> Try running with
> 
> MaxMemFree 1
> 
> in httpd.conf instead of using pool debugging. This should give you the same
> malloc / free behaviour as with pool debugging as needed for ASAN, but uses
> the non debugging code.

Okay, so this gives some interesting new insights:

* With this "MaxMemFree 1" option set and *without* the pool debugger I now get ASAN uaf warnings at a much faster and more reliable rate than ever before.

* I however don't get the pthread Assertion warning.

As for whether the pthread assertion warning can be reproduced without the pool debugger: Craig confirmed to me that he was able to get that, however with a previous version of apache. We mailed you that in a mail in June.
So for the pthread assertion it may very well be that we're facing two separate issues here: One that is fixed in the latest version and shows up without pool debugging and one that is not fixed, but only happens with pool debugging.
Comment 5 hanno 2019-01-23 10:53:09 UTC
(In reply to hanno from comment #4)
> * With this "MaxMemFree 1" option set and *without* the pool debugger I now
> get ASAN uaf warnings at a much faster and more reliable rate than ever
> before.

I have to retract that statement.
This was a fluke, because I had apr built without the pool debugger, but apr-util was built while the apr pool debugger was still on. This caused the errors to pop up quickly.