Bug 44975

Summary: memory leak with mod_ssl and zlib compression
Product: Apache httpd-2 Reporter: Dustin Kirkland <dustin.kirkland>
Component: mod_sslAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: normal CC: mchen
Priority: P2 Keywords: FAQ
Version: 2.2.8   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description Dustin Kirkland 2008-05-12 08:52:53 UTC
We have encountered an easily reproducible memory leak with Apache2 + mod_ssl with zlib encryption enabled.

Reproducing the problem is as simple as running Apache-bench against a vulnerable host:
  ab -n 10000 -c 20 -f tls1 https://vulnerable.host.example.com:443/

Vulnerable hosts seem to be (apache >= 2.2.4) + (openssl >= 0.9.8e).

Depending on how much memory is available on the server, you may need to scale the value of -n up or down.  With 128MB in a virtual machine -n 1000 is enough to manifest the problem.

On the client side, you will begin seeing:
  SSL handshake failed (5).
  SSL read failed - closing connection

On the server side under Linux, the kernel Out-of-memory (OOM) killer starts reaping runaway Apache2 instances.
  Out of memory: kill process XXXX (apache2) 

This bug is being tracked in Ubuntu's bug tracker here:
  https://bugs.edge.launchpad.net/ubuntu/+source/apache2/+bug/224945

As this looks to be an issue with OpenSSL, it has been reported there as well:
  http://marc.info/?l=openssl-dev&m=121060672602371&w=2

:-Dustin
Comment 1 Ruediger Pluem 2008-05-12 13:24:24 UTC
Can you check if r654119 (http://svn.apache.org/viewvc?view=rev&revision=654119) fixes your problem?
Comment 2 Dustin Kirkland 2008-05-12 14:44:50 UTC
Hi Ruediger-

I applied that patch to our Apache 2.2.8 and that gets us halfway there.

It solves the Out-of-memory errors and the killing of the Apache processes for consuming all system memory.

However, on the client end when running ab, I still see lots of:
SSL read failed - closing connection
SSL handshake failed (5).

It looks like it's just dropping the SSL connections when it gets busy.

As recently as Apache 2.2.3, I can run the same ab tests with no problems on either the client (dropping SSL connections) or the server (running out of memory).

:-Dustin
Comment 3 Dustin Kirkland 2008-05-13 07:34:45 UTC
Actually, I stand corrected.  Please disregard my previous post.  I did not apply the patch cleanly, and therefore those tests were invalid.

I have since cleanly applied the patch, and it does in fact solve the issue.

Thank you very much for your prompt response.  We should be publishing updated packages for Ubuntu shortly.

Thanks,
:-Dustin
Comment 4 Ruediger Pluem 2008-05-13 12:31:34 UTC
Proposed for backport to 2.2.x as r655982 (http://svn.apache.org/viewvc?view=rev&revision=655982).
Comment 5 Michael Chen 2008-11-18 15:25:58 UTC
For Apache 2.0.63 running on Windows 2003, I use the following to perform the test repeatedly.
ab -n 10000 -c 20 -f tls1 https://vulnerable.host.example.com:443/

The memory consumed by Apache keep growing and growing. Setting MaxRequestPerThread seems the only way to prevent it from happening...

Not sure when we can have a patch or new release for Apache 2.0.x in the near future.
Comment 6 William A. Rowe Jr. 2008-11-18 15:34:01 UTC
This is by design...

httpd parks the memory allocations per-thread, and recycles this allocation for 
the next request.  To alter this behavior, see the MaxMemFree directive, and
absolutely tune ThreadsPerChild to a number corresponding to your actual max
concurrent traffic.
Comment 7 Michael Chen 2008-11-18 16:35:30 UTC
(In reply to comment #6)
> This is by design...
> httpd parks the memory allocations per-thread, and recycles this allocation for 
> the next request.  To alter this behavior, see the MaxMemFree directive, and
> absolutely tune ThreadsPerChild to a number corresponding to your actual max
> concurrent traffic.

I use the same test scenario for Apache2.2.10 but the memory usage is always around 20,000kb. for Apache 2.0.63, it will grow up 490,000KB after I run 
abs -n 100000 -c 149 -f tls1 https://localhost:443/ about 10 times. Any comment on that?


I use the following Apache downloaded from http://httpd.apache.org.
apache2.0.63-x86-openssl-0.9.7m.msi
apache2.2.10-x86-openssl-0.9.8i.msi

Comment 8 Ruediger Pluem 2008-11-19 00:55:02 UTC
mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not backported to 2.0.x. Please use 2.2.x if you experience this problem.
Comment 9 Ruediger Pluem 2008-11-19 01:09:32 UTC
(In reply to comment #8)
> mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not
> backported to 2.0.x. Please use 2.2.x if you experience this problem.
> 

Sorry I got confused by the zlib in the subject and read mod_deflate instead of mod_ssl.
Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x.
So please use 2.2.9 and up if you are affected by this issue.
Comment 10 Ruediger Pluem 2008-11-19 01:11:44 UTC
(In reply to comment #9)

> Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x.

Must be 2.0.x instead of 2.2.x of course.

Comment 11 Michael Chen 2008-11-19 10:43:17 UTC
(In reply to comment #8)
> mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not
> backported to 2.0.x. Please use 2.2.x if you experience this problem.

I don't think this issue is related to mod_deflate.
If I use ab instead abs and run the following test 10 times:
ab -n 100000 -c 149 http://localhost:80/

The memory consumed by Apache will grow from 10,064KB and then is stablized to around 30,904KB.

It means that it only happens on https which might be related to mod_ssl...
Comment 12 Michael Chen 2008-11-19 10:47:36 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x.
> Must be 2.0.x instead of 2.2.x of course.

Yes, I am hopping that similar patch in comment #1 should be backported to 2.0.x. I know that the patch is backported to 2.2.x since I cannot reproduce the problem on 2.2.10.
Comment 13 Michael Chen 2008-11-19 10:50:51 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not
> > backported to 2.0.x. Please use 2.2.x if you experience this problem.
> > 
> Sorry I got confused by the zlib in the subject and read mod_deflate instead of
> mod_ssl.
> Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x.
> So please use 2.2.9 and up if you are affected by this issue.

I cannot use 2.2.x... In fact, we are using apache 2.0.x as reverse proxy and applied some patches to enable features like IPV6. We have to continue to use Apache 2.0.x for a while...
Comment 14 Michael Chen 2008-11-19 10:56:05 UTC
I just search google with the keyword "Apache 2.0.59 mod_ssl memory leak" and found that it is not just me who encountered this issue.

http://mail-archives.apache.org/mod_mbox/httpd-users/200709.mbox/%3C1E5DDB0AD7F8FA4EA12097917C2FD18D027A0A1E06@BLRKECMBX04.ad.infosys.com%3E

In fact, it is our customer who reported this issue and ask for a solution. So far, it seems that the only solution for now is to set MaxRequestPerThread to non zero...
Comment 15 Ruediger Pluem 2008-11-19 12:51:06 UTC
Try backporting the patch to 2.0.x by yourself. The patch itself is quite simple:
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ssl/mod_ssl.c?r1=654119&r2=654118&pathrev=654119&view=patch
Comment 16 Michael Chen 2008-11-19 14:53:51 UTC
(In reply to comment #15)
> Try backporting the patch to 2.0.x by yourself. The patch itself is quite
> simple:
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ssl/mod_ssl.c?r1=654119&r2=654118&pathrev=654119&view=patch

Yes, the patch itself is quite simple if you diff the file between apache 2.2.8 and apache 2.2.10. If you diff mod_ssl.c between Apache 2.0.63 and apache 2.2.10, there are quite a few differences between them.

Can I simply copy mod_ssl.c from apache 2.2.10 over mod_ssl.c from Apache 2.0.63 and expect it to work?

BTW, I am not even sure if they are the same problem. I just know that when I run abs -n 100000 -c 149 -f tls1 https://localhost:443 10 times, Apache will consume around 490,000KB. It will increase about 51MB each time I run it.

When I run ab -n 100000 -c 149 http://localhost:80, I do not see the issue.
Comment 17 William A. Rowe Jr. 2008-11-19 15:10:49 UTC
Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade
entirely to apache 2.2 or hold on for the patch.

There's no promise to ever ship 2.0.64, probably only if a vulnerability is
discovered, but I plan to backport this patch, and will look at attaching
that backported patch to this incident.
Comment 18 Michael Chen 2008-11-20 11:19:30 UTC
(In reply to comment #17)
> Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade
> entirely to apache 2.2 or hold on for the patch.
> There's no promise to ever ship 2.0.64, probably only if a vulnerability is
> discovered, but I plan to backport this patch, and will look at attaching
> that backported patch to this incident.

CVE-2008-1678 seems to be the cause of this bug... I hope this justifies a new release,e.g. 2.0.64...
Comment 19 Michael Chen 2008-11-24 14:24:53 UTC
(In reply to comment #17)
> Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade
> entirely to apache 2.2 or hold on for the patch.
> There's no promise to ever ship 2.0.64, probably only if a vulnerability is
> discovered, but I plan to backport this patch, and will look at attaching
> that backported patch to this incident.

Our customer is pushing us for a solution. Do you know any work around that can prevent the problem? If so, that will be great.
Comment 20 Michael Chen 2009-01-13 05:56:28 UTC
(In reply to comment #17)
> Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade
> entirely to apache 2.2 or hold on for the patch.
> There's no promise to ever ship 2.0.64, probably only if a vulnerability is
> discovered, but I plan to backport this patch, and will look at attaching
> that backported patch to this incident.

Any rough estimate when the patch will be available?
Comment 21 William A. Rowe Jr. 2009-01-13 10:40:23 UTC
ERR_free_strings and CRYPTO_cleanup_all_ex_data are never called by mod_ssl
in the httpd 2.0.x tree now.  Apparently they might never have been.

Therefore this ticket doesn't apply at all to 2.0.  If you have memory issues
they are elsewhere in the code, see similar tickets.  Also note that not much
attention was paid in 2.0 to OpenSSL 0.9.8 as 0.9.7 was current.  You really
must upgrade your software in sync, you can't expect project maintainers to
be in sync with every flavor of dependent library releases, especially ancient
and future ones.

Suggestions; revert to OpenSSL 0.9.7, or upgrade to httpd 2.2.x.

Closing
Comment 22 Michael Chen 2009-01-15 11:48:51 UTC
(In reply to comment #21)
> ERR_free_strings and CRYPTO_cleanup_all_ex_data are never called by mod_ssl
> in the httpd 2.0.x tree now.  Apparently they might never have been.
> Therefore this ticket doesn't apply at all to 2.0.  If you have memory issues
> they are elsewhere in the code, see similar tickets.  Also note that not much
> attention was paid in 2.0 to OpenSSL 0.9.8 as 0.9.7 was current.  You really
> must upgrade your software in sync, you can't expect project maintainers to
> be in sync with every flavor of dependent library releases, especially ancient
> and future ones.
> Suggestions; revert to OpenSSL 0.9.7, or upgrade to httpd 2.2.x.
> Closing

I feel puzzled about your comment:
Suggestions; revert to OpenSSL 0.9.7, or upgrade to httpd 2.2.x.

I download the following installer from Apache web site:
apache2.0.63-x86-openssl-0.9.7m.msi

It uses OpenSSL 0.9.7 alreay and still has the same problem. The summary field for this bug is "memory leak with mod_ssl and zlib compression" and I can use the same test sequence to reproduce the problem, e.g. abs -n 100000 -c 149 -f tls1 https://localhost:443/.

Should I report it in a seperate bug?


Comment 23 William A. Rowe Jr. 2009-01-15 12:16:38 UTC
Dustin's bug report can be better understood here;

https://bugzilla.redhat.com/show_bug.cgi?id=447268

Michael, you either are looking at a leak in mod_ssl/openssl or mod_deflate/zlib 
but not the combination which existed in 0.9.8e until 2.2.10.

Please don't file a bug report until we know which bug (probably already
reported) your problem corresponds to.  Could I ask you to go ahead and try
reproducing the problem with only SSL or mod_deflate enabled to narrow down
which of these two modules is faulty?  Go ahead and stick with the 0.9.7 and
apache 2.0 flavor.  Then we'll look at 2.2 patches for the corresponding module.

But your issue is not the one documented in this PR. 
Comment 24 Michael Chen 2009-01-15 16:05:24 UTC
(In reply to comment #23)
> Dustin's bug report can be better understood here;
> https://bugzilla.redhat.com/show_bug.cgi?id=447268
> Michael, you either are looking at a leak in mod_ssl/openssl or
> mod_deflate/zlib 
> but not the combination which existed in 0.9.8e until 2.2.10.
> Please don't file a bug report until we know which bug (probably already
> reported) your problem corresponds to.  Could I ask you to go ahead and try
> reproducing the problem with only SSL or mod_deflate enabled to narrow down
> which of these two modules is faulty?  Go ahead and stick with the 0.9.7 and
> apache 2.0 flavor.  Then we'll look at 2.2 patches for the corresponding
> module.
> But your issue is not the one documented in this PR. 

We do not use mod_deflate. I also installed apache2.0.63-x86-openssl-0.9.7m.msi, and mod_deflate is not used at all. 

Hopefully this helps.

Comment 25 Joe Orton 2009-01-16 02:01:34 UTC
I'm not sure where the claim that this doesn't affect OpenSSL 0.9.7 comes from.  My analysis of this last year was:

""
Having done some more testing, it looks like the current behaviour in
OpenSSL was changed here:

  http://cvs.openssl.org/chngview?cn=15897

so the issue can only be triggered with OpenSSL versions >= 0.9.7f.
""
Comment 26 Joe Orton 2009-01-16 06:14:47 UTC
My mistake, after double-checking this the OpenSSL code in question appears to be in 0.9.8e and later.

Another known memory leak in mod_ssl is the dbm session cache. 
Comment 27 Michael Chen 2009-01-21 14:55:42 UTC
(In reply to comment #26)
> My mistake, after double-checking this the OpenSSL code in question appears to
> be in 0.9.8e and later.
> Another known memory leak in mod_ssl is the dbm session cache. 

I've done some more investigation by taking the source code from 2.2.10 and build it and use our current httpd.conf from 2.0.x. To my surprise, I can still reproduce the problem with the source code in 2.2.10. If download 2.2.10 installer and use that, the problem does not exist.

I then do a diff and migrate the change one by one using httpd.conf from 2.2.10 to the one from 2.0.x and identify the source of the problem.

This is the settings in httpd-ssl.conf from Apache 2.2.10.
#   to use and second the expiring timeout (in seconds).
#SSLSessionCache         "dbm:c:/cache/2210-2/out_nti86/iw-webd/logs/ssl_scache"
SSLSessionCache        "shmcb:c:/cache/2210-2/out_nti86/iw-webd/logs/ssl_scache(512000)"
SSLSessionCacheTimeout  300

This is the settings in ssl.conf from apache 2.0.x.
#SSLSessionCache        none
#SSLSessionCache        shmht:logs/ssl_scache(512000)
#SSLSessionCache        shmcb:logs/ssl_scache(512000)
SSLSessionCache         dbm:logs/ssl_scache

It means that in Apache 2.2.x, shmcb/path/to/cache is used while in Apache 2.0.x, dbm/path/to/cache is used which causes the problem.

In Apache 2.2.10, if I switch to use dbm/path/to/cache, I can reproduce the problem. In Apache 2.0.x, if I switch to use /shm/path/to/path, the problem goes away.

After I found this, I did a search to see if there is any mod_ssl/SSLSessionCache issue which causes the memory memory leak and I did find it at https://issues.apache.org/bugzilla/show_bug.cgi?id=25667

I am not sure why bug 25667 is left open and the solution is just to fix the configuration file and update the doc so that user using apache 2.0.x will be aware of that?