Summary: | memory leak with mod_ssl and zlib compression | ||
---|---|---|---|
Product: | Apache httpd-2 | Reporter: | Dustin Kirkland <dustin.kirkland> |
Component: | mod_ssl | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | mchen |
Priority: | P2 | Keywords: | FAQ |
Version: | 2.2.8 | ||
Target Milestone: | --- | ||
Hardware: | PC | ||
OS: | Linux |
Description
Dustin Kirkland
2008-05-12 08:52:53 UTC
Can you check if r654119 (http://svn.apache.org/viewvc?view=rev&revision=654119) fixes your problem? Hi Ruediger- I applied that patch to our Apache 2.2.8 and that gets us halfway there. It solves the Out-of-memory errors and the killing of the Apache processes for consuming all system memory. However, on the client end when running ab, I still see lots of: SSL read failed - closing connection SSL handshake failed (5). It looks like it's just dropping the SSL connections when it gets busy. As recently as Apache 2.2.3, I can run the same ab tests with no problems on either the client (dropping SSL connections) or the server (running out of memory). :-Dustin Actually, I stand corrected. Please disregard my previous post. I did not apply the patch cleanly, and therefore those tests were invalid. I have since cleanly applied the patch, and it does in fact solve the issue. Thank you very much for your prompt response. We should be publishing updated packages for Ubuntu shortly. Thanks, :-Dustin Proposed for backport to 2.2.x as r655982 (http://svn.apache.org/viewvc?view=rev&revision=655982). For Apache 2.0.63 running on Windows 2003, I use the following to perform the test repeatedly. ab -n 10000 -c 20 -f tls1 https://vulnerable.host.example.com:443/ The memory consumed by Apache keep growing and growing. Setting MaxRequestPerThread seems the only way to prevent it from happening... Not sure when we can have a patch or new release for Apache 2.0.x in the near future. This is by design... httpd parks the memory allocations per-thread, and recycles this allocation for the next request. To alter this behavior, see the MaxMemFree directive, and absolutely tune ThreadsPerChild to a number corresponding to your actual max concurrent traffic. (In reply to comment #6) > This is by design... > httpd parks the memory allocations per-thread, and recycles this allocation for > the next request. To alter this behavior, see the MaxMemFree directive, and > absolutely tune ThreadsPerChild to a number corresponding to your actual max > concurrent traffic. I use the same test scenario for Apache2.2.10 but the memory usage is always around 20,000kb. for Apache 2.0.63, it will grow up 490,000KB after I run abs -n 100000 -c 149 -f tls1 https://localhost:443/ about 10 times. Any comment on that? I use the following Apache downloaded from http://httpd.apache.org. apache2.0.63-x86-openssl-0.9.7m.msi apache2.2.10-x86-openssl-0.9.8i.msi mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not backported to 2.0.x. Please use 2.2.x if you experience this problem. (In reply to comment #8) > mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not > backported to 2.0.x. Please use 2.2.x if you experience this problem. > Sorry I got confused by the zlib in the subject and read mod_deflate instead of mod_ssl. Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x. So please use 2.2.9 and up if you are affected by this issue. (In reply to comment #9) > Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x. Must be 2.0.x instead of 2.2.x of course. (In reply to comment #8) > mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not > backported to 2.0.x. Please use 2.2.x if you experience this problem. I don't think this issue is related to mod_deflate. If I use ab instead abs and run the following test 10 times: ab -n 100000 -c 149 http://localhost:80/ The memory consumed by Apache will grow from 10,064KB and then is stablized to around 30,904KB. It means that it only happens on https which might be related to mod_ssl... (In reply to comment #10) > (In reply to comment #9) > > Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x. > Must be 2.0.x instead of 2.2.x of course. Yes, I am hopping that similar patch in comment #1 should be backported to 2.0.x. I know that the patch is backported to 2.2.x since I cannot reproduce the problem on 2.2.10. (In reply to comment #9) > (In reply to comment #8) > > mod_deflate has a memory leak in 2.0.x. This is fixed since 2.2.4 but was not > > backported to 2.0.x. Please use 2.2.x if you experience this problem. > > > Sorry I got confused by the zlib in the subject and read mod_deflate instead of > mod_ssl. > Nevertheless the patch in comment #1 was backported to 2.2.9 but not to 2.2.x. > So please use 2.2.9 and up if you are affected by this issue. I cannot use 2.2.x... In fact, we are using apache 2.0.x as reverse proxy and applied some patches to enable features like IPV6. We have to continue to use Apache 2.0.x for a while... I just search google with the keyword "Apache 2.0.59 mod_ssl memory leak" and found that it is not just me who encountered this issue. http://mail-archives.apache.org/mod_mbox/httpd-users/200709.mbox/%3C1E5DDB0AD7F8FA4EA12097917C2FD18D027A0A1E06@BLRKECMBX04.ad.infosys.com%3E In fact, it is our customer who reported this issue and ask for a solution. So far, it seems that the only solution for now is to set MaxRequestPerThread to non zero... Try backporting the patch to 2.0.x by yourself. The patch itself is quite simple: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ssl/mod_ssl.c?r1=654119&r2=654118&pathrev=654119&view=patch (In reply to comment #15) > Try backporting the patch to 2.0.x by yourself. The patch itself is quite > simple: > http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ssl/mod_ssl.c?r1=654119&r2=654118&pathrev=654119&view=patch Yes, the patch itself is quite simple if you diff the file between apache 2.2.8 and apache 2.2.10. If you diff mod_ssl.c between Apache 2.0.63 and apache 2.2.10, there are quite a few differences between them. Can I simply copy mod_ssl.c from apache 2.2.10 over mod_ssl.c from Apache 2.0.63 and expect it to work? BTW, I am not even sure if they are the same problem. I just know that when I run abs -n 100000 -c 149 -f tls1 https://localhost:443 10 times, Apache will consume around 490,000KB. It will increase about 51MB each time I run it. When I run ab -n 100000 -c 149 http://localhost:80, I do not see the issue. Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade entirely to apache 2.2 or hold on for the patch. There's no promise to ever ship 2.0.64, probably only if a vulnerability is discovered, but I plan to backport this patch, and will look at attaching that backported patch to this incident. (In reply to comment #17) > Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade > entirely to apache 2.2 or hold on for the patch. > There's no promise to ever ship 2.0.64, probably only if a vulnerability is > discovered, but I plan to backport this patch, and will look at attaching > that backported patch to this incident. CVE-2008-1678 seems to be the cause of this bug... I hope this justifies a new release,e.g. 2.0.64... (In reply to comment #17) > Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade > entirely to apache 2.2 or hold on for the patch. > There's no promise to ever ship 2.0.64, probably only if a vulnerability is > discovered, but I plan to backport this patch, and will look at attaching > that backported patch to this incident. Our customer is pushing us for a solution. Do you know any work around that can prevent the problem? If so, that will be great. (In reply to comment #17) > Minor revisions (2.0 vs 2.2 vs 2.4) are not binary compatible, you must upgrade > entirely to apache 2.2 or hold on for the patch. > There's no promise to ever ship 2.0.64, probably only if a vulnerability is > discovered, but I plan to backport this patch, and will look at attaching > that backported patch to this incident. Any rough estimate when the patch will be available? ERR_free_strings and CRYPTO_cleanup_all_ex_data are never called by mod_ssl in the httpd 2.0.x tree now. Apparently they might never have been. Therefore this ticket doesn't apply at all to 2.0. If you have memory issues they are elsewhere in the code, see similar tickets. Also note that not much attention was paid in 2.0 to OpenSSL 0.9.8 as 0.9.7 was current. You really must upgrade your software in sync, you can't expect project maintainers to be in sync with every flavor of dependent library releases, especially ancient and future ones. Suggestions; revert to OpenSSL 0.9.7, or upgrade to httpd 2.2.x. Closing (In reply to comment #21) > ERR_free_strings and CRYPTO_cleanup_all_ex_data are never called by mod_ssl > in the httpd 2.0.x tree now. Apparently they might never have been. > Therefore this ticket doesn't apply at all to 2.0. If you have memory issues > they are elsewhere in the code, see similar tickets. Also note that not much > attention was paid in 2.0 to OpenSSL 0.9.8 as 0.9.7 was current. You really > must upgrade your software in sync, you can't expect project maintainers to > be in sync with every flavor of dependent library releases, especially ancient > and future ones. > Suggestions; revert to OpenSSL 0.9.7, or upgrade to httpd 2.2.x. > Closing I feel puzzled about your comment: Suggestions; revert to OpenSSL 0.9.7, or upgrade to httpd 2.2.x. I download the following installer from Apache web site: apache2.0.63-x86-openssl-0.9.7m.msi It uses OpenSSL 0.9.7 alreay and still has the same problem. The summary field for this bug is "memory leak with mod_ssl and zlib compression" and I can use the same test sequence to reproduce the problem, e.g. abs -n 100000 -c 149 -f tls1 https://localhost:443/. Should I report it in a seperate bug? Dustin's bug report can be better understood here; https://bugzilla.redhat.com/show_bug.cgi?id=447268 Michael, you either are looking at a leak in mod_ssl/openssl or mod_deflate/zlib but not the combination which existed in 0.9.8e until 2.2.10. Please don't file a bug report until we know which bug (probably already reported) your problem corresponds to. Could I ask you to go ahead and try reproducing the problem with only SSL or mod_deflate enabled to narrow down which of these two modules is faulty? Go ahead and stick with the 0.9.7 and apache 2.0 flavor. Then we'll look at 2.2 patches for the corresponding module. But your issue is not the one documented in this PR. (In reply to comment #23) > Dustin's bug report can be better understood here; > https://bugzilla.redhat.com/show_bug.cgi?id=447268 > Michael, you either are looking at a leak in mod_ssl/openssl or > mod_deflate/zlib > but not the combination which existed in 0.9.8e until 2.2.10. > Please don't file a bug report until we know which bug (probably already > reported) your problem corresponds to. Could I ask you to go ahead and try > reproducing the problem with only SSL or mod_deflate enabled to narrow down > which of these two modules is faulty? Go ahead and stick with the 0.9.7 and > apache 2.0 flavor. Then we'll look at 2.2 patches for the corresponding > module. > But your issue is not the one documented in this PR. We do not use mod_deflate. I also installed apache2.0.63-x86-openssl-0.9.7m.msi, and mod_deflate is not used at all. Hopefully this helps. I'm not sure where the claim that this doesn't affect OpenSSL 0.9.7 comes from. My analysis of this last year was: "" Having done some more testing, it looks like the current behaviour in OpenSSL was changed here: http://cvs.openssl.org/chngview?cn=15897 so the issue can only be triggered with OpenSSL versions >= 0.9.7f. "" My mistake, after double-checking this the OpenSSL code in question appears to be in 0.9.8e and later. Another known memory leak in mod_ssl is the dbm session cache. (In reply to comment #26) > My mistake, after double-checking this the OpenSSL code in question appears to > be in 0.9.8e and later. > Another known memory leak in mod_ssl is the dbm session cache. I've done some more investigation by taking the source code from 2.2.10 and build it and use our current httpd.conf from 2.0.x. To my surprise, I can still reproduce the problem with the source code in 2.2.10. If download 2.2.10 installer and use that, the problem does not exist. I then do a diff and migrate the change one by one using httpd.conf from 2.2.10 to the one from 2.0.x and identify the source of the problem. This is the settings in httpd-ssl.conf from Apache 2.2.10. # to use and second the expiring timeout (in seconds). #SSLSessionCache "dbm:c:/cache/2210-2/out_nti86/iw-webd/logs/ssl_scache" SSLSessionCache "shmcb:c:/cache/2210-2/out_nti86/iw-webd/logs/ssl_scache(512000)" SSLSessionCacheTimeout 300 This is the settings in ssl.conf from apache 2.0.x. #SSLSessionCache none #SSLSessionCache shmht:logs/ssl_scache(512000) #SSLSessionCache shmcb:logs/ssl_scache(512000) SSLSessionCache dbm:logs/ssl_scache It means that in Apache 2.2.x, shmcb/path/to/cache is used while in Apache 2.0.x, dbm/path/to/cache is used which causes the problem. In Apache 2.2.10, if I switch to use dbm/path/to/cache, I can reproduce the problem. In Apache 2.0.x, if I switch to use /shm/path/to/path, the problem goes away. After I found this, I did a search to see if there is any mod_ssl/SSLSessionCache issue which causes the memory memory leak and I did find it at https://issues.apache.org/bugzilla/show_bug.cgi?id=25667 I am not sure why bug 25667 is left open and the solution is just to fix the configuration file and update the doc so that user using apache 2.0.x will be aware of that? |