I posted two patches to fix memory leaks in issues 25659 and 25667.  I don't 
know if they are directly related to what you are seeing but I haven't received 
any feedback on whether anyone else has tried them out.  Let me knoe if you 
think they may be related.

David,

I tried both of the bug fixes you posted (25659, 25667) and they did not help, 
I suspect the problem is related to the way threads handle memory verses 
forked processes.

Ken

FYI - I am seeing multiples of 8K chuncks of memroy disapear at a time. I 
don't know if this helps?

That would most likely be a heap bucket not being freed somewhere.

I'm still unable to reproduce this on Linux.  Can you post the mod_ssl
configuration you're using?

Here is the ssl.conf file:

#
# This is the Apache server configuration file providing SSL support.
# It contains the configuration directives to instruct the server how to
# serve pages over an https connection. For detailing information about these 
# directives see <URL:http://httpd.apache.org/docs-2.0/mod/mod_ssl.html>
#
#   For the moment, see <URL:http://www.modssl.org/docs/> for this info. 
#   The documents are still being prepared from material donated by the
#   modssl project.
# 
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
<IfDefine SSL>

#   Until documentation is completed, please check http://www.modssl.org/
#   for additional config examples and module docmentation.  Directives
#   and features of mod_ssl are largely unchanged from the mod_ssl project
#   for Apache 1.3.

#
# When we also provide SSL we have to listen to the 
# standard HTTP port (see above) and to the HTTPS port
#
# Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
#       Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443"
#
Listen 443

##
##  SSL Global Context
##
##  All SSL configuration in this context applies both to
##  the main server and all SSL-enabled virtual hosts.
##

#
#   Some MIME-types for downloading Certificates and CRLs
#
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

#   Pass Phrase Dialog:
#   Configure the pass phrase gathering process.
#   The filtering dialog program (`builtin' is a internal
#   terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog  builtin

#   Inter-Process Session Cache:
#   Configure the SSL Session Cache: First the mechanism 
#   to use and second the expiring timeout (in seconds).
#SSLSessionCache        none
#SSLSessionCache        shmht:logs/ssl_scache(512000)
#SSLSessionCache        shmcb:logs/ssl_scache(512000)
SSLSessionCache         dbm:logs/ssl_scache
SSLSessionCacheTimeout  300

#   Semaphore:
#   Configure the path to the mutual exclusion semaphore the
#   SSL engine uses internally for inter-process synchronization. 
SSLMutex  file:logs/ssl_mutex

#   Pseudo Random Number Generator (PRNG):
#   Configure one or more sources to seed the PRNG of the 
#   SSL library. The seed data should be of good random quality.
#   WARNING! On some platforms /dev/random blocks if not enough entropy
#   is available. This means you then cannot use the /dev/random device
#   because it would lead to very long connection times (as long as
#   it requires to make more entropy available). But usually those
#   platforms additionally provide a /dev/urandom device which doesn't
#   block. So, if available, use this one instead. Read the mod_ssl User
#   Manual for more details.
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
#SSLRandomSeed startup file:/dev/random  512
#SSLRandomSeed startup file:/dev/urandom 512
#SSLRandomSeed connect file:/dev/random  512
#SSLRandomSeed connect file:/dev/urandom 512

##
## SSL Virtual Host Context
##

<VirtualHost _default_:443>

#  General setup for the virtual host
DocumentRoot "/Apache2/htdocs"
ServerName new.host.name:443
ServerAdmin you@your.address
ErrorLog logs/error_log
TransferLog logs/access_log

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   SSL Cipher Suite:
#   List the ciphers that the client is permitted to negotiate.
#   See the mod_ssl documentation for a complete list.
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

#   Server Certificate:
#   Point SSLCertificateFile at a PEM encoded certificate.  If
#   the certificate is encrypted, then you will be prompted for a
#   pass phrase.  Note that a kill -HUP will prompt again.  Keep
#   in mind that if you have both an RSA and a DSA certificate you
#   can configure both in parallel (to also allow the use of DSA
#   ciphers, etc.)
SSLCertificateFile /Apache2/conf/ssl.crt/server.crt
#SSLCertificateFile /Apache2/conf/ssl.crt/server-dsa.crt

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile /Apache2/conf/ssl.key/server.key
#SSLCertificateKeyFile /Apache2/conf/ssl.key/server-dsa.key

#   Server Certificate Chain:
#   Point SSLCertificateChainFile at a file containing the
#   concatenation of PEM encoded CA certificates which form the
#   certificate chain for the server certificate. Alternatively
#   the referenced file can be the same as SSLCertificateFile
#   when the CA certificates are directly appended to the server
#   certificate for convinience.
#SSLCertificateChainFile /Apache2/conf/ssl.crt/ca.crt

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#   Note: Inside SSLCACertificatePath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
#SSLCACertificatePath /Apache2/conf/ssl.crt
#SSLCACertificateFile /Apache2/conf/ssl.crt/ca-bundle.crt

#   Certificate Revocation Lists (CRL):
#   Set the CA revocation path where to find CA CRLs for client
#   authentication or alternatively one huge file containing all
#   of them (file must be PEM encoded)
#   Note: Inside SSLCARevocationPath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
#SSLCARevocationPath /Apache2/conf/ssl.crl
#SSLCARevocationFile /Apache2/conf/ssl.crl/ca-bundle.crl

#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth  10

#   Access Control:
#   With SSLRequire you can do per-directory access control based
#   on arbitrary complex boolean expressions containing server
#   variable checks and other lookup directives.  The syntax is a
#   mixture between C and Perl.  See the mod_ssl documentation
#   for more details.
#<Location />
#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
#</Location>

#   SSL Engine Options:
#   Set various options for the SSL engine.
#   o FakeBasicAuth:
#     Translate the client X.509 into a Basic Authorisation.  This means that
#     the standard Auth/DBMAuth methods can be used for access control.  The
#     user name is the `one line' version of the client's X.509 certificate.
#     Note that no password is obtained from the user. Every entry in the user
#     file needs this password: `xxj31ZMTZzkVA'.
#   o ExportCertData:
#     This exports two additional environment variables: SSL_CLIENT_CERT and
#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
#     server (always existing) and the client (only existing when client
#     authentication is used). This can be used to import the certificates
#     into CGI scripts.
#   o StdEnvVars:
#     This exports the standard SSL/TLS related `SSL_*' environment variables.
#     Per default this exportation is switched off for performance reasons,
#     because the extraction step is an expensive operation and is usually
#     useless for serving static content. So one usually enables the
#     exportation for CGI and SSI requests only.
#   o CompatEnvVars:
#     This exports obsolete environment variables for backward compatibility
#     to Apache-SSL 1.x, mod_ssl 2.0.x, Sioux 1.0 and Stronghold 2.x. Use this
#     to provide compatibility to existing CGI scripts.
#   o StrictRequire:
#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
#     under a "Satisfy any" situation, i.e. when it applies access is denied
#     and no other module can change it.
#   o OptRenegotiate:
#     This enables optimized SSL connection renegotiation handling when SSL
#     directives are used in per-directory context. 
#SSLOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/Apache2/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait for
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is send or allowed to received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
#   o ssl-accurate-shutdown:
#     This forces an accurate shutdown when the connection is closed, i.e. a
#     SSL close notify alert is send and mod_ssl waits for the close notify
#     alert of the client. This is 100% SSL/TLS standard compliant, but in
#     practice often causes hanging connections with brain-dead browsers. Use
#     this only for browsers where you know that their SSL implementation
#     works correctly. 
#   Notice: Most problems of broken clients are also related to the HTTP
#   keep-alive facility, so you usually additionally want to disable
#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
#   "force-response-1.0" for this.
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

#   Per-Server Logging:
#   The home of a custom SSL log file. Use this when you want a
#   compact non-error SSL logfile on a virtual host basis.
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>                                  

</IfDefine>

BTW - Check your log file to see if there is a SIGNAL 11 SEGFAULT that will 
cause Apache to restart the threads and make it appear that everything is OK.

Here is some more info from on our Linux Heads:

While attempting to locate the cause for what appears to be a memory 
consumption problem in the SSL code, the server segmentation faults. The first 
worker child & all of its child threads continue to consume memory while the 
parent stays the same or gets a little smaller.  The child threads never give 
the memory back unless restarted.  Please advise if this is an expected 
behavior.

Running with 'SSLSessionCache none' doesn't consume memory (and doesn't seg 
fault), but it performs poorly when using 2048 bit keys.

I observed the segmentation fault issue in mod_ssl while running the small 
script listed below.  Based on the stack information the issue appears to be 
in shmcb_cton_memcpy() during an attempt to remove a session id.  The server 
keeps on reponding, but all the child threads die and are restarted. I am not 
sure what is happening, but the following variables seem to get corrupted:

The stack trace shows these are supposed to be:

src_offset=6402 
src_len=10240

Inside the frame they have these values:

(gdb) print src_offset (in edi register)
$55 = 3183473748  
(gdb) print src_len    (in edx register) 
$56 = 3183464512

The configuration file, and my initial debug session are attached.

Apache error_log
...
[Mon Mar 15 11:21:33 2004] [notice] Apache/2.0.48 configured -- resuming 
normal operations [Mon Mar 15 11:25:28 2004] [error] server reached MaxClients 
setting, consider raising the MaxClients setting [Mon Mar 15 11:38:29 2004] 
[notice] child pid 1065 exit signal Segmentation fault (11) [Mon Mar 15 
12:06:28 2004] [notice] child pid 1154 exit signal Segmentation fault (11) 
[Mon Mar 15 12:44:49 2004] [notice] child pid 1258 exit signal Segmentation 
fault (11) [Mon Mar 15 13:04:40 2004] [notice] child pid 1315 exit signal 
Segmentation fault (11) [Mon Mar 15 13:17:29 2004] [notice] child pid 1363 
exit signal Segmentation fault (11) [Mon Mar 15 13:45:12 2004] [notice] child 
pid 1401 exit signal Segmentation fault (11) ...

OS RedHat 7.3 

gcc-2.96-113
glibc-2.2.5-43
openssl-0.9.6b-35.7

Apache 2.0.48 Build Script:

./configure  --with-program-name=leakd --with-port=9200 --with-mpm=worker --
enable-ssl=shared --enable-maintainer-mode \ --enable-proxy=shared --enable-
cgi=shared --enable-setenvif=shared --enable-cgi=shared --enable-access=shared 
\ --enable-rewrite=shared --enable-dir=shared --enable-actions=shared --enable-
mime=shared --enable-proxy_connect=shared \ --enable-proxy_http=shared --
enable-negotiation=shared --enable-alias=shared --enable-env=shared --enable-
dir=shared \ --enable-mod-actions=shared --enable-log-config=shared --enable-
imap=shared --enable-headers=shared \ --enable-layout=webserver --disable-
autoindex --disable-userdir --disable-usertrack --disable-cgid \ --disable-
asis --disable-auth --disable-auth_digest --disable-auth_dbm --disable-
auth_anon --disable-dav \ --disable-dav_fs --disable-vhost_alias --disable-
unique_id --disable-speling --disable-cern_meta --disable-include \ --disable-
expires --enable-status=shared --enable-info=shared

ldd leakd:

        libssl.so.2 => /lib/libssl.so.2 (0x40024000)
        libcrypto.so.2 => /lib/libcrypto.so.2 (0x40052000)
        libaprutil-0.so.0 => /usr/webserver/lib/libaprutil-0.so.0 (0x40119000)
        libgdbm.so.2 => /usr/lib/libgdbm.so.2 (0x4012d000)
        libdb-3.3.so => /lib/libdb-3.3.so (0x40133000)
        libexpat.so.0 => /usr/lib/libexpat.so.0 (0x401c2000)
        libapr-0.so.0 => /usr/webserver/lib/libapr-0.so.0 (0x401e1000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x40200000)
        librt.so.1 => /lib/librt.so.1 (0x40215000)
        libm.so.6 => /lib/libm.so.6 (0x40226000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x40247000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x40274000)
        libdl.so.2 => /lib/libdl.so.2 (0x40288000)
        libc.so.6 => /lib/libc.so.6 (0x4028c000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)


Simple script on external machine downloads copies of the stock Apache 
index.html.en page under both unsecure & secure sites:

#!/bin/sh
counter=0
limit=32000
while [ "$counter" -lt "$limit" ]
do
  wget -O - http://myboxaddr:9200
  wget -O - https://myboxaddr:9201
  counter=`expr $counter + 1`
  echo "Count=> $counter"
done

Hmmm, haven't seen shmcb segfaults in a while.  Tried any different SSLMutex
settings, e.g. "SSLMutex default"?  Can you file a separate bug on that, and
include the backtrace for the shmcb segfaults.

After tagging memory and running tests over the weekend it appears that calls 
to OPENSSL_malloc in the following files failed to release memory by calling 
the corresponding OPENSSL_free:

bn_bind.c - 494,788 outstanding OPENSSL_malloc
bn_lib.c  - 123,673 outstanding OPENSSL_malloc

I am not sure how this relates to the Apache/mod_ssl threaded MPMs; though, it 
does appear to be a problem.

Any ideas?

I have narrowed it down to the function BN_BLINDING_new in the file 
crypto\bn\bn_blind.c, the memory allocated for the BN_BLINDING structure never 
gets freed. I am assuming that the BIGNUM structures allocated with BN_new 
inside of BN_BLINDING never gets freed also.

Here are my test results after running 24 hours monitoring the OPENSSL_malloc 
and OPENSSL_free calls:

1. BN_BLINDING – allocations 53,615, frees 0, outstanding 53,615 
2. BN_new – allocations 8,347,200, frees 8,127,872 outstanding 219,328 
3. I also track the heap and it grows proportional to the lack of BN frees

The patches released to enable RSA blinding in OpenSSL initially had
thread-safety issues; the fixes for those issues may well have introduced leaks...

Try reproducing using a vanilla OpenSSL 0.9.6m release: if it's still a problem,
report the bug back at openssl.org.  It seems unlikely this is a mod_ssl issue
now, agreed?

The blinging leak is fixed in OpenSSL 0.9.7d - mod_ssl still has a huge leak in 
the threaded MPMs using session caching

Can you try the following patch ?
It seems to fix the mem leak on HP-UX atleast :)

RCS file: /home/cvs/httpd-2.0/modules/ssl/ssl_engine_init.c,v
retrieving revision 1.126
diff -u -r1.126 ssl_engine_init.c
--- ssl_engine_init.c   5 Mar 2004 02:44:40 -0000       1.126
+++ ssl_engine_init.c   25 Mar 2004 23:27:02 -0000
@@ -450,7 +450,7 @@
          * to ignore process local-caching and
          * to always get/set/delete sessions using mod_ssl's callbacks.
          */
-        cache_mode = SSL_SESS_CACHE_SERVER|SSL_SESS_CACHE_NO_INTERNAL_LOOKUP;
+        cache_mode = SSL_SESS_CACHE_SERVER|SSL_SESS_CACHE_NO_INTERNAL;
     }
 
     SSL_CTX_set_session_cache_mode(ctx, cache_mode);

Is this fixed in CVS or not? If not, it's not fixed.

Yes, this is fixed in HEAD and for 2.0.50.

Thanks.