Bug 56919 - Creating a large number of SSL sites using DBDDriver pgsql causes a SIGSEGV / SIGILL on load
Summary: Creating a large number of SSL sites using DBDDriver pgsql causes a SIGSEGV /...
Status: RESOLVED DUPLICATE of bug 54357
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_ssl (show other bugs)
Version: 2.4.10
Hardware: PC Linux
: P2 major (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-06 13:11 UTC by Alex Bligh
Modified: 2014-09-12 05:01 UTC (History)
1 user (show)



Attachments
Perl program to generate config to replicate the bug (508 bytes, text/x-perl-script)
2014-09-06 13:11 UTC, Alex Bligh
Details
Perl program to replicate the bug under 2.4.10 (570 bytes, text/x-perl-script)
2014-09-08 17:12 UTC, Alex Bligh
Details
Demonstration patch to work around the bug (541 bytes, patch)
2014-09-08 18:41 UTC, Alex Bligh
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Bligh 2014-09-06 13:11:42 UTC
Created attachment 31968 [details]
Perl program to generate config to replicate the bug

On apache 2.4.7, loading a significant number of SSL sites causes a SIGSEGV / SIGILL on load.

The number of sites required seems inversely proportional to the number of module loads each site does. If each site does about 12 module loads, you only need 5 or 6 sites to crash it. With one module load, you need about 62. I thus believe this is in fact a problem with module loading.

I am using Ubuntu 14.04.1, with apache 2.4.7-1ubuntu4.1.

I am using the attached perl program to generate a single active configuration file and have left everything else as default. The perl program simply generates a number of sites of the form:

<VirtualHost 127.0.0.1:$port>
    ServerName 127.0.0.1:$port

    SSLEngine on
    SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
    SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

    DBDriver pgsql
</VirtualHost>

When launched I see an illegal instruction signal and Apache fails to start. Details below.

The problem does not appear to occur with DBDriver mysql (I've tried 2,000 sites) or with no DBDriver line. Putting 'DBDriver pgsql' outside the definition of the sites does not remedy the problem.

This looks similar to:
  https://issues.apache.org/bugzilla/show_bug.cgi?id=54357
except that is reported against 2.5 HEAD and only appears to involve a restart (not the initial start).

It also looks similar to:
  https://issues.apache.org/bugzilla/show_bug.cgi?id=54357
except I am not using certificate stapling.

Gory details:

root@nimtest:/home/amb/apache2/apache2-2.4.7# gdb --args ./apache2 -k start -X -e Debug
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./apache2...done.
(gdb) run
Starting program: /home/amb/apache2/apache2-2.4.7/apache2 -k start -X -e Debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Sat Sep 06 12:56:33.073345 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so
[Sat Sep 06 12:56:33.076683 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so
[Sat Sep 06 12:56:33.079835 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so
[Sat Sep 06 12:56:33.083569 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so
[Sat Sep 06 12:56:33.086495 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so
[Sat Sep 06 12:56:33.089830 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so
[Sat Sep 06 12:56:33.092941 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so
[Sat Sep 06 12:56:33.096129 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so
[Sat Sep 06 12:56:33.098897 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so
[Sat Sep 06 12:56:33.102491 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so
[Sat Sep 06 12:56:33.106266 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module cgi_module from /usr/lib/apache2/modules/mod_cgi.so
[Sat Sep 06 12:56:33.111027 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module dbd_module from /usr/lib/apache2/modules/mod_dbd.so
[Sat Sep 06 12:56:33.115365 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so
[Sat Sep 06 12:56:33.119236 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so
[Sat Sep 06 12:56:33.122860 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so
[Sat Sep 06 12:56:33.126453 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so
[Sat Sep 06 12:56:33.130551 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so
[Sat Sep 06 12:56:33.134840 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module lbmethod_byrequests_module from /usr/lib/apache2/modules/mod_lbmethod_byrequests.so
[Sat Sep 06 12:56:33.146143 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so
[Sat Sep 06 12:56:33.154798 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so
[Sat Sep 06 12:56:33.163326 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module negotiation_module from /usr/lib/apache2/modules/mod_negotiation.so
[Sat Sep 06 12:56:33.325670 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module php5_module from /usr/lib/apache2/modules/libphp5.so
[Sat Sep 06 12:56:33.331395 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module proxy_module from /usr/lib/apache2/modules/mod_proxy.so
[Sat Sep 06 12:56:33.335814 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module proxy_balancer_module from /usr/lib/apache2/modules/mod_proxy_balancer.so
[Sat Sep 06 12:56:33.339894 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module proxy_http_module from /usr/lib/apache2/modules/mod_proxy_http.so
[Sat Sep 06 12:56:33.344144 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so
[Sat Sep 06 12:56:33.349207 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module rewrite_module from /usr/lib/apache2/modules/mod_rewrite.so
[Sat Sep 06 12:56:33.353118 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so
[Sat Sep 06 12:56:33.357029 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module slotmem_shm_module from /usr/lib/apache2/modules/mod_slotmem_shm.so
[Sat Sep 06 12:56:33.362100 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so
[Sat Sep 06 12:56:33.378035 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module ssl_module from /usr/lib/apache2/modules/mod_ssl.so
[Sat Sep 06 12:56:33.382573 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so
[Sat Sep 06 12:56:33.386906 2014] [so:debug] [pid 46718] mod_so.c(266): AH01575: loaded module substitute_module from /usr/lib/apache2/modules/mod_substitute.so
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[New Thread 0x7fffe75a1700 (LWP 46726)]
[Thread 0x7fffe75a1700 (LWP 46726) exited]

Program received signal SIGILL, Illegal instruction.
0x00007ffff07fdaf4 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
(gdb) bt full
#0  0x00007ffff07fdaf4 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
No symbol table info available.
#1  0x00007ffff2b677a6 in int_free_ex_data (class_index=<optimized out>, obj=0x555555b29870, ad=0x555555b29898) at ex_data.c:522
        mx = 1
        i = 0
        item = 0x555555831cf0
        ptr = <optimized out>
        storage = 0x555555949aa0
#2  0x00007ffff2c08061 in x509_cb (operation=operation@entry=3, pval=pval@entry=0x7fffffffc1b8, it=it@entry=0x7ffff2ec3780 <X509_it>,
    exarg=exarg@entry=0x0) at x_x509.c:113
        ret = 0x555555b29870
#3  0x00007ffff2c0bfea in asn1_item_combine_free (pval=pval@entry=0x7fffffffc1b8, it=it@entry=0x7ffff2ec3780 <X509_it>, combine=combine@entry=0)
    at tasn_fre.c:173
        tt = <optimized out>
        seqtt = <optimized out>
        ef = <optimized out>
        cf = <optimized out>
        aux = <optimized out>
        asn1_cb = 0x7ffff2c07fa0 <x509_cb>
        i = <optimized out>
#4  0x00007ffff2c0c1c5 in ASN1_item_free (val=val@entry=0x555555b29870, it=it@entry=0x7ffff2ec3780 <X509_it>) at tasn_fre.c:71
No locals.
#5  0x00007ffff2c0814c in X509_free (a=a@entry=0x555555b29870) at x_x509.c:141
No locals.
#6  0x00007ffff07f00b8 in ssl_pphrase_Handle (s=s@entry=0x7ffff7fc2de0, p=p@entry=0x7ffff7fc0028) at ssl_engine_pphrase.c:275
        using_cache = 0
        mc = 0x7ffff7fc6950
        sc = 0x7ffff7ea2df0
        pServ = 0x7ffff7ea5e18
        cpVHostID = 0x7fffebe75100 "127.0.0.1:10099"
        szPath = "/etc/ssl/certs/ssl-cert-snakeoil.pem\000\177\000\000\000\000\000\000\000\000\000\000 \313\377\377\377\177\000\000\000\303\377\377\377\177\000\000\000\000\000\000\000\000\000\000\313\063\217\363\377\177\000\000 \303\377\377\377\177\000\000\260\307\377\377\377\177\000\000\320\312\377\377\377\177\000\000+", '\000' <repeats 11 times>, "\b", '\000' <repeats 51 times>, "\247\340X\360\377\177", '\000' <repeats 42 times>...
        pPrivateKey = <optimized out>
        asn1 = <optimized out>
        ucp = 0x555555832986 ""
        length = <optimized out>
        pX509Cert = 0x555555b29870
        bReadable = <optimized out>
        aPassPhrase = 0x7fffebe750c0
        nPassPhrase = 0
        nPassPhraseCur = 496363737
        cpPassPhraseCur = 0x53051b36f95004a2 <error: Cannot access memory at address 0x53051b36f95004a2>
        nPassPhraseRetry = <optimized out>
        nPassPhraseDialog = 0
        nPassPhraseDialogCur = 1377665596
        bPassPhraseDialogOnce = 4063986178
        cpp = <optimized out>
        i = 0
        j = 0
        algoCert = 1
        algoKey = 0
        at = <optimized out>
        an = 0x7ffff07ff3c2 "RSA"
        pkey_mtime = 0
        rv = <optimized out>
#7  0x00007ffff07e5658 in ssl_init_Module (p=0x7ffff7ff0028, plog=<optimized out>, ptemp=0x7ffff7fc0028, base_server=0x7ffff7fc2de0)
    at ssl_engine_init.c:194
        mc = <optimized out>
        sc = <optimized out>
        s = 0x0
---Type <return> to continue, or q <return> to quit---
#8  0x00005555555aa2a9 in ap_run_post_config (pconf=0x7ffff7ff0028, plog=0x7ffff7fbe028, ptemp=0x7ffff7fc0028, s=0x7ffff7fc2de0) at config.c:103
        pHook = 0x7ffff7e4e788
        n = 14
        rv = 0
#9  0x000055555558ae07 in main (argc=6, argv=0x7fffffffe548) at main.c:765
        c = 101 'e'
        showcompile = 0
        showdirectives = 0
        confname = 0x5555555ca607 "apache2.conf"
        def_server_root = 0x5555555ca5fa "/etc/apache2"
        temp_error_log = 0x0
        error = <optimized out>
        process = 0x7ffff7ff2118
        pconf = 0x7ffff7ff0028
        plog = 0x7ffff7fbe028
        ptemp = 0x7ffff7fc0028
        pcommands = 0x7ffff7fc8028
        opt = 0x7ffff7fc8118
        rv = <optimized out>
        mod = 0x5555557ec160 <ap_prelinked_modules+64>
        opt_arg = 0x7fffffffe7f6 "Debug"
        signal_server = <optimized out>
(gdb)

root@nimtest:/home/amb/apache2/apache2-2.4.7# apache2ctl -S
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
127.0.0.1:10000        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:2)
127.0.0.1:10001        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:12)
127.0.0.1:10002        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:22)
127.0.0.1:10003        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:32)
127.0.0.1:10004        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:42)
127.0.0.1:10005        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:52)
127.0.0.1:10006        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:62)
127.0.0.1:10007        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:72)
127.0.0.1:10008        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:82)
127.0.0.1:10009        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:92)
127.0.0.1:10010        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:102)
127.0.0.1:10011        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:112)
127.0.0.1:10012        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:122)
127.0.0.1:10013        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:132)
127.0.0.1:10014        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:142)
127.0.0.1:10015        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:152)
127.0.0.1:10016        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:162)
127.0.0.1:10017        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:172)
127.0.0.1:10018        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:182)
127.0.0.1:10019        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:192)
127.0.0.1:10020        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:202)
127.0.0.1:10021        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:212)
127.0.0.1:10022        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:222)
127.0.0.1:10023        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:232)
127.0.0.1:10024        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:242)
127.0.0.1:10025        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:252)
127.0.0.1:10026        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:262)
127.0.0.1:10027        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:272)
127.0.0.1:10028        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:282)
127.0.0.1:10029        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:292)
127.0.0.1:10030        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:302)
127.0.0.1:10031        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:312)
127.0.0.1:10032        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:322)
127.0.0.1:10033        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:332)
127.0.0.1:10034        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:342)
127.0.0.1:10035        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:352)
127.0.0.1:10036        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:362)
127.0.0.1:10037        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:372)
127.0.0.1:10038        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:382)
127.0.0.1:10039        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:392)
127.0.0.1:10040        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:402)
127.0.0.1:10041        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:412)
127.0.0.1:10042        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:422)
127.0.0.1:10043        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:432)
127.0.0.1:10044        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:442)
127.0.0.1:10045        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:452)
127.0.0.1:10046        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:462)
127.0.0.1:10047        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:472)
127.0.0.1:10048        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:482)
127.0.0.1:10049        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:492)
127.0.0.1:10050        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:502)
127.0.0.1:10051        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:512)
127.0.0.1:10052        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:522)
127.0.0.1:10053        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:532)
127.0.0.1:10054        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:542)
127.0.0.1:10055        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:552)
127.0.0.1:10056        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:562)
127.0.0.1:10057        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:572)
127.0.0.1:10058        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:582)
127.0.0.1:10059        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:592)
127.0.0.1:10060        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:602)
127.0.0.1:10061        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:612)
127.0.0.1:10062        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:622)
127.0.0.1:10063        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:632)
127.0.0.1:10064        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:642)
127.0.0.1:10065        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:652)
127.0.0.1:10066        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:662)
127.0.0.1:10067        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:672)
127.0.0.1:10068        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:682)
127.0.0.1:10069        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:692)
127.0.0.1:10070        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:702)
127.0.0.1:10071        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:712)
127.0.0.1:10072        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:722)
127.0.0.1:10073        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:732)
127.0.0.1:10074        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:742)
127.0.0.1:10075        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:752)
127.0.0.1:10076        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:762)
127.0.0.1:10077        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:772)
127.0.0.1:10078        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:782)
127.0.0.1:10079        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:792)
127.0.0.1:10080        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:802)
127.0.0.1:10081        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:812)
127.0.0.1:10082        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:822)
127.0.0.1:10083        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:832)
127.0.0.1:10084        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:842)
127.0.0.1:10085        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:852)
127.0.0.1:10086        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:862)
127.0.0.1:10087        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:872)
127.0.0.1:10088        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:882)
127.0.0.1:10089        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:892)
127.0.0.1:10090        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:902)
127.0.0.1:10091        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:912)
127.0.0.1:10092        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:922)
127.0.0.1:10093        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:932)
127.0.0.1:10094        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:942)
127.0.0.1:10095        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:952)
127.0.0.1:10096        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:962)
127.0.0.1:10097        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:972)
127.0.0.1:10098        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:982)
127.0.0.1:10099        127.0.0.1 (/etc/apache2/sites-enabled/test.conf:992)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/lock/apache2" mechanism=fcntl
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33
Comment 1 Rainer Jung 2014-09-06 13:32:54 UTC
BZ 54357 contains another user report of a crash in the same stack directly after start for 2.4.7, which was commented by the same user as being fixed for him after updating to 2.4.9.

Any chance you can update to latest 2.4 and try again?
Comment 2 Alex Bligh 2014-09-06 15:41:48 UTC
[ Note for anyone trying to duplicate this: On a clean container on the same machine, I needed 141 or more sites to duplicate this. Also it appears it is necessary to enable mod_php]

On 2.4.10_1ubuntu1 (utopic version recompiled for trusty), this appears not to occur, which is good news.

BZ 54357 appears to involve certificate stapling, which I have switched off (I believe that's the default). I would rather use 2.4.7 if possible simply because that is the stock version Ubuntu distribute and support. Failing that, I'm happy to identify the specific issue, recompile, and try to persuade Ubuntu to apply a patch to 2.4.7. Any idea what the underlying issue is here, or how I might work around it without an upgrade?
Comment 3 Alex Bligh 2014-09-08 15:56:02 UTC
Looks like I spoke too soon. This *DOES* occur on 2.4.10, it's just more difficult to replicate. Of course it replicates just fine with my real-world example.

Here's a backtrace of it dying on 2.4.10. I will try to amend the test case to replicate this. In the meantime is there anything further I can do to debug this?

root@nimtest:~# gdb --args /usr/sbin/apache2 -k start -X -e Debug
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug//usr/sbin/apache2...done.
done.
(gdb) run
Starting program: /usr/sbin/apache2 -k start -X -e Debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Mon Sep 08 15:53:47.686373 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so
[Mon Sep 08 15:53:47.690215 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so
[Mon Sep 08 15:53:47.695217 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so
[Mon Sep 08 15:53:47.697928 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so
[Mon Sep 08 15:53:47.703892 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so
[Mon Sep 08 15:53:47.708513 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so
[Mon Sep 08 15:53:47.714280 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so
[Mon Sep 08 15:53:47.717910 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so
[Mon Sep 08 15:53:47.725992 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so
[Mon Sep 08 15:53:47.733997 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so
[Mon Sep 08 15:53:47.739117 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module cache_module from /usr/lib/apache2/modules/mod_cache.so
[Mon Sep 08 15:53:47.744871 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module cgi_module from /usr/lib/apache2/modules/mod_cgi.so
[Mon Sep 08 15:53:47.750762 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module dbd_module from /usr/lib/apache2/modules/mod_dbd.so
[Mon Sep 08 15:53:47.757628 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so
[Mon Sep 08 15:53:47.765739 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so
[Mon Sep 08 15:53:47.772183 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so
[Mon Sep 08 15:53:47.780369 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so
[Mon Sep 08 15:53:47.788832 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so
[Mon Sep 08 15:53:47.794207 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module ident_module from /usr/lib/apache2/modules/mod_ident2.so
[Mon Sep 08 15:53:47.797959 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module lbmethod_byrequests_module from /usr/lib/apache2/modules/mod_lbmethod_byrequests.so
[Mon Sep 08 15:53:47.801879 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so
[Mon Sep 08 15:53:47.806730 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so
[Mon Sep 08 15:53:47.813710 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module negotiation_module from /usr/lib/apache2/modules/mod_negotiation.so
[Mon Sep 08 15:53:47.952346 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module php5_module from /usr/lib/apache2/modules/libphp5.so
[Mon Sep 08 15:53:47.957451 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module proxy_module from /usr/lib/apache2/modules/mod_proxy.so
[Mon Sep 08 15:53:47.960908 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module proxy_balancer_module from /usr/lib/apache2/modules/mod_proxy_balancer.so
[Mon Sep 08 15:53:47.964292 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module proxy_http_module from /usr/lib/apache2/modules/mod_proxy_http.so
[Mon Sep 08 15:53:47.967260 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so
[Mon Sep 08 15:53:47.971368 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module rewrite_module from /usr/lib/apache2/modules/mod_rewrite.so
[Mon Sep 08 15:53:47.974517 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so
[Mon Sep 08 15:53:47.977591 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module slotmem_shm_module from /usr/lib/apache2/modules/mod_slotmem_shm.so
[Mon Sep 08 15:53:47.980582 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so
[Mon Sep 08 15:53:47.990700 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module ssl_module from /usr/lib/apache2/modules/mod_ssl.so
[Mon Sep 08 15:53:47.994364 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so
[Mon Sep 08 15:53:47.997902 2014] [so:debug] [pid 15446] mod_so.c(266): AH01575: loaded module substitute_module from /usr/lib/apache2/modules/mod_substitute.so
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-extility-amber-listen.conf:15
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[New Thread 0x7fffe7166700 (LWP 15462)]
[Thread 0x7fffe7166700 (LWP 15462) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff03e5139 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
(gdb) bt full
#0  0x00007ffff03e5139 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
No symbol table info available.
#1  0x00007ffff274f7a6 in int_free_ex_data (class_index=<optimized out>, obj=0x555555b03830, ad=0x555555b03858) at ex_data.c:522
        mx = 2
        i = 0
        item = 0x5555558331f0
        ptr = <optimized out>
        storage = 0x555555b03ab0
#2  0x00007ffff27f0061 in x509_cb (operation=operation@entry=3, pval=pval@entry=0x7fffffffdfe8, it=it@entry=0x7ffff2aab780 <X509_it>,
    exarg=exarg@entry=0x0) at x_x509.c:113
        ret = 0x555555b03830
#3  0x00007ffff27f3fea in asn1_item_combine_free (pval=pval@entry=0x7fffffffdfe8, it=it@entry=0x7ffff2aab780 <X509_it>, combine=combine@entry=0)
    at tasn_fre.c:173
        tt = <optimized out>
        seqtt = <optimized out>
        ef = <optimized out>
        cf = <optimized out>
        aux = <optimized out>
        asn1_cb = 0x7ffff27effa0 <x509_cb>
        i = <optimized out>
#4  0x00007ffff27f41c5 in ASN1_item_free (val=0x555555b03830, it=it@entry=0x7ffff2aab780 <X509_it>) at tasn_fre.c:71
No locals.
#5  0x00007ffff27f014c in X509_free (a=<optimized out>) at x_x509.c:141
No locals.
#6  0x00007ffff24caf2d in SSL_load_client_CA_file (file=<optimized out>) at ssl_cert.c:726
        in = 0x555555b02990
        x = 0x555555b03830
        xn = <optimized out>
        ret = <optimized out>
        sk = 0x555555b04b70
#7  0x00007ffff03ca871 in ssl_init_PushCAList (ca_list=0x555555b04190, s=0x7fffebad9238, ptemp=0x7ffff7fc0028, file=<optimized out>)
    at ssl_engine_init.c:1587
        n = <optimized out>
        sk = <optimized out>
#8  0x00007ffff03cae50 in ssl_init_FindCAList (s=s@entry=0x7fffebad9238, ptemp=ptemp@entry=0x7ffff7fc0028,
    ca_file=0x7fffebad75c0 "/etc/ssl/certs/extility-cluster-ca.crt", ca_path=0x7fffebad7578 "/etc/ssl/none") at ssl_engine_init.c:1637
        ca_list = 0x555555b04190
#9  0x00007ffff03cb38e in ssl_init_ctx_verify (p=0x7ffff7ff0028, mctx=0x7ffff7e18140, ptemp=0x7ffff7fc0028, s=0x7fffebad9238)
    at ssl_engine_init.c:674
        ctx = 0x555555b01d30
        verify = <optimized out>
        ca_list = <optimized out>
#10 ssl_init_ctx (s=0x7fffebad9238, p=0x7ffff7ff0028, ptemp=0x7ffff7fc0028, mctx=0x7ffff7e18140) at ssl_engine_init.c:863
No locals.
#11 0x00007ffff03cc4d8 in ssl_init_server_ctx (pphrases=0x7ffff7eab110, sc=0x7ffff7e3ff50, ptemp=0x7ffff7fc0028, p=0x7ffff7ff0028, s=0x7fffebad9238)
    at ssl_engine_init.c:1370
        rv = <optimized out>
#12 ssl_init_ConfigureServer (s=s@entry=0x7fffebad9238, p=p@entry=0x7ffff7ff0028, ptemp=ptemp@entry=0x7ffff7fc0028, sc=0x7ffff7e3ff50,
    pphrases=pphrases@entry=0x7ffff7eab110) at ssl_engine_init.c:1469
No locals.
#13 0x00007ffff03cd319 in ssl_init_Module (p=0x7ffff7ff0028, plog=<optimized out>, ptemp=0x7ffff7fc0028, base_server=0x7ffff7fc2de0)
    at ssl_engine_init.c:304
        mc = <optimized out>
        sc = <optimized out>
        s = 0x7fffebad9238
        rv = <optimized out>
        pphrases = 0x7ffff7eab110
#14 0x00005555555ab019 in ap_run_post_config (pconf=0x7ffff7ff0028, plog=0x7ffff7fbe028, ptemp=0x7ffff7fc0028, s=0x7ffff7fc2de0) at config.c:103
        pHook = 0x7ffff7eef0c0
        n = 15
        rv = 540686391
#15 0x000055555558b137 in main (argc=6, argv=0x7fffffffe598) at main.c:765
        c = 101 'e'
        showcompile = 0
        showdirectives = 0
        confname = 0x5555555cb4e7 "apache2.conf"
        def_server_root = 0x5555555cb4da "/etc/apache2"
        temp_error_log = 0x0
        error = <optimized out>
        process = 0x7ffff7ff2118
        pconf = 0x7ffff7ff0028
        plog = 0x7ffff7fbe028
        ptemp = 0x7ffff7fc0028
        pcommands = 0x7ffff7fc8028
        opt = 0x7ffff7fc8118
        rv = <optimized out>
        mod = 0x5555557ed160 <ap_prelinked_modules+64>
        opt_arg = 0x7fffffffe826 "Debug"
        signal_server = <optimized out>
(gdb)
Comment 4 Alex Bligh 2014-09-08 17:12:55 UTC
Created attachment 31975 [details]
Perl program to replicate the bug under 2.4.10

I've attached what I believe is the minimal perl program to replicate the bug under 2.4.10.

This simply adds one line to the server config:

    SSLCACertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem

Perhaps there was something fixed in normal SSLCertificate loading that has yet to be fixed in CA certificate loading.
Comment 5 Alex Bligh 2014-09-08 17:13:56 UTC
(note, you can have a single site that uses an SSLCACertificateFile after a lot of sites not using them, and still see the bug)
Comment 6 Alex Bligh 2014-09-08 18:20:00 UTC
I believe I now understand the root cause of this bug and it's very NASTY. I don't think it's properly fixed in 2.4.10 for any certificates.

What appears to be happening is this. In main.c (prior to line 702), apache processes the config file. This dlopen()'s mod_ssl, and calls ssl_init_Module(). At main.c line 707, inside the 'for (;;)' loop around reloads, it does an apr_pool_clear(). This dlclose()'s all the modules that have been open, and reprocesses the configuration, which again dlopen()'s mod_ssl and calls ssl_init_Module() again.

However, inspection shows that mod_ssl isn't always loaded at the same address. If you have a large configuration, that's more likely (not sure whether it's a memory leak, or fragmentation, or what, but this is the cause). For instance, I put a breakpoint on ssl_init_Module(), and then ran apache2 and got the output below. You can see the location of ssl_init_Module has changed.

This in itself would not be an issue. However, when the certificates are first loaded through openssl, they are set up with a free_func (in the openssl structure) that points to something in mod_ssl freeing the additional storage - see http://osxr.org/openssl/source/crypto/ex_data.c#0566

When the config file is reprocessed, that free_func's address changes. However, openssl object has not (yet) been freed. When it is, the free_func() is called using the PREVIOUS address associated with mod_ssl. What I believe is happening here is that the SSL library is checking to see whether a certificate with the same CN has already been loaded, here:
 http://osxr.org/openssl/source/ssl/ssl_cert.c#0707
and deinitialising mod_ssl is not clearing the loaded certificates. This is probably deliberate as there may be other users of the openssl library that might also be loading certificates.

I don't really know how one would go about fixing this. The least horrible option I think would be to never dlclose() a module once it is loaded. IE a reload of apache2 would leave modules in RAM (but presumably deinited), so that they would always be at the same place. A reload would then leave them loaded.

Breakpoint 1, ssl_init_Module (p=0x7ffff7ff0028, plog=0x7ffff7fbe028, ptemp=0x7ffff7fbc028, base_server=0x7ffff7fc1ec8) at ssl_engine_init.c:138
138	{
(gdb) print pc
No symbol "pc" in current context.
(gdb) print &ssl_init_Module
$1 = (apr_status_t (*)(apr_pool_t *, apr_pool_t *, apr_pool_t *, server_rec *)) 0x7ffff03d7000 <ssl_init_Module>
(gdb) cont
Continuing.
warning: Temporarily disabling breakpoints for unloaded shared library "/usr/lib/apache2/modules/mod_ssl.so"
[New Thread 0x7fffe6fee700 (LWP 56253)]
[Thread 0x7fffe6fee700 (LWP 56253) exited]

Breakpoint 1, ssl_init_Module (p=0x7ffff7ff0028, plog=0x7ffff7fbe028, ptemp=0x7ffff7fc0028, base_server=0x7ffff7fc2de0) at ssl_engine_init.c:138
138	{
(gdb) print &ssl_init_Module
$2 = (apr_status_t (*)(apr_pool_t *, apr_pool_t *, apr_pool_t *, server_rec *)) 0x7ffff03cd000 <ssl_init_Module>
(g
Comment 7 Alex Bligh 2014-09-08 18:41:08 UTC
Created attachment 31977 [details]
Demonstration patch to work around the bug

A minimum patch for this bug is attached. This swaps the SEGV for a memory leak, on the basis that a memory leak is probably less bad. I am neither sure this is suitable or a complete solution.

The patch works as follows: the problem is that the address of certinfo_free is being stored somewhere deep in openssl. When modssl is dlclosed()'d and dlopen()'d again, the address of certinfo_free may change. openssl then calls the free function at its old location, and SEGV / illegal instruction ensues. By not providing a free function for the extra data, we avoid openssl calling anything.

This appears to avoid the test case crashing, which at least means the problem is correctly identified.
Comment 8 Ruediger Pluem 2014-09-08 19:29:50 UTC
I would guess the proper cleanup is missing in ssl_init_ModuleKill.
Comment 9 Alex Bligh 2014-09-09 11:28:59 UTC
I'm not sure what the proper way to fix is this.

It's tempting to call CRYPTO_cleanup_all_ex_data, but I don't think that's the right solution. Firstly ssl_cleanup_pre_config says:

    /* Also don't call CRYPTO_cleanup_all_ex_data here; any registered
     * ex_data indices may have been cached in static variables in
     * OpenSSL; removing them may cause havoc.  Notably, with OpenSSL
     * versions >= 0.9.8f, COMP_CTX cleanups would not be run, which
     * could result in a per-connection memory leak (!). */

Secondly some other ssl user (for instance a DBD driver using an SSL interface to the database) may not take kindly to us stomping on its data.

It's tempting to remove the index that X509_get_ex_new_index added, removing the data, save that as far as I can see openssl doesn't have an API call to do that.

That would leave us attempting to ensure that every single object that mod_ssl allocates is freed. But firstly, I'm not sure how to do that, and secondly this won't fix the problem where there is some other ssl user that also allocates objects. It would also be inherently fragile.

The final option would be to rewrite the stapling code so it didn't use ex_data at all. To me this seems like the best route, but I don't understand the stapling code well enough to do it.

Is there some easier option I have missed?
Comment 10 Kaspar Brand 2014-09-10 15:06:07 UTC
Thank you for the thorough debugging and analysis, Alex. I think it's really a duplicate of bug 54357, and it would be best to dupe this one into it (or vice versa).

(In reply to Alex Bligh from comment #9)
> The final option would be to rewrite the stapling code so it didn't use
> ex_data at all. To me this seems like the best route, but I don't understand
> the stapling code well enough to do it.
> 
> Is there some easier option I have missed?

One option might be to avoid ex_data fiddling in the "first round", based on a ssl_config_global_isfixed() check - i.e., something like this (untested):

Index: ssl_engine_init.c
===================================================================
--- ssl_engine_init.c   (revision 1624017)
+++ ssl_engine_init.c   (working copy)
@@ -272,7 +272,9 @@
         return HTTP_INTERNAL_SERVER_ERROR;
     }
 #ifdef HAVE_OCSP_STAPLING
-    ssl_stapling_ex_init();
+    if (ssl_config_global_isfixed(mc) == TRUE) {
+        ssl_stapling_ex_init();
+    }
 #endif

     /*
@@ -1067,6 +1069,7 @@
          * later, we defer to the code in ssl_init_server_ctx.
          */
         if ((mctx->stapling_enabled == TRUE) &&
+            (ssl_config_global_isfixed(mc) == TRUE) &&
             !ssl_stapling_init_cert(s, mctx, cert)) {
             ap_log_error(APLOG_MARK, APLOG_ERR, 0, s, APLOGNO(02567)
                          "Unable to configure certificate %s for stapling",
@@ -1418,7 +1421,8 @@
      * (late) point makes sure that we catch both certificates loaded
      * via SSLCertificateFile and SSLOpenSSLConfCmd Certificate.
      */
-    if (sc->server->stapling_enabled == TRUE) {
+    if ((sc->server->stapling_enabled == TRUE) &&
+        (ssl_config_global_isfixed(myModConfig(s)) == TRUE)) {
         X509 *cert;
         int i = 0;
         int ret = SSL_CTX_set_current_cert(sc->server->ssl_ctx,


Getting rid of ex_data might be cleaner in the end, and was actually one of Joe's questions on the dev list in October 2009:

https://mail-archives.apache.org/mod_mbox/httpd-dev/200910.mbox/%3C20091025200721.GA20714@redhat.com%3E

(see also bug 43822)
Comment 11 Ruediger Pluem 2014-09-11 06:09:25 UTC
(In reply to Kaspar Brand from comment #10)

> One option might be to avoid ex_data fiddling in the "first round", based on
> a ssl_config_global_isfixed() check - i.e., something like this (untested):
> 
> Index: ssl_engine_init.c
> ===================================================================
> --- ssl_engine_init.c   (revision 1624017)
> +++ ssl_engine_init.c   (working copy)
> @@ -272,7 +272,9 @@
>          return HTTP_INTERNAL_SERVER_ERROR;
>      }
>  #ifdef HAVE_OCSP_STAPLING
> -    ssl_stapling_ex_init();
> +    if (ssl_config_global_isfixed(mc) == TRUE) {
> +        ssl_stapling_ex_init();
> +    }
>  #endif

Maybe I am missing something, but we always call

ssl_config_global_fix(mc);

a few lines above. So the condition would be always true.
Comment 12 Ruediger Pluem 2014-09-11 06:09:52 UTC

*** This bug has been marked as a duplicate of bug 54357 ***
Comment 13 Kaspar Brand 2014-09-12 05:01:56 UTC
(In reply to Ruediger Pluem from comment #11)
> Maybe I am missing something, but we always call
> 
> ssl_config_global_fix(mc);
> 
> a few lines above. So the condition would be always true.

You're absolutely right, my bad. Forget about my idea in comment 10 completely, as it also wouldn't work for restarts.