Bug 62552 - Loading mod_md segfaults when using DBDriver from mod_authn_dbd
Summary: Loading mod_md segfaults when using DBDriver from mod_authn_dbd
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_md (show other bugs)
Version: 2.4.34
Hardware: PC Linux
: P3 major with 3 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-07-18 09:54 UTC by Thomas Witzenrath
Modified: 2020-11-27 20:21 UTC (History)
0 users



Attachments
Backtrace from GDB (21.19 KB, text/plain)
2018-07-18 09:54 UTC, Thomas Witzenrath
Details
Minimum httpd.conf to hit the problem (968 bytes, text/plain)
2018-07-18 10:25 UTC, Thomas Witzenrath
Details
Backtrace from minimum config (10.24 KB, text/plain)
2018-07-18 12:39 UTC, Thomas Witzenrath
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Witzenrath 2018-07-18 09:54:46 UTC
Created attachment 36037 [details]
Backtrace from GDB

When mod_md is loaded and a virtualhost uses the "DBDriver mysql"-Directive from mod_authn_dbd, httpd segfaults reliably on startup.

This also happens in Version 2.4.34 which is not selectable.
Comment 1 Thomas Witzenrath 2018-07-18 10:25:43 UTC
Created attachment 36039 [details]
Minimum httpd.conf to hit the problem
Comment 2 Thomas Witzenrath 2018-07-18 10:26:38 UTC
elukey suggested in the #httpd-dev IRC Channel to add that we're running openssl-1.0.2o
Comment 3 Joe Orton 2018-07-18 10:36:35 UTC
Is mysql linked against the same version of OpenSSL as httpd?
Comment 4 Thomas Witzenrath 2018-07-18 11:10:04 UTC
yes it is
Comment 5 Thomas Witzenrath 2018-07-18 11:10:37 UTC
yes it is
Comment 6 Stefan Eissing 2018-07-18 11:48:54 UTC
In the stacktrace you load mod_md after mod_authn_dbd. In the minimal config, you load it before.

Is the stack trace still in mod_md? (If the problem lies in some openssl initialization ordering, it should change, right?)
Comment 7 Stefan Eissing 2018-07-18 11:50:10 UTC
Correction, you load mod_dbd after, mod_authn_dbd ist still before. What happens if you place mod_authn_dbd after? Thanks.

(bit overloaded today)
Comment 8 Thomas Witzenrath 2018-07-18 11:57:57 UTC
After changing the order of mod_authn_dbd and mod_md, it still segfaulted. But this got me to think about the ordering of mod_ssl and mod_md. 

When placing the 

LoadModule md_module modules/mod_md.so

after (!)

LoadModule ssl_module modules/mod_ssl.so

httpd starts happily. So it seems to be a problem with mod_ssl and mod_md?
Comment 9 Stefan Eissing 2018-07-18 11:59:37 UTC
Isn't mod_md loaded _after_ mod_ssl in the attached backtrace?
Comment 10 Ruediger Pluem 2018-07-18 12:00:13 UTC
Added 2.4.34 and set it.
Comment 11 Thomas Witzenrath 2018-07-18 12:09:29 UTC
[Wed Jul 18 11:23:52.739142 2018] [so:debug] [pid 10467:tid 140737354008448] mod_so.c(266): AH01575: loaded module md_module from /usr/lib64/apache2/modules/mod_md.so
[..]
[Wed Jul 18 11:23:52.903327 2018] [so:debug] [pid 10467:tid 140737354008448] mod_so.c(266): AH01575: loaded module ssl_module from /usr/lib64/apache2/modules/mod_ssl.so

No, it was never loaded before mod_ssl
Comment 12 Thomas Witzenrath 2018-07-18 12:11:42 UTC
Still this only happens when mod_authn_dbd is used. A different server uses the exact same config, just without the Virtualhost that states "DBDriver mysql", and has been running without a segfault for days.
Comment 13 Stefan Eissing 2018-07-18 12:19:59 UTC
I agree that something seems strange, which normally indicates that there is a hidden variable which we are not aware of yet.

In your backtrace attached here, I read:

[Wed Jul 18 11:23:52.903327 2018] [so:debug] [pid 10467:tid 140737354008448] mod_so.c(266): AH01575: loaded module ssl_module from 
...
Program received signal SIGSEGV, Segmentation fault.
0x00007fffef1c2610 in ?? ()

(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7fe0780 (LWP 10467)):
...
#8  0x00005555555c0638 in load_module (cmd=0x7fffffffdaa0, dummy=<optimized out>, modname=0x555555853a68 "md_module", filename=0x555555853a78 "modules/mod_md.so") at mod_so.c:302
#9  0x00005555555ac410 in invoke_cmd (cmd=0x5555557f35e0 <so_cmds>, parms=parms@entry=0x7fffffffdaa0, mconfig=mconfig@entry=0x7fffffffd5b0, args=<optimized out>, args@entry=0x55555582dc53 "md_module modules/mod_md.so") at config.c:948

That is why I thought mod_ssl has been loaded before mod_md. How am I reading that wrong?

Once we know for sure exactly which ordering is problematic, maybe we can find the difference between mod_ssl's initalization of openssl and the way mod_md does it.

The other thing that worries me is: if you say that only the configuration command

DBDriver mysql

triggers the segfault then this can only happen if that config line comes _before_ the LoadModule of mod_md. But in your minimal example, this is not the case. So, I think we do not have a clear model of what happens when...
Comment 14 Thomas Witzenrath 2018-07-18 12:39:22 UTC
Created attachment 36040 [details]
Backtrace from minimum config
Comment 15 Stefan Eissing 2018-07-18 13:07:30 UTC
The minimal backtrace shows where exactly the crash in openssl 1.0.2 happens:

Thread 1 (Thread 0x7ffff7fe0780 (LWP 31111)):
#0  0x00007ffff4346610 in ?? ()
#1  0x00007ffff4f07935 in ssleay_rand_status () at md_rand.c:577
#2  0x00007ffff54b312c in md_crypt_init (pool=pool@entry=0x555555803388) at md_crypt.c:136

And md_rand.c:577 in OpenSSL 1.0.2 is:

571: static int ssleay_rand_status(void)
572: {
573:     CRYPTO_THREADID cur;
574:     int ret;
575:     int do_not_lock;
576: 
577:     CRYPTO_THREADID_current(&cur);

Which hints at problems with initialization order in openssl usage *before* mod_ssl installs its own crypto lock handler.

OpenSSL 1.1.x eliminated that locking callback, I think. For an apache with
several modules using OpenSSL, how would we ensure that initialization order is deterministic. And what if modules want to access openssl *before* the post config phase?
Comment 16 Manuel Schmitt (manitu) 2018-09-19 03:06:35 UTC
We have a similar issue at

    https://bugs.gentoo.org/666406

with the difference that it has NOT to do with mod_md (we are not using it).

We can confirm "our" bug with apache 2.4.33 and 2.4.34. Our module order and list is:

	LoadModule access_compat_module modules/mod_access_compat.so
	LoadModule actions_module modules/mod_actions.so
	LoadModule alias_module modules/mod_alias.so
	LoadModule auth_basic_module modules/mod_auth_basic.so
	LoadModule authn_anon_module modules/mod_authn_anon.so
	LoadModule authn_core_module modules/mod_authn_core.so
	LoadModule authn_dbd_module modules/mod_authn_dbd.so
	LoadModule authn_dbm_module modules/mod_authn_dbm.so
	LoadModule authn_file_module modules/mod_authn_file.so
	LoadModule authz_core_module modules/mod_authz_core.so
	LoadModule authz_dbd_module modules/mod_authz_dbd.so
	LoadModule authz_dbm_module modules/mod_authz_dbm.so
	LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
	LoadModule authz_host_module modules/mod_authz_host.so
	LoadModule authz_owner_module modules/mod_authz_owner.so
	LoadModule authz_user_module modules/mod_authz_user.so
	LoadModule autoindex_module modules/mod_autoindex.so
	LoadModule cgi_module modules/mod_cgi.so
	LoadModule cgid_module modules/mod_cgid.so
	LoadModule dbd_module modules/mod_dbd.so
	LoadModule deflate_module modules/mod_deflate.so
	LoadModule dir_module modules/mod_dir.so
	LoadModule env_module modules/mod_env.so
	LoadModule expires_module modules/mod_expires.so
	LoadModule ext_filter_module modules/mod_ext_filter.so
	LoadModule filter_module modules/mod_filter.so
	LoadModule headers_module modules/mod_headers.so
	LoadModule include_module modules/mod_include.so
	LoadModule info_module modules/mod_info.so
	LoadModule log_config_module modules/mod_log_config.so
	LoadModule logio_module modules/mod_logio.so
	LoadModule macro_module modules/mod_macro.so
	LoadModule mime_module modules/mod_mime.so
	LoadModule mime_magic_module modules/mod_mime_magic.so
	LoadModule negotiation_module modules/mod_negotiation.so
	LoadModule reqtimeout_module modules/mod_reqtimeout.so
	LoadModule rewrite_module modules/mod_rewrite.so
	LoadModule setenvif_module modules/mod_setenvif.so
	LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
	LoadModule speling_module modules/mod_speling.so
	LoadModule ssl_module modules/mod_ssl.so
	LoadModule status_module modules/mod_status.so
	LoadModule unique_id_module modules/mod_unique_id.so
	LoadModule unixd_module modules/mod_unixd.so
	LoadModule usertrack_module modules/mod_usertrack.so
	LoadModule vhost_alias_module modules/mod_vhost_alias.so

The problem does NOT arise with 2.4.27.

More over we did find out that it happens only if

a) ssl module is loaded
b) a "DBDriver mysql" is used

together - but if there is only a) or b).

The issue is also there when using openssl 1.0.2o instead of p.


Since we are using Gentoo, we also did a full rebuild of all components (apache, apr, apr-util, openssl, mysql client lib) against each other (especially openssl). Nothing helped.
Comment 17 Manuel Schmitt (manitu) 2018-09-19 03:09:11 UTC
Sorry for the typo, Imeant

(In reply to Manuel Schmitt (manitu) from comment #16)

> More over we did find out that it happens only if
> 
> a) ssl module is loaded
> b) a "DBDriver mysql" is used
> 
> together - but if there is only a) or b).

... but NOT if there is only a) or b) alone.
Comment 18 Thomas Witzenrath 2018-09-19 05:31:35 UTC
could you try moving the mod_ssl loading before the mod_auth* modules? To my understanding that should "fix" the problem.
Comment 19 Manuel Schmitt (manitu) 2018-09-19 05:54:16 UTC
No, for us (2.4.34) moving loading mod_ssl very on top did _NOT_ solve it. Same segfault.
Comment 20 Thomas Witzenrath 2018-09-24 07:08:29 UTC
We are running apache 2.4.34 on Gentoo as well, with DBDriver mysql and ssl module. But I was wrong with the ordering of the module loads. Try loading the ssl module AFTER the dbd modules.
Comment 21 Manuel Schmitt (manitu) 2018-09-25 04:55:29 UTC
Also happens with 2.4.35
Comment 22 Manuel Schmitt (manitu) 2018-09-25 04:59:12 UTC
(In reply to Thomas Witzenrath from comment #20)
> We are running apache 2.4.34 on Gentoo as well, with DBDriver mysql and ssl
> module. But I was wrong with the ordering of the module loads. Try loading
> the ssl module AFTER the dbd modules.

SSL is loaded AFTER dbd stuff on Gentoo per default. So it is on with us.
Comment 23 Manuel Schmitt (manitu) 2018-09-25 05:02:37 UTC
> (In reply to Thomas Witzenrath from comment #20)
> > We are running apache 2.4.34 on Gentoo as well, with DBDriver mysql and ssl
> > module. But I was wrong with the ordering of the module loads. Try loading
> > the ssl module AFTER the dbd modules.
> 
> SSL is loaded AFTER dbd stuff on Gentoo per default. So it is on with us.

We did try to move loading the ssl module on each position before and after all other modules. Happens all the time.