Starting with a fresh installation of CentOS 6.5 and a fresh compilation of the latest MySQL 5.5 and Apache (with latest apr and apr-util). Minimal changes to the default configuration, to permit the use of mod_dbd and mod_ssl Only one non-ssl site (with dbd auth) and only one ssl site - With apache 2.4.9, it crash if in the ssl site I use SSLCACertificateFile ( I have some sites that use client certificates). Removing that directive, apache starts and dbd authentication works. - With apache 2.4.7, it crash with any SSL directive configured - Both installation works if apache starts without the DBDriver directive. Adding the directive and doing a graceful restart works. - With LogLevel = Debug or tracex there isn't any interesting in the log - Running httpd -X crash at startup without any log - Running gdb with "run -X" works!!! I think I hit a bug. Please advise me if you need other info or tests. I'm not a developer, so i never used gbd or strace. If you need it please give me detailed instrucion.
maybe related to this unreleased fix; *) mod_ssl: add workaround for SSLCertificateFile when using OpenSSL versions before 0.9.8h and not specifying an SSLCertificateChainFile (regression introduced with 2.4.8). PR 56410. [Kaspar Brand]
Hello, I have Openssl 1.0.1e. I tried to apply the patch but the problem remains. Thanks, Andrea
backtrace of the crash?
I'm not a developer and never used gdb or strace. Which are exactly the commands to run?
this might be enough of a primer: http://httpd.apache.org/dev/debugging.html#crashes
some libs/modules doesn't have symbols. Probably I have to enabled it at compile time? Which is the flag for apache? --enable-maintainer-mode or --enable-debugger-mode? GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/apache/sbin/httpd...done. [New Thread 1834] Missing separate debuginfo for /usr/local/apache/lib/libaprutil-1.so.0 Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8b/30b896070dbf90b7ed83dfb7aa47d590f47e45 Missing separate debuginfo for /usr/local/apache/lib/libexpat.so.0 Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/51/b7c6a8c513773485000c9d0765e448dd39d20e Missing separate debuginfo for /usr/local/apache/lib/libapr-1.so.0 Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c4/a7c7c66fd027fd0d7fe2a83a76fa756d8dde5d Missing separate debuginfo for /usr/local/apache/lib/apr-util-1/apr_dbd_mysql-1.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/40/2bcb8d7d2abcb94aee8a1a2bcd63f7681ccf2e Missing separate debuginfo for /usr/local/mysql/lib/libmysqlclient.so.18 Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/dd/6ae1ad0185be9e1e98f680c9328f96f1429ad8 Missing separate debuginfo for /usr/local/apache/modules/mod_authn_file.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/78/460a5903f37f5cf01df52c3d50198ef69a519b Missing separate debuginfo for /usr/local/apache/modules/mod_authn_dbd.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/76/23e4732826d4cfebac5c8a22f87ece64e7ccb4 Missing separate debuginfo for /usr/local/apache/modules/mod_authn_core.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e2/f0799447ec1d5132bfd5396e86695203bdad95 Missing separate debuginfo for /usr/local/apache/modules/mod_authz_host.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a1/a280ac5d204bf82ed0382f33c3f12afb071c59 Missing separate debuginfo for /usr/local/apache/modules/mod_authz_groupfile.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/35/63689551aeced5c6e6db6c4b1f7ddba39f0108 Missing separate debuginfo for /usr/local/apache/modules/mod_authz_user.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/2d/df0a5feed81965132ddec2680240a223416ebf Missing separate debuginfo for /usr/local/apache/modules/mod_authz_dbd.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/55/ddeec7469acdf09b6c46147dd8bb2d7ec6e72e Missing separate debuginfo for /usr/local/apache/modules/mod_authz_core.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3c/5d13c73b8d96e66eb55c2c0f1607986c9632ea Missing separate debuginfo for /usr/local/apache/modules/mod_access_compat.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7d/9e35fc4875bebf310c43b3a2953db6aea43737 Missing separate debuginfo for /usr/local/apache/modules/mod_auth_basic.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/96/2312048bd331d73048421b84f334a8e254c90f Missing separate debuginfo for /usr/local/apache/modules/mod_socache_shmcb.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/da/ffd2f0aabb4e501f175f2a4e0f9e327b7df0e2 Missing separate debuginfo for /usr/local/apache/modules/mod_dbd.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/cf/343fec8de8099569fdd1c8b96229d4257c8f9a Missing separate debuginfo for /usr/local/apache/modules/mod_reqtimeout.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e3/ed1589946792acd6efc2ff060ab76427f20436 Missing separate debuginfo for /usr/local/apache/modules/mod_filter.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/5f/cd7c177d1f88f83f2781d4b7ecfbb68ae07bb7 Missing separate debuginfo for /usr/local/apache/modules/mod_mime.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ca/71a21c41db40d9905a5574fb22b6e23d4e9af2 Missing separate debuginfo for /usr/local/apache/modules/mod_log_config.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b4/bce19840aa08b1fc6ea16828a046f52c88aaaa Missing separate debuginfo for /usr/local/apache/modules/mod_env.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a0/cff45a3c05876579fe227eb1fd4c174a0c74b9 Missing separate debuginfo for /usr/local/apache/modules/mod_headers.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/98/144de1306576f4ae32132e00b1668334639823 Missing separate debuginfo for /usr/local/apache/modules/mod_setenvif.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/06/75aa1c4e88035d0a2e0c414085164aa238be6c Missing separate debuginfo for /usr/local/apache/modules/mod_version.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/79/4f5c1f90a85e0d49e62787b98ffbaf2a75439c Missing separate debuginfo for /usr/local/apache/modules/mod_ssl.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/45/fa8db1d8f83844c8ba7736bacdecdb00ddff1e Missing separate debuginfo for /usr/local/apache/modules/mod_unixd.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e0/8e263de1775049f3939e2efb213ea3190187df Missing separate debuginfo for /usr/local/apache/modules/mod_status.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e7/ba48d34d014d6ce03af53c9b25018996651592 Missing separate debuginfo for /usr/local/apache/modules/mod_autoindex.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/12/90144c8f058247c8056f7c10d1d7cebaf41d2e Missing separate debuginfo for /usr/local/apache/modules/mod_dir.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b1/67e725fc33fca643ab2e501ec8c1b1af02b972 Missing separate debuginfo for /usr/local/apache/modules/mod_alias.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bf/c384cf34e5e8b0903202a78abf8273f2eabd32 Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/40/7f18ed95e0603bd83bee56d1f59f06291d8016 Reading symbols from /lib/libpcre.so.0.0.1...Reading symbols from /usr/lib/debug/lib/libpcre.so.0.0.1.debug...done. done. Loaded symbols for /lib/libpcre.so.0.0.1 Reading symbols from /usr/local/apache/lib/libaprutil-1.so.0...done. Loaded symbols for /usr/local/apache/lib/libaprutil-1.so.0 Reading symbols from /usr/local/apache/lib/libexpat.so.0...done. Loaded symbols for /usr/local/apache/lib/libexpat.so.0 Reading symbols from /usr/local/apache/lib/libapr-1.so.0...done. Loaded symbols for /usr/local/apache/lib/libapr-1.so.0 Reading symbols from /lib/librt-2.12.so...Reading symbols from /usr/lib/debug/lib/librt-2.12.so.debug...done. done. Loaded symbols for /lib/librt-2.12.so Reading symbols from /lib/libcrypt-2.12.so...Reading symbols from /usr/lib/debug/lib/libcrypt-2.12.so.debug...done. done. Loaded symbols for /lib/libcrypt-2.12.so Reading symbols from /lib/libpthread-2.12.so...Reading symbols from /usr/lib/debug/lib/libpthread-2.12.so.debug...done. [Thread debugging using libthread_db enabled] done. Loaded symbols for /lib/libpthread-2.12.so Reading symbols from /lib/libc-2.12.so...Reading symbols from /usr/lib/debug/lib/libc-2.12.so.debug...done. done. Loaded symbols for /lib/libc-2.12.so Reading symbols from /lib/ld-2.12.so...Reading symbols from /usr/lib/debug/lib/ld-2.12.so.debug...done. done. Loaded symbols for /lib/ld-2.12.so Reading symbols from /lib/libfreebl3.so...Reading symbols from /usr/lib/debug/lib/libfreebl3.so.debug...done. done. Loaded symbols for /lib/libfreebl3.so Reading symbols from /lib/libdl-2.12.so...Reading symbols from /usr/lib/debug/lib/libdl-2.12.so.debug...done. done. Loaded symbols for /lib/libdl-2.12.so Reading symbols from /usr/lib/libssl.so.1.0.1e...Reading symbols from /usr/lib/debug/usr/lib/libssl.so.1.0.1e.debug...done. done. Loaded symbols for /usr/lib/libssl.so.1.0.1e Reading symbols from /usr/lib/libcrypto.so.1.0.1e...Reading symbols from /usr/lib/debug/usr/lib/libcrypto.so.1.0.1e.debug...done. done. Loaded symbols for /usr/lib/libcrypto.so.1.0.1e Reading symbols from /lib/libgssapi_krb5.so.2.2...Reading symbols from /usr/lib/debug/lib/libgssapi_krb5.so.2.2.debug...done. done. Loaded symbols for /lib/libgssapi_krb5.so.2.2 Reading symbols from /lib/libkrb5.so.3.3...Reading symbols from /usr/lib/debug/lib/libkrb5.so.3.3.debug...done. done. Loaded symbols for /lib/libkrb5.so.3.3 Reading symbols from /lib/libcom_err.so.2.1...Reading symbols from /usr/lib/debug/lib/libcom_err.so.2.1.debug...done. done. Loaded symbols for /lib/libcom_err.so.2.1 Reading symbols from /lib/libk5crypto.so.3.1...Reading symbols from /usr/lib/debug/lib/libk5crypto.so.3.1.debug...done. done. Loaded symbols for /lib/libk5crypto.so.3.1 Reading symbols from /lib/libresolv-2.12.so...Reading symbols from /usr/lib/debug/lib/libresolv-2.12.so.debug...done. done. Loaded symbols for /lib/libresolv-2.12.so Reading symbols from /lib/libz.so.1.2.3...Reading symbols from /usr/lib/debug/lib/libz.so.1.2.3.debug...done. done. Loaded symbols for /lib/libz.so.1.2.3 Reading symbols from /lib/libkrb5support.so.0.1...Reading symbols from /usr/lib/debug/lib/libkrb5support.so.0.1.debug...done. done. Loaded symbols for /lib/libkrb5support.so.0.1 Reading symbols from /lib/libkeyutils.so.1.3...Reading symbols from /usr/lib/debug/lib/libkeyutils.so.1.3.debug...done. done. Loaded symbols for /lib/libkeyutils.so.1.3 Reading symbols from /lib/libselinux.so.1...Reading symbols from /usr/lib/debug/lib/libselinux.so.1.debug...done. done. Loaded symbols for /lib/libselinux.so.1 Reading symbols from /lib/libnss_files-2.12.so...Reading symbols from /usr/lib/debug/lib/libnss_files-2.12.so.debug...done. done. Loaded symbols for /lib/libnss_files-2.12.so Reading symbols from /usr/local/apache/lib/apr-util-1/apr_dbd_mysql-1.so...done. Loaded symbols for /usr/local/apache/lib/apr-util-1/apr_dbd_mysql-1.so Reading symbols from /usr/local/mysql/lib/libmysqlclient.so.18...(no debugging symbols found)...done. Loaded symbols for /usr/local/mysql/lib/libmysqlclient.so.18 Reading symbols from /lib/libm-2.12.so...Reading symbols from /usr/lib/debug/lib/libm-2.12.so.debug...done. done. Loaded symbols for /lib/libm-2.12.so Reading symbols from /usr/lib/libstdc++.so.6.0.13...Reading symbols from /usr/lib/debug/usr/lib/libstdc++.so.6.0.13.debug...done. done. Loaded symbols for /usr/lib/libstdc++.so.6.0.13 Reading symbols from /lib/libgcc_s-4.4.7-20120601.so.1...Reading symbols from /usr/lib/debug/lib/libgcc_s-4.4.7-20120601.so.1.debug...done. done. Loaded symbols for /lib/libgcc_s-4.4.7-20120601.so.1 Reading symbols from /usr/local/apache/modules/mod_authn_file.so...done. Loaded symbols for /usr/local/apache/modules/mod_authn_file.so Reading symbols from /usr/local/apache/modules/mod_authn_dbd.so...done. Loaded symbols for /usr/local/apache/modules/mod_authn_dbd.so Reading symbols from /usr/local/apache/modules/mod_authn_core.so...done. Loaded symbols for /usr/local/apache/modules/mod_authn_core.so Reading symbols from /usr/local/apache/modules/mod_authz_host.so...done. Loaded symbols for /usr/local/apache/modules/mod_authz_host.so Reading symbols from /usr/local/apache/modules/mod_authz_groupfile.so...done. Loaded symbols for /usr/local/apache/modules/mod_authz_groupfile.so Reading symbols from /usr/local/apache/modules/mod_authz_user.so...done. Loaded symbols for /usr/local/apache/modules/mod_authz_user.so Reading symbols from /usr/local/apache/modules/mod_authz_dbd.so...done. Loaded symbols for /usr/local/apache/modules/mod_authz_dbd.so Reading symbols from /usr/local/apache/modules/mod_authz_core.so...done. Loaded symbols for /usr/local/apache/modules/mod_authz_core.so Reading symbols from /usr/local/apache/modules/mod_access_compat.so...done. Loaded symbols for /usr/local/apache/modules/mod_access_compat.so Reading symbols from /usr/local/apache/modules/mod_auth_basic.so...done. Loaded symbols for /usr/local/apache/modules/mod_auth_basic.so Reading symbols from /usr/local/apache/modules/mod_socache_shmcb.so...done. Loaded symbols for /usr/local/apache/modules/mod_socache_shmcb.so Reading symbols from /usr/local/apache/modules/mod_dbd.so...done. Loaded symbols for /usr/local/apache/modules/mod_dbd.so Reading symbols from /usr/local/apache/modules/mod_reqtimeout.so...done. Loaded symbols for /usr/local/apache/modules/mod_reqtimeout.so Reading symbols from /usr/local/apache/modules/mod_filter.so...done. Loaded symbols for /usr/local/apache/modules/mod_filter.so Reading symbols from /usr/local/apache/modules/mod_mime.so...done. Loaded symbols for /usr/local/apache/modules/mod_mime.so Reading symbols from /usr/local/apache/modules/mod_log_config.so...done. Loaded symbols for /usr/local/apache/modules/mod_log_config.so Reading symbols from /usr/local/apache/modules/mod_env.so...done. Loaded symbols for /usr/local/apache/modules/mod_env.so Reading symbols from /usr/local/apache/modules/mod_headers.so...done. Loaded symbols for /usr/local/apache/modules/mod_headers.so Reading symbols from /usr/local/apache/modules/mod_setenvif.so...done. Loaded symbols for /usr/local/apache/modules/mod_setenvif.so Reading symbols from /usr/local/apache/modules/mod_version.so...done. Loaded symbols for /usr/local/apache/modules/mod_version.so Reading symbols from /usr/local/apache/modules/mod_ssl.so...done. Loaded symbols for /usr/local/apache/modules/mod_ssl.so Reading symbols from /usr/local/apache/modules/mod_unixd.so...done. Loaded symbols for /usr/local/apache/modules/mod_unixd.so Reading symbols from /usr/local/apache/modules/mod_status.so...done. Loaded symbols for /usr/local/apache/modules/mod_status.so Reading symbols from /usr/local/apache/modules/mod_autoindex.so...done. Loaded symbols for /usr/local/apache/modules/mod_autoindex.so Reading symbols from /usr/local/apache/modules/mod_dir.so...done. Loaded symbols for /usr/local/apache/modules/mod_dir.so Reading symbols from /usr/local/apache/modules/mod_alias.so...done. Loaded symbols for /usr/local/apache/modules/mod_alias.so Core was generated by `/usr/local/apache/sbin/httpd -X'. Program terminated with signal 11, Segmentation fault. #0 0x0062ee90 in ?? () (gdb) bt full #0 0x0062ee90 in ?? () No symbol table info available. #1 0x00ffe2e5 in int_free_ex_data (class_index=10, obj=0x8d15d50, ad=0x8d15d68) at ex_data.c:522 mx = 2 i = <value optimized out> item = 0x8d3b6f0 ptr = <value optimized out> storage = 0x8d15eb8 #2 0x00ffdc84 in CRYPTO_free_ex_data (class_index=10, obj=0x8d15d50, ad=0x8d15d68) at ex_data.c:592 No locals. #3 0x01096977 in x509_cb (operation=3, pval=0xbfd7b100, it=0x1171bb8, exarg=0x0) at x_x509.c:113 ret = 0x8d15d50 #4 0x0109bd9c in asn1_item_combine_free (pval=<value optimized out>, it=<value optimized out>, combine=0) at tasn_fre.c:173 tt = <value optimized out> seqtt = <value optimized out> ef = <value optimized out> cf = <value optimized out> aux = <value optimized out> asn1_cb = 0x1096870 <x509_cb> i = <value optimized out> #5 0x0109bfc9 in ASN1_item_free (val=0x8d15d50, it=0x1171bb8) at tasn_fre.c:71 No locals. #6 0x01096b28 in X509_free (a=0x8d15d50) at x_x509.c:141 No locals. #7 0x004684b3 in SSL_load_client_CA_file (file=0x8d41fb8 "/usr/local/ssl/private/tismca.crt") at ssl_cert.c:726 in = 0x8d16c68 x = 0x8d15d50 xn = <value optimized out> ret = 0x8d16298 sk = 0x8d16828 #8 0x0088cc85 in ssl_init_PushCAList (ca_list=0x8d38b68, s=0x8d41658, ptemp=0x8d0c9c8, file=0x8d41fb8 "/usr/local/ssl/private/tismca.crt") at ssl_engine_init.c:1510 n = <value optimized out> sk = <value optimized out> #9 0x0088ce2f in ssl_init_FindCAList (s=0x8d41658, ptemp=0x8d0c9c8, ca_file=0x8d41fb8 "/usr/local/ssl/private/tismca.crt", ca_path=0x0) at ssl_engine_init.c:1560 ca_list = 0x8d38b68 #10 0x0088deb4 in ssl_init_ctx_verify (s=0x8d41658, p=0x8ce00a8, ptemp=0x8d0c9c8, mctx=0x8d3cd28) at ssl_engine_init.c:596 ctx = 0x8d38180 verify = <value optimized out> ca_list = <value optimized out> #11 ssl_init_ctx (s=0x8d41658, p=0x8ce00a8, ptemp=0x8d0c9c8, mctx=0x8d3cd28) at ssl_engine_init.c:785 rv = 147938640 #12 0x0088ea21 in ssl_init_server_ctx (s=0x8d41658, p=0x8ce00a8, ptemp=0x8d0c9c8, sc=0x8d3cc48, pphrases=0x8d492c8) at ssl_engine_init.c:1290 rv = <value optimized out> ---Type <return> to continue, or q <return> to quit--- #13 ssl_init_ConfigureServer (s=0x8d41658, p=0x8ce00a8, ptemp=0x8d0c9c8, sc=0x8d3cc48, pphrases=0x8d492c8) at ssl_engine_init.c:1385 rv = 148117080 #14 0x008900a6 in ssl_init_Module (p=0x8ce00a8, plog=0x8d069b0, ptemp=0x8d0c9c8, base_server=0x8d04ee8) at ssl_engine_init.c:228 mc = <value optimized out> sc = <value optimized out> s = 0x8d41658 rv = <value optimized out> pphrases = 0x8d492c8 #15 0x0808a1b7 in ap_run_post_config (pconf=0x8ce00a8, plog=0x8d069b0, ptemp=0x8d0c9c8, s=0x8d04ee8) at config.c:106 pHook = <value optimized out> n = <value optimized out> rv = 0 #16 0x0806a5f9 in main (argc=2, argv=0xbfd7b5d4) at main.c:765 c = 88 'X' showcompile = 0 showdirectives = 147710256 confname = 0x80aaf8c "/usr/local/etc/httpd/httpd.conf" def_server_root = 0x80aa718 "/usr/local/apache" temp_error_log = 0x0 error = <value optimized out> process = 0x8cde130 pconf = 0x8ce00a8 plog = 0x8d069b0 ptemp = 0x8d0c9c8 pcommands = 0x8d02130 opt = 0x8d021d0 rv = <value optimized out> mod = <value optimized out> opt_arg = 0x0 signal_server = <value optimized out> (gdb) (gdb) quit