Bug 62447 - apache 2.4.33+ssl fail to start on Solaris 11
Summary: apache 2.4.33+ssl fail to start on Solaris 11
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Runtime Config (show other bugs)
Version: 2.4.33
Hardware: Sun Solaris
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Depends on:
Reported: 2018-06-12 07:21 UTC by Shital Patil
Modified: 2018-08-21 10:38 UTC (History)
0 users

truss -Ff ./apachectl start (689.43 KB, text/plain)
2018-06-12 07:25 UTC, Shital Patil

Note You need to log in before you can comment on or make changes to this bug.
Description Shital Patil 2018-06-12 07:21:56 UTC

Comment 1 Shital Patil 2018-06-12 07:25:53 UTC
Created attachment 35963 [details]
truss -Ff ./apachectl start

truss also doesnot give much information on start failure
Comment 2 Shital Patil 2018-06-12 07:40:21 UTC
This issue is reproducible only at our customer's system.
It works fine on our in-house solaris system.

Output of ./apachectl -V
Server version: Apache/2.4.33 (Unix)
Server built:   Apr  9 2018 06:29:39
Server's Module Magic Number: 20120211:76
Server loaded:  APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
Comment 3 Shital Patil 2018-06-12 10:42:24 UTC
MDB stack as below
> ::stack 100
0xffffffff78a2ee58(ffffffff7fffebf0, ffffffffffffffff, 1e, ffffffff7e5a9858, 74, ffffffff7e6e4650)
libcrypto.so.1.0.0`ERR_get_state+0x30(ffffffff7fffec00, 1, 1298, 1000, 12b8, 1000)
libcrypto.so.1.0.0`ERR_clear_error+4(100248a40, 8000, 0, 1000, 1000, 1040)
libcrypto.so.1.0.0`ENGINE_load_builtin_engines+0x14(1, ffffffff7e867d68, 1001894a0, 1001888e0, 38, 101010101010101)
0xffffffff7681103c(4, 1001ce328, 1001cc318, 0, 100254ee0, 100254f00)
ap_run_pre_config+0x60(1001a0158, 1001ce328, 1001cc318, 1e0, 1, 210)
main+0xa9c(1001a0158, 10018ecd8, 100078000, 10018ec00, 100192400, 100191800)
_start+0x7c(0, 0, 0, 0, 0, 0)
Comment 4 Rainer Jung 2018-06-12 12:24:54 UTC
You truss output indicates a segmentation fault. Do you get a core file, maybe after adjusting coreadm? If you get a core file, what does "pstack /path/to/my/core" show?
Comment 5 Shital Patil 2018-06-12 12:28:34 UTC
(In reply to Rainer Jung from comment #4)
> You truss output indicates a segmentation fault. Do you get a core file,
> maybe after adjusting coreadm? If you get a core file, what does "pstack
> /path/to/my/core" show?

Please see comment 3 for mdb stack
Comment 6 Yann Ylavic 2018-06-12 12:41:31 UTC
Is the openssl library version linked at runtime (customer's system) the same as the one used at build time (your system?) ?
Comment 7 Rainer Jung 2018-06-12 12:57:00 UTC
I had hoped for additional resp. complementary info from pstack.
Comment 8 Shital Patil 2018-06-12 13:03:54 UTC
We bundle openssl with apache itself so openssl libraries are same.

But below are the one's which are linked at runtime
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
**      libucrypto.so.1 =>       /lib/64/libucrypto.so.1
        libelf.so.1 =>   /lib/64/libelf.so.1
**      libcryptoutil.so.1 =>    /lib/64/libcryptoutil.so.1
        libz.so.1 =>     /lib/64/libz.so.1

Would libcryptoutil or libucrypto be the cause?
Comment 9 Shital Patil 2018-08-21 05:01:41 UTC
We found that this issue was caused due to compatibility of openssl library. Using solaris system's library resolves the issue.

Our apache server ships : OpenSSL 1.0.2o : HTTPD not starting in SSL mode
Solaris system library  : OpenSSL 1.0.2k : Moved to lover version provided by Solaris itself. Working fine and HTTPD comes up well.

So copying libssl.so.1.0.0 and libcrypto.so.1.0.0 from lib/64(or other place from solaris) to apache/openssl/lib.

Now the question remains is why HTTPD doesnot report anything in logs.
Comment 10 Eric Covener 2018-08-21 10:38:56 UTC
Early crashes in the parent won't write to httpds logs.