Bug 62447

Created attachment 35963 [details]
truss -Ff ./apachectl start

truss also doesnot give much information on start failure

This issue is reproducible only at our customer's system.
It works fine on our in-house solaris system.

Output of ./apachectl -V
Server version: Apache/2.4.33 (Unix)
Server built:   Apr  9 2018 06:29:39
Server's Module Magic Number: 20120211:76
Server loaded:  APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/PTCAPACHEBUILD"
 -D SUEXEC_BIN="/PTCAPACHEBUILD/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

MDB stack as below
> ::stack 100
0xffffffff78a2ee58(ffffffff7fffebf0, ffffffffffffffff, 1e, ffffffff7e5a9858, 74, ffffffff7e6e4650)
libcrypto.so.1.0.0`ERR_get_state+0x30(ffffffff7fffec00, 1, 1298, 1000, 12b8, 1000)
libcrypto.so.1.0.0`ERR_clear_error+4(100248a40, 8000, 0, 1000, 1000, 1040)
libcrypto.so.1.0.0`ENGINE_load_builtin_engines+0x14(1, ffffffff7e867d68, 1001894a0, 1001888e0, 38, 101010101010101)
0xffffffff7681103c(4, 1001ce328, 1001cc318, 0, 100254ee0, 100254f00)
ap_run_pre_config+0x60(1001a0158, 1001ce328, 1001cc318, 1e0, 1, 210)
main+0xa9c(1001a0158, 10018ecd8, 100078000, 10018ec00, 100192400, 100191800)
_start+0x7c(0, 0, 0, 0, 0, 0)

You truss output indicates a segmentation fault. Do you get a core file, maybe after adjusting coreadm? If you get a core file, what does "pstack /path/to/my/core" show?

(In reply to Rainer Jung from comment #4)
> You truss output indicates a segmentation fault. Do you get a core file,
> maybe after adjusting coreadm? If you get a core file, what does "pstack
> /path/to/my/core" show?

Please see comment 3 for mdb stack

Is the openssl library version linked at runtime (customer's system) the same as the one used at build time (your system?) ?

I had hoped for additional resp. complementary info from pstack.

We bundle openssl with apache itself so openssl libraries are same.

But below are the one's which are linked at runtime
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
**      libucrypto.so.1 =>       /lib/64/libucrypto.so.1
        libelf.so.1 =>   /lib/64/libelf.so.1
**      libcryptoutil.so.1 =>    /lib/64/libcryptoutil.so.1
        libz.so.1 =>     /lib/64/libz.so.1

Would libcryptoutil or libucrypto be the cause?

We found that this issue was caused due to compatibility of openssl library. Using solaris system's library resolves the issue.

Our apache server ships : OpenSSL 1.0.2o : HTTPD not starting in SSL mode
Solaris system library  : OpenSSL 1.0.2k : Moved to lover version provided by Solaris itself. Working fine and HTTPD comes up well.

So copying libssl.so.1.0.0 and libcrypto.so.1.0.0 from lib/64(or other place from solaris) to apache/openssl/lib.


Now the question remains is why HTTPD doesnot report anything in logs.

Early crashes in the parent won't write to httpds logs.