Bug 56091

Summary: SSL handshake error when httpd compiled on Oracle T4 system
Product: Apache httpd-2 Reporter: Sean Timmins <stimmins>
Component: mod_sslAssignee: Apache HTTPD Bugs Mailing List <bugs>
Severity: normal Keywords: MassUpdate
Priority: P2    
Version: 2.2.26   
Target Milestone: ---   
Hardware: Sun   
OS: Solaris   

Description Sean Timmins 2014-01-30 13:13:02 UTC
Compiling httpd v2.2.x (have tried 2.2.11, 2.2.22 and 2.2.26) on an Oracle T4 we get an occational SSL error in a seemingly random fashion.

I have tried a couple of different versions of openssl as well.

If I compile on an Oracle SPARC V445 and then copy mod_ssl.so into the install compiled on the T4 the the error does not occur.

The differences in the source tree after running the same configure on both machines are minor. Consisting of

1) config.log and config.status files 
2) HAVE_FDATASYNC is set to 1 in srclib/apr/include/arch/unix/apr_private.h on the T4 compile
3) A minor difference in srclib/apr-util/include/apu.h, srclib/apr-util/include/private/apu_config.h and srclib/apr-util/Makefile because an sqlite3 compatible library is on one of the machines and not the other.

gcc version used v3.4.6 (without GNU binutils)
openssl versions tried 0.9.8 and 1.0.0
OS: Solaris 10

From the error log we do not usually get any entries at all. In testing we sometimes get the following line, but there is no evidence it is related.

"[warn] (45)Deadlock situation detected/avoided: Failed to acquire SSL session cache lock"

It is possible the error occurs more often during higher load, as it happenes very rarely under low load, but this eveluation is more than a little subjective at the moment.

The SSL error as seen in the browser is:

"An error occurred during a connection to sp.onlinelibrary.wiley.com. SSL received an unexpected New Session Ticket handshake message. (Error code: ssl_error_rx_unexpected_new_session_ticket)

 The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
  Please contact the web site owners to inform them of this problem. Alternatively, use the command found in the help menu to report this broken site."

I realise this may not be an apache httpd problem, but I have to start somewhere.
Comment 1 Jeff Trawick 2014-01-30 13:26:43 UTC
There are a lot of variations and more than one symptom.  I don't have a magic answer, but some hints below should help eliminate some of the search space.
Unless you have third-party modules that use apr_file_datasync(), this won't affect httpd.
> sqlite3
I think you would have had to explicitly configure the use of such a db in httpd (e.g., for session cache) for this to have made a difference.
You can eliminate this variation by adding "--without-sqlite3" to the configure options.
> (45)Deadlock situation detected/avoided: Failed to acquire SSL session cache lock
You need to configure a different SSL mutex.  Unless you have child process crashes (messages with "exit signal" in your error log), set "SSLMutex pthread".  This happens due to a poor default mutex choice for Solaris (fixed in apr 1.3.10) *OR* from your configuration of "SSLMutex fcntl".  (Or I'm confused.)  If you have process crashes ("exit signal"), you need to get cores and backtraces.  Please let us know if that is happening.
Perhaps somebody recognizes the session ticket error.
What level of APR is being compiled with and used at runtime?  (This is displayed from "httpd -V".)
Comment 2 Sean Timmins 2014-01-30 14:28:04 UTC
The version os of APR are those that come with the particular version of the source. From the outpout of httpd -V:

httpd v2.2.11 -> apr v1.3.3, apr-util v1.3.4
httpd v2.2.22 -> apr v1.4.5, apr-util v1.4.1
httpd v2.2.26 -> apr v1.4.8, apr-util v1.5.2

The configure line used to build all three versions was:

./configure \
    --prefix=/apps/wps/common/httpd-2.2.26 \
    --enable-so \
    --with-mpm=worker \
    --enable-cgid \
    --with-ssl=/usr/local/ssl \
    --enable-mods-shared="all ssl cache proxy authn_alias mem_cache file_cache charset_lite dav_lock disk_cache"

But no other changes to the configure environment were made. We don't actually use sqlite3.

We are getting both "exit signal Bus error (10)" and "exit signal Segmentation fault (11)" but no cores at the moment, I'll find out what is happening to them and get a backtrace. We are only getting these on the production server which is running the shibboleth apache module.

"SSLMutex pthread" got rid of the Deadlock warning on the server we are testing this on (which is not producing any cores). And I can generate the SSL error without any core messages in the error log on the production server.

I'll continue digging
Comment 3 Rainer Jung 2014-01-30 15:25:55 UTC
Since you have a working setup with the same httpd version on a v445, you could build the webserver on that system against the same OpenSSL-Version as on the T4. That would allow you to simply swap the OpenSSL binary library file (so file) on the T4 and replace it with the file from the V445 to check, whether the OpenSSL lib itself makes the difference (which I suspect).


Comment 4 William A. Rowe Jr. 2018-11-07 21:08:51 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.