Bug 60441

Summary: SSL handshake failed
Product: Apache httpd-2 Reporter: Tiziano <tiziano.digennaro>
Component: AllAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: critical CC: tiziano.digennaro
Priority: P2 Keywords: MassUpdate
Version: 2.2.31   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description Tiziano 2016-12-05 08:50:44 UTC
In our production system we have a custom compiled version of apache 2.2.31 on a debian machine with OpenSSL 1.0.1u

Server version: Apache/2.2.31 (Unix)
Server built:   Oct 31 2016 13:51:59
Server's Module Magic Number: 20051115:40
Server loaded:  APR 1.5.2, APR-Util 1.5.4
Compiled using: APR 1.5.2, APR-Util 1.5.4
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/loadbalancer/apache"
 -D SUEXEC_BIN="/opt/loadbalancer/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Sometimes during high load we got from the client and ssl handshake failure.
I was able to reproduce the bug with apache banchmark. It always happens above 4 as concurrency level (ab -c 5 -n 2000 -p post.body)
Here the tcpdump that show after the Change Cipher Spec it always happens this issue.
12	0.001043	193.33.xxx.xxx	10.xxx.xxx.xxx	TCP	68	10498→443 [SYN] Seq=0 Win=8192 Len=0 MSS=1420 WS=256 SACK_PERM=1
26	0.016894	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TCP	62	10498→443 [ACK] Seq=1 Ack=1 Win=66560 Len=0
35	0.020474	193.xxx.xxx.xxx	10.xxx.xxx.xxx TLSv1.2	573	Client Hello
49	0.042871	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TCP	62	10498→443 [ACK] Seq=518 Ack=2841 Win=66560 Len=0
50	0.043990	193.xxx.xxx.xxx 10.xxx.xxx.xxx	TCP	62	10498→443 [ACK] Seq=518 Ack=4873 Win=66560 Len=0
54	0.045390	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TLSv1.2	182	Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
89	0.091799	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TLSv1.2	412	Application Data
109	0.131868	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TCP	62	10498→443 [ACK] Seq=1000 Ack=5936 Win=65536 Len=0
110	0.132318	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TLSv1.2	87	Encrypted Alert
111	0.132563	193.xxx.xxx.xxx	10.xxx.xxx.xxx	TCP	62	10498→443 [FIN, ACK] Seq=1031 Ack=5936 Win=65536 Len=0

I always got and "Content Type: Alert (21)" as Encrypted Alert.
I'm not able to got any error message on the apache itselfe, so could be a problem in openssl itselfe
Comment 1 Tiziano 2016-12-05 12:12:24 UTC
Heres the ssl option that currently i'm using

SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLPassPhraseDialog  builtin
SSLSessionCache        shmcb:/var/run/apache/ssl_scache(1024000)
SSLSessionCacheTimeout  300
SSLMutex  file:/var/run/apache/ssl_mutex


SSLEngine on
SSLProtocol -ALL +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
SSLHonorCipherOrder On
SSLCertificateFile file
SSLCertificateKeyFile file
SSLCACertificateFile file
SSLOptions +StdEnvVars
SSLProxyEngine			On
Comment 2 Eric Covener 2016-12-06 17:27:09 UTC
You'll have to find some more data here -- error log, some optional feature that casues the issue, or some sign of a regression.
Comment 3 William A. Rowe Jr. 2018-11-07 21:09:17 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.