Bug 55148

Summary: Error during SSL Handshake with remote server
Product: Apache httpd-2 Reporter: Allen Zhao <allen.zhao>
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: normal CC: sbhanwra18
Priority: P2 Keywords: MassUpdate
Version: 2.2.24   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description Allen Zhao 2013-06-26 20:20:10 UTC
We upgrade our apache from 2.2.17 to 2.2.24. We use the same setting. However, we keep getting 502 bad gateway issue.

I tried following settings as well, but no luck.
    SSLProxyCACertificateFile /work/users/infra/proxy/proxyCA.crt
    SSLProxyMachineCertificateFile /work/users/infra/proxy/lp97643.pem
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerExpire off

I have verified by proxyCA with curl, it works fine.

I struggled with this issue for couple of weeks. I doubt this might be new bug.

Thanks a lot,

The error log:
[Wed Jun 26 19:08:35 2013] [error] (502)Unknown error 502: proxy: pass request body failed to 142.63.42.254:443 
[Wed Jun 26 19:08:35 2013] [error] [client 192.168.156.135] proxy: Error during SSL Handshake with remote server returned by /Offline/, referer: https://abc.xyz.com/Offline/
[Wed Jun 26 19:08:35 2013] [error] proxy: pass request body failed to 142.63.42.254:443 from 192.168.156.135 ()


The config:
NameVirtualHost *:50211
<VirtualHost *:50211>
    ServerAdmin admin@example.com
    DocumentRoot "/work/users/infra/proxy/PR_Offline_https/htdocs"
    <Directory "/work/users/infra/proxy/PR_Offline_https/htdocs">
        Allow from all
    </Directory>
    SSLEngine on
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
    SSLProxyEngine on
    SSLCertificateFile      /work/users/infra/proxy/lp97643.crt
    SSLCertificateKeyFile   /work/users/infra/proxy/lp97643.key
    RequestHeader set X-Authenticated-User %{REMOTE_USER}e
    ProxyRequests On
    ProxyVia On
    ProxyPreserveHost On
    ProxyPass /Offline http://142.63.42.254/Offline/
    ProxyPassReverse /Offline http://142.63.42.254/OfflineS/
    BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
    SetEnv force-proxy-request-1.0 1
    SetEnv proxy-nokeepalive 1
</VirtualHost>

The compile settings:

bin/httpd -V
Server version: Apache/2.2.24 (Unix)
Server built:   May 21 2013 14:49:46
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.6, APR-Util 1.4.1
Compiled using: APR 1.4.6, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/apps/infra/apache/2.2.24"
 -D SUEXEC_BIN="/apps/infra/apache/2.2.24/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
Comment 1 Allen Zhao 2013-06-26 20:27:37 UTC
I also built openssl from source version: 1.0.1e
Comment 2 Allen Zhao 2013-06-27 14:13:44 UTC
I rebuilt apache with openssl 1.0.0d. it works.

Any idear?

Thx a lot,
Comment 3 Eric Covener 2013-06-27 14:36:54 UTC
Can you validate a connection to the backend server with openssl s_client between the two builds?
Comment 4 Allen Zhao 2013-09-03 20:07:04 UTC
1.0.0d works fine.

1.0.1e: it doesn't read anything from stdin (e.g. enter a http request)

I got same issue with 2.2.25/1.0.1e.

2.2.25/1.0.0d works fine. This looks OpenSSL related.

bin/openssl s_client -host 172.23.199.200 -port 443
CONNECTED(00000003)

write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 321 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
Comment 5 Allen Zhao 2013-09-03 20:17:15 UTC
Here is part of the output from 1.0.0d:

bin/openssl s_client -host 172.23.199.200 -port 443
CONNECTED(00000003)
depth=1 C = CA, ST = Ontario, L = Toronto, O = TELUS, OU = Application Infrastructure, CN = www.telus.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=telusmobility.tmi.telus.com
   i:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
 1 s:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
   i:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDIDCCAomgAwIBAgIDEADpMA0GCSqGSIb3DQEBBQUAMH4xCzAJBgNVBAYTAkNB
MRAwDgYDVQQIEwdPbnRhcmlvMRAwDgYDVQQHEwdUb3JvbnRvMQ4wDAYDVQQKEwVU
RUxVUzEjMCEGA1UECxMaQXBwbGljYXRpb24gSW5mcmFzdHJ1Y3R1cmUxFjAUBgNV
BAMTDXd3dy50ZWx1cy5jb20wHhcNMDkwMzAyMTYxNTA4WhcNMTQwMzAxMTYxNTA4
WjCBjDELMAkGA1UEBhMCQ0ExEDAOBgNVBAgTB09udGFyaW8xEDAOBgNVBAcTB1Rv
cm9udG8xDjAMBgNVBAoTBVRFTFVTMSMwIQYDVQQLExpBcHBsaWNhdGlvbiBJbmZy
YXN0cnVjdHVyZTEkMCIGA1UEAxMbdGVsdXNtb2JpbGl0eS50bWkudGVsdXMuY29t
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCUi+ni+nyqZUugOOmkFovxmYi8
N/bARsvInCK6bXzYDhQQJr6+NtX5LytUDhDYHQXQVAL9Lm0wtOcNsrFLjEukvPuc
SI2Fr/4HWajIIA6uebFPIz5AvJh7jVYsDvv8/XcC+VgBh2TeJtSEZ5sDLj6mEF2y
ThAhffEHS3Ye5Ij+5wIDAQABo4GcMIGZMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEB
BAQDAgXgMAsGA1UdDwQEAwIF4DAsBglghkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5l
cmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFI3c2E1SsZBpaTYZOd3y1yUdWZTH
MB8GA1UdIwQYMBaAFFmwVgMX8YForRj3nr1434gfsUwaMA0GCSqGSIb3DQEBBQUA
A4GBABb/lsKiK42488U0w3oJuRJdIl6IRdmZTCp223tFpFihc0Se7jeiJeHKPniq
eNFnuKWbc52wiyLk98q313vp3DV3wfAafcYo77KwiXVPx9PUFcsJvZb9kJ8M6CxG
vLwicZg0gZFHegQRUkyKeYJHTcLbRZOyR5huX3gqklPJYlhm
-----END CERTIFICATE-----
subject=/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=telusmobility.tmi.telus.com
issuer=/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
---
No client certificate CA names sent
---
SSL handshake has read 1824 bytes and written 392 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID: 7549FF55BA4A41504A7E0C5AC261BC44BEFAA5E9CBEF366D7213C9A0DF2147BD
    Session-ID-ctx:
    Master-Key: 2D0F124D2315E89C48F4DD3573B1985716C56C90C4D6D723CB35701C0F0EA31AF47C84D3B772EC6DCD669A3D008C0771
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1378238532
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
GET /

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Comment 6 Allen Zhao 2013-09-03 20:23:56 UTC
For 1.0.1e, if I add -ssl3, it works:

bin/openssl s_client -host 172.23.199.200 -port 443 -ssl3
CONNECTED(00000003)
depth=1 C = CA, ST = Ontario, L = Toronto, O = TELUS, OU = Application Infrastructure, CN = www.telus.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=telusmobility.tmi.telus.com
   i:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
 1 s:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
   i:/C=CA/ST=Ontario/L=Toronto/O=TELUS/OU=Application Infrastructure/CN=www.telus.com
---
Server certificate
-----BEGIN CERTIFICATE-----
...
Comment 7 Allen Zhao 2013-09-05 05:37:55 UTC
This seems caused by TLSV1.2.

I solve the problem by adding following line to httpd.conf.

SSLProxyProtocol +SSLv2 +SSLv3 +TLSv1 +TLSv1.1

Thanks a lot,

Allen
Comment 8 William A. Rowe Jr. 2018-11-07 21:08:22 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.