Bug 55782 - ProxyPass'ing to HTTPS server via proxypass creates SNI failure
Summary: ProxyPass'ing to HTTPS server via proxypass creates SNI failure
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.4.6
Hardware: All All
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-11-15 12:42 UTC by Andre W.
Modified: 2017-07-25 19:03 UTC (History)
0 users



Attachments
Don't reuse a SSL backend connection whose SNI differs (4.42 KB, patch)
2014-02-21 12:47 UTC, Yann Ylavic
Details | Diff
Backport to 2.2.x (3.51 KB, patch)
2014-02-28 14:16 UTC, Yann Ylavic
Details | Diff
Latest backport to 2.2.x proposal (3.47 KB, patch)
2014-02-28 15:01 UTC, Yann Ylavic
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andre W. 2013-11-15 12:42:14 UTC
I tried to pass an HTTPS request from an apache 2.2.25 to another apache 2.2.25, which also runs on https. Sometimes the second apache prints the follwoing failure 

[error] Hostname localhost provided via SNI and hostname abc.com provided via HTTP are different

and a 400 Bad Request failure is displayed inside the browser. The curious think is that this only happens in 10-15% of the requests!?

The first apache is configured the following way:

SSLProxyEngine On
ProxyPreserveHost On

ProxyPass / https://abc.com/
ProxyPassReverse / https://abc.com/

The servername is set via start parameter "-C 'ServerName abc.com'". The only option, which completly solved the problem is to disable TSLv1.x for the backend communication via setting "SSLProxyProtocol SSLv3".

It seems that the failure is based on the following changes in apache 2.2.25 with the activation of SNI for mod_proxy:

„
  *) mod_ssl/proxy: enable the SNI extension for backend TLS connections
     [Kaspar Brand]

  *) mod_proxy: Use the the same hostname for SNI as for the HTTP request when
     forwarding to SSL backends. PR 53134.
     [Michael Weiser <michael weiser.dinsnail.net>, Ruediger Pluem]
„

http://svn.apache.org/repos/asf/httpd/httpd/branches/2.2.x/CHANGES
Comment 1 Kaspar Brand 2013-11-26 05:32:35 UTC
Do the requests to the front end server differ in the Host: header they include? If so, can you append "disablereuse=on" to the ProxyPass statement and see if it makes a difference?
Comment 2 Andre W. 2013-12-02 08:57:18 UTC
Hello Kaspar,

there are no changes inside the header from my side or from the apache configuration.

The second apache prints the following error message everytime, when the failure comes up:

[Mon Dec 02 09:48:55 2013] [error] Hostname localhost provided via SNI and hostname abc.com provided via HTTP are different

The curious think i asked myself is, why does the apache set "localhost" as SNI and not abc.com like he transmits inside the header? The servername of the first apache is abc.com.

Best regards,
André
Comment 3 Kaspar Brand 2013-12-03 05:17:31 UTC
By "requests to the front end server", I only meant those coming in on that server, not those leaving it (i.e. the requests to the backend). I assume that currently, your frontend receives requests for both its FQDN ("abc.com") and for "localhost".

Perhaps you're misunderstanding what ProxyPreserveHost is for. It mostly doesn't make sense to turn it on when proxying SSL URLs (see e.g this thread here: https://mail-archives.apache.org/mod_mbox/httpd-dev/201304.mbox/%3CCD8A0375.2AAF0%25eugenel@amazon.com%3E).

Did you already try if "disablereuse=on" (see comment 1) makes these errors go away? (Alternatively, you can simply drop the "ProxyPreserveHost On" directive - which would actually be the better fix, unless you're trying to do some kind of "proxied mass name-based virtual hosting", as mentioned in the documentation.)
Comment 4 Andre W. 2013-12-03 11:43:36 UTC
I did the test by multiple ways, via jMeter or browser and everytime i called the frontend-server with the same URL, also the misleading failures were gone after i disabled TSL 1.2 for the backend communication.

I also tested the option "disablereuse=on" (sorry, i forgot to mention that in my previous post), but i had the same issue again. 

I'm also aware about the topic/bug of "SSLProxyCheckPeerCN / ProxyPreserveHost" and we definitly don't have such an option running, also this is not the problem in that case.

Best regards,
André
Comment 5 Kaspar Brand 2013-12-04 08:14:44 UTC
(In reply to Andre W. from comment #4)
> I did the test by multiple ways, via jMeter or browser and everytime i
> called the frontend-server with the same URL, also the misleading failures
> were gone after i disabled TSL 1.2 for the backend communication.

If disabling TLSv1.2 (as opposed to TLSv1 or TLSv1.1) makes the issue disappear, then it's probably not strictly related to SNI.

> I'm also aware about the topic/bug of "SSLProxyCheckPeerCN /
> ProxyPreserveHost" and we definitly don't have such an option running, also
> this is not the problem in that case.

In your original report (from 15 November), you're quoting from your "first apache" configuration, which includes "ProxyPreserveHost On", so I wonder why you're now saying that you "definitly don't have such an option running". The fact that "localhost" is put into the SNI extension for connections going to the backend is basically due to the following code:

http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?revision=1497470&view=markup#l2028

(To confirm what mod_ssl puts into the SNI extension, set the LogLevel to debug - see http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/ssl/ssl_engine_io.c?revision=1501712&view=markup#l1074)
Comment 6 Andre W. 2013-12-04 09:24:18 UTC
My comment is a little bit misleading, what I meant is that i don't run the combination of "ProxyPresrveHost On" and "SSLProxyCheckPeerCN On" at the same time.

I will check that that an come back here as soon as I have new information.

Best regards,
André
Comment 7 Andre W. 2013-12-04 14:30:56 UTC
I have run the test again, on debug level and found out the following.

In this case it works (SNi is set to jee-eval1.abc.com):
[Wed Dec 04 15:12:33 2013] [debug] mod_proxy_http.c(1974): proxy: HTTP: serving URL https://jee-eval2.abc.com/clusterjsp/
[Wed Dec 04 15:12:33 2013] [debug] proxy_util.c(2018): proxy: HTTPS: has acquired connection for (jee-eval2.abc.com)
[Wed Dec 04 15:12:33 2013] [debug] proxy_util.c(2074): proxy: connecting https://jee-eval2.abc.com/clusterjsp/ to jee-eval2.abc.com:443
[Wed Dec 04 15:12:33 2013] [debug] proxy_util.c(2200): proxy: connected /clusterjsp/ to jee-eval2.abc.com:443
[Wed Dec 04 15:12:33 2013] [debug] proxy_util.c(2451): proxy: HTTPS: fam 2 socket created to connect to jee-eval2.abc.com
[Wed Dec 04 15:12:33 2013] [debug] proxy_util.c(2583): proxy: HTTPS: connection complete to 160.50.128.170:443 (jee-eval2.abc.com)
[Wed Dec 04 15:12:33 2013] [info] [client 160.50.128.170] Connection to child 0 established (server jee-eval1.abc.com:443)
[Wed Dec 04 15:12:33 2013] [info] Seeding PRNG with 512 bytes of entropy
[Wed Dec 04 15:12:33 2013] [debug] ssl_engine_io.c(1087): [client 160.50.128.170] SNI extension for SSL Proxy request set to 'jee-eval1.abc.com'

In this case it didn't works (SNI is set to localhost):
[Wed Dec 04 15:22:02 2013] [debug] ssl_engine_kernel.c(1884): OpenSSL: Read: SSL negotiation finished successfully
[Wed Dec 04 15:22:02 2013] [debug] proxy_util.c(2074): proxy: connecting https://jee-eval2.abc.com/ to jee-eval2.abc.com:443
[Wed Dec 04 15:22:02 2013] [debug] proxy_util.c(2200): proxy: connected / to jee-eval2.abc.com:443
[Wed Dec 04 15:22:02 2013] [debug] proxy_util.c(2451): proxy: HTTPS: fam 2 socket created to connect to jee-eval2.abc.com
[Wed Dec 04 15:22:02 2013] [debug] proxy_util.c(2583): proxy: HTTPS: connection complete to 160.50.128.170:443 (jee-eval2.abc.com)
[Wed Dec 04 15:22:02 2013] [info] [client 160.50.128.170] Connection to child 0 established (server jee-eval1.abc.com:443)
[Wed Dec 04 15:22:02 2013] [info] Seeding PRNG with 512 bytes of entropy
[Wed Dec 04 15:22:02 2013] [debug] ssl_engine_io.c(1087): [client 160.50.128.170] SNI extension for SSL Proxy request set to 'localhost'
[Wed Dec 04 15:22:02 2013] [debug] ssl_engine_kernel.c(1871): OpenSSL: Handshake: start

But this is not really constant in the behaviour at this point, so if he would set localhost everytime i would understand that.
Comment 8 Kaspar Brand 2013-12-05 05:55:15 UTC
(In reply to Andre W. from comment #7)
> But this is not really constant in the behaviour at this point, so if he
> would set localhost everytime i would understand that.

As mentioned before, this depends on what Host: header is found in the incoming requests. I recommend logging %{Host}i for the requests coming in on the "first apache", and correlating them with the proxy debug logs. Chances are very, very high that "localhost" is put into the SNI extension for those requests with a "Host: localhost" HTTP header.
Comment 9 Andre W. 2013-12-09 10:30:15 UTC
So, i turned up the debug level once more and added some additional information into the header and i got still bad requests with an 400 code:

IE it happens more often, then on FF or Chrome, but maybe this was only a lucks constellation.

IE:
10.145.12.69 jee-eval1.abc.com - [09/Dec/2013:10:53:59 +0100] "GET /clusterjsp/ HTTP/1.1" 400 347 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; InfoPath.2; Tablet PC 2.0)" "jee-eval1.abc.com"

FF:
10.145.12.69 jee-eval1.abc.com - [09/Dec/2013:10:55:20 +0100] "GET /clusterjsp/ HTTP/1.1" 400 347 "-" "Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0" "jee-eval1.abc.com"

Chrome:
10.145.12.69 jee-eval1.abc.com - [09/Dec/2013:11:03:30 +0100] "GET /clusterjsp/ HTTP/1.1" 400 347 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36" "jee-eval1.abc.com"

So from the current look into the access logs it seems that everytime, when i get an 400 the correct hostname was transmitted.
Comment 10 Kaspar Brand 2013-12-10 05:49:03 UTC
(In reply to Andre W. from comment #9)
> So, i turned up the debug level once more and added some additional
> information into the header and i got still bad requests with an 400 code:

I assume that this is from what you call the "second apache". Note that in comment 8, I was specifically talking about 

> logging %{Host}i for the requests coming in on the "first apache",
> and correlating them with the proxy debug logs
Comment 11 Yann Ylavic 2014-02-21 12:47:15 UTC
Created attachment 31342 [details]
Don't reuse a SSL backend connection whose SNI differs

Following http://mail-archives.apache.org/mod_mbox/httpd-dev/201402.mbox/%3C5144800.Zjx3nWxCiJ@nudel%3E, a backend connection can be reused for a new request but with a different hostname/SNI.

This patch avoids this by saving the connection's SSL hostname in the proxy_conn_rec and checking it against the new request's one, closing if it differs.
Comment 12 Yann Ylavic 2014-02-27 15:47:19 UTC
Commited in r1572630.
Comment 13 Andre W. 2014-02-28 13:52:04 UTC
Hello Yann,

I'm not really able to apply this patch to version 2.2.25 or 2.2.26, are there to many changes in the already existing branch?

Cheers,
André
Comment 14 Yann Ylavic 2014-02-28 14:16:49 UTC
Created attachment 31354 [details]
Backport to 2.2.x

Andre,
could you try this patch?

Regards.
Comment 15 Yann Ylavic 2014-02-28 15:01:22 UTC
Created attachment 31356 [details]
Latest backport to 2.2.x proposal

The previous patch was missing the case where no SNI is in use on the reused connection but the new request requires one.

Fixed in r1572967 and this backport proposal.
Comment 16 Andre W. 2014-03-07 09:13:39 UTC
Hello Yann,

the patch worked well for me!

Best regards,
André
Comment 17 Gilsberty Boscolo 2017-07-17 12:55:12 UTC
This issue or something really similar to it is happening with Apache 2.4.6 on CentOS 7.

The scenario is about the same: one apache reverse proxy accepting https and proxying it to an apache application using https as well.

When the application server has a heavy load as a result from the below "ab" command the client trying to access an url different from the one being heavilly used gets 400 bad request and the server logs an AH02032 error.

command -> ab -n 10000 -c 100 https://site1.com

client -> https://site2.com

error logged -> AH02032: Hostname site1.com provided via SNI and hostname site2.com provided via HTTP are different

Without forcing the server with the heavy load the problem happens less offen and with the heavy load it happens in 100% of the tests.

The workaround using SSLv3 between proxy <-> application is working fine so the problem happens with TLSv1.x!

I'm happy to provide any additional information that may be needed.
Comment 18 Eric Covener 2017-07-17 13:33:08 UTC
This has been fixed in 2.4.x since 2.4.10, if you don't have the patch in your distros 2.4.6+patches, you'll need to work with them.

If you think you've already got the patch, please open a new bug report and reproduce it on a modern source release.
Comment 19 Gilsberty Boscolo 2017-07-25 19:03:23 UTC
Thank you for the information about the first release that received the patch (2.4.10).
It will help new visitors as it helped me.