Bug 39673 - mod_proxy opens connections that disturb NTLM
Summary: mod_proxy opens connections that disturb NTLM
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.2.11
Hardware: All All
: P2 normal with 17 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate, PatchAvailable
: 39680 (view as bug list)
Depends on:
Blocks:
 
Reported: 2006-05-29 12:47 UTC by Olivier BOEL
Modified: 2018-11-07 21:08 UTC (History)
8 users (show)



Attachments
proxy NTLM requests received over the same client connection via the same backend connection (4.02 KB, patch)
2013-02-26 18:00 UTC, Micha Lenk
Details | Diff
mod_proxy_http one for one connection (8.79 KB, patch)
2013-09-26 18:39 UTC, Yann Ylavic
Details | Diff
mod_proxy_http one for one connection (9.18 KB, patch)
2013-09-27 14:00 UTC, Yann Ylavic
Details | Diff
mod_proxy one for one connection (9.51 KB, patch)
2013-09-27 14:19 UTC, Yann Ylavic
Details | Diff
Handle aside connections in mod_proxy (2.4.x) (12.23 KB, patch)
2014-06-26 14:18 UTC, Yann Ylavic
Details | Diff
Handle aside connections in mod_proxy (2.4.x) (12.35 KB, patch)
2014-06-26 14:53 UTC, Yann Ylavic
Details | Diff
Handle aside connections in mod_proxy (2.4.x) (12.41 KB, patch)
2014-06-26 14:56 UTC, Yann Ylavic
Details | Diff
Handle aside connections in mod_proxy (2.4.9) (12.33 KB, patch)
2014-06-26 15:08 UTC, Yann Ylavic
Details | Diff
Handle aside connections in mod_proxy (trunk) (11.89 KB, patch)
2014-09-22 07:49 UTC, jkaluza
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Olivier BOEL 2006-05-29 12:47:21 UTC
The following configuration works fine with Apache 2.0 :
- client : IE + Windows XP
- Reverse Proxy : Apache 2.0 running on solaris 9
- IIS (Windows) server with NTLM authentication enabled
When a protected page (via ACL on the IIS server) is accessed by the 
client, thanks to Windows integrated authentication, the page is 
displayed with any user intervention (user identification prompt).

Since Apache 2.2, user receives an identification prompt and, although 
the username/password he enters are correct, he is not authorized.

Looking at the network traffic, it seems that the NTLM authentication 
process is made of 3 requests.
Between the client and the RP, they use a single connection (same 
port).
Between the RP and the IIS server, they use a single connection if the 
RP is running on Apache 2.0; however, with Apache 2.2, the 3 requests 
use 3 different connections (3 ports), which make NTLM fail.
This problem is reproductible at will.
I tried the "ProxyPass keepalive=On" directive but it didn't help.


Is there a workaround?
Comment 1 Ruediger Pluem 2006-05-29 19:41:57 UTC
Have you tried upgrading to 2.2.2? Keepalive in the proxy does not work
correctly with 2.2.0.
Comment 2 Olivier BOEL 2006-05-30 08:28:38 UTC
Indeed, Apache 2.2.2 brings some improvement :
1. ProxyPass directive works fine (i.e. uses the same port for all 
connections), even without the "keepalive=On" option
2. However, RewriteRule ^(.*)$ http://windows_server$1 [P,NE] still uses 
different ports to communicate with the back-end Windows server although the 
communication between the client and the RP uses the same port. In other words, 
the Rewrite module, when used in conjunction with the Proxy module, "splits" 
the communication on different ports, resulting in NTLM failure.

Thanks for your help!
Comment 3 Olivier BOEL 2006-05-30 08:31:36 UTC
Indeed, Apache 2.2.2 brings some improvement :
1. ProxyPass directive works fine (i.e. uses the same port for all 
connections), even without the "keepalive=On" option
2. However, RewriteRule ^(.*)$ http://windows_server$1 [P,NE] still uses 
different ports to communicate with the back-end Windows server although the 
communication between the client and the RP uses the same port. In other words, 
the Rewrite module, when used in conjunction with the Proxy module, "splits" 
the communication on different ports, resulting in NTLM failure.

Thanks for your help!
Comment 4 Ruediger Pluem 2006-05-30 19:54:32 UTC
*** Bug 39680 has been marked as a duplicate of this bug. ***
Comment 5 Ruediger Pluem 2006-05-30 20:13:00 UTC
Meanwhile I found some time to have a look how NTLM auth actually works. As a
result I found out that the current 2.2.x proxy implementation does NOT support
NTLM, because there is no guarantee that the same backend connection is used for
the next request on a keepalive frontend connection. Each request from a
frontend connection leases a backend connection from a connection pool for the
request and returns it back to the pool immediately after the request has been
processed. If the next request on this keepalive frontend connection is
processed it may lease a different backend connection from the pool. As far as I
understand NTLM this approach is not compatible with NTLM. I will send a follow
up on the dev list for further discussion.
Comment 6 William A. Rowe Jr. 2006-05-30 20:43:48 UTC
NTLM Auth by Microsoft violates HTTP/1.1 RFC's all of which defined the protocol
as stateless.  It might behove us either to

 1) provide a config directive to proxy-stateful and not recycle connections
    to specific / pattern match of hosts, and/or

 2) detect NTLM crap and mark the connection as non-recycleable.

Comment 7 Peter Pearce 2007-12-17 00:49:48 UTC
"Indeed, Apache 2.2.2 brings some improvement :"

As does 2.2.4, however, it seems that 2.2.6 breaks it again. This was
immediately obvious after upgrade with no change in configs. (keepalives off)
Comment 8 Ryan Malayter 2009-01-10 12:51:46 UTC
I noticed there has been no activity for over a year, so I figured I would provide an update.

I am experiencing the same issue in a forward-proxy scenario using mod_proxy (so the issue is not limited to reverse-proxy setups). NTLM authentication over a non-HTTP connection using mod_proxy fails. Interestingly, NTLM authentication over a tunneled SSL connection using mod_proxy_connect seems to work just fine.

This is using version 2.2.11. None of the environment variables for mod_proxy_http or ProxyPass options seem to help.

I know that NTLM authentication may be requiring some non-RFC behavior (reusing the same connection). However, there are many other places in httpd where options are provided to work around bugs in non-RFC-compliant software. Is it possible to do that here?

What other information might be helpful to provide to get a workaround documented or a patch in place that lets us use NTLM? I am not much of a C/C++ jockey anymore, but I can certainly help with packet traces, testing different options, whatever.

I will try to have a look at the Squid bug tracker to see if there were any NTLM issues for their code and if so, how they resolved them. Because I know NTLM works in Squid proxies (somehow).
Comment 9 Lukas Koranda 2009-02-12 07:02:29 UTC
I would like to support comment sent by Ryan. I have absolutely the same experience. It looks that there is needed to do an workaround for correct behavior for NTLM which is metioned above in this posting. I've tested latest patched httpd 2.2.3 RHEL version with mod_proxy and it is impossible to get it working. 
Comment 10 Ryan Malayter 2009-02-12 07:18:24 UTC
I switched hardware and OS to ALL for this bug, since this was originally reported on SPARC, but also in my testing is an issue on Windows/i386 as well as Linux/amd64.

I also, as promised, checked out the status of NTLM proxy support in Squid, which I had seen work before, and it is not good. Apparently, the issue has been "fixed" in some 2.7 releases, broken in others, and still appears to be broken in the latest stable releases of 2.7 and 3.0 Squid.

I think, perhaps, my testing with Squid that seemed to work was only because I was lucky and happened to be re-using the same back-end connections.

Is there a way to make back-end connections "sticky" for the same front end client/destination combination? Wouldn't that fix the issue? A simple hash table would suffice to track this, wouldn't it?

Comment 11 oscar 2010-09-23 05:34:36 UTC
Hi, 

I am experiencing the same issue. Is there any new or workarround for this issue ?

Thanks a lot!!!!!
Comment 12 Ray Van Dolson 2010-09-23 11:18:18 UTC
(In reply to comment #11)
> Hi, 
> 
> I am experiencing the same issue. Is there any new or workarround for this
> issue ?
> 
> Thanks a lot!!!!!

Don't use NTLM, switch to Apache 2.0.x or MS ISA :)
Comment 13 Micha Lenk 2013-02-26 18:00:03 UTC
Created attachment 29994 [details]
proxy NTLM requests received over the same client connection via the same backend connection

Looking at how mod_proxy_ftp.c solves a similar problem, I tried to solve that issue with the attached patch (also attached to bugzilla). If a NTLM request is detected, the cleanup of the currently leased backend connection is skipped. Instead the backend connection is registered with the client connection pool, so that it is closed and cleaned up as soon as the client disconnects. Additionally the backend connection is registered as a config record in the client connection, so that it can get re-used for subsequent requests on the same client connection.
Comment 14 Micha Lenk 2013-03-01 12:56:14 UTC
My patch (attachment 29994 [details]) is broken and causes segfaults, so please don't use it. I am working on an improved/fixed version.
Comment 15 Vivek Thomas 2013-09-24 15:20:28 UTC
Is there any workaround or fix for this issue in Apache 2.4.x?
Comment 16 Yann Ylavic 2013-09-26 18:39:26 UTC
Created attachment 30887 [details]
mod_proxy_http one for one connection

This patch (2.4.x) associates the backend's connection with the client's one, when the env variable "proxy-1for1-connection" is defined.

One can use a SetEnv or a RewriteRule like the following to get the expected result :

 RewriteCond %{HTTP:Authorization} ^NTLM
 RewriteRule ^ - [E=proxy-1for1-connection]

Note: be sure to only set the env variable in a relevant Location or one can use an Authorization header to force mod_proxy in this mode, which is less efficient...

Regarding the patch, the rationale is that mod_proxy can't use a backend connection from the reslist for the whole lifetime of a client connection, or the other clients would miss that backend connection during the client's Keep-Alive times.

Hence the new APR_OPTIONAL_FN(ap_proxy_conn_create), used by mod_proxy_http to get an "aside" (off the reslist) connection and bind it to the client's connection (using c->conn_config and built on c->pool, once). The connection will be reused for the next requests, until the client closes (both are destroyed then).

Since I don't know what is really feasible in terms of API/ABI changes in 2.4.x, ap_proxy_conn_create is private (APR_DECLAREd_OPTIONAL in proxy_util.h, which AFAICT is not public). Maybe the ap_proxy_conn_destroy/clear/close functions and the the "1for1" handling in mod_proxy_http could also be usefull for other mod_proxy_* modules, I guess this is off topic here...

To preserve the API still, this patch circumvents the proxy_conn_rec's flag 'inreslist', which used to be a guard not to put back the same connection twice in the reslist, and now indicates whether the connection comes from the reslist or not. Ideally the patch should create another flag (with an API change), but anyhow this guard seems useless should the connection be released twice (the damage is done), there is still the possibility that another thread gets this connection before the second release.

Finally, I would add that it is not only a NTLM compatibility problem, some devices (BigIp LTM's with some configuration IIRC), will simply refuse multiple clients (sessions) in the same TCP connection. Although this can surely be disabled on the device (and is indeed a questionable feature), some like it, and even some want mod_proxy in front of it...

If the 1for1 feature worth being in mod_proxy (that way), I can provide a trunk patch too.
Comment 17 Yann Ylavic 2013-09-27 14:00:03 UTC
Created attachment 30888 [details]
mod_proxy_http one for one connection

The previous patch segfaults when the backend closes the connection by itself, or when is_address_reusable is unset, or when disablereuse is on (however the two latter cases make few sense in 1for1 mode).

This new patch fix these issues, and it won't enable 1for1 mode if the address is not reusable or disablereuse is on.

To come (next comment), another patch that let mod_proxy do the job instead of mod_proxy_http (I find it more simple, so maybe with more chances to be accepted).
Comment 18 Yann Ylavic 2013-09-27 14:19:55 UTC
Created attachment 30889 [details]
mod_proxy one for one connection

As said in the previous comment, here is a new patch which is maybe more simple.

The rationale is the same (backend's connection attached to the client's one, out of the reslist), but the job is done by the new ap_proxy_acquire_connection_ex function (declared optional in proxy_util.h too), provided that the caller gives it a conn_rec to attach to (when the conn_rec is NULL, it is the legacy ap_proxy_acquire_connection function, which unfortunately takes a server_rec and not a request_rec as argument).

Therefore the changes in mod_proxy_http and the interaction with mod_proxy are simpler, and other proxy modules can use ap_proxy_acquire_connection_ex to do the same.

Hope this patch (or the other) has a chance to be accepted, since I don't really understand why NTLM is less HTTP compliant than wstunnel or any other streaming over HTTP (do you plan to close/reuse the backend connection during client's inactivity for these?).
Comment 19 Vittorio Guglielmo 2014-06-21 08:07:42 UTC
Hi all,

I tried Yann's mod_proxy_one patch against apache-2.4.7 (the only version on which I could apply w/o errors) and it works fine for HTTP.

Instead, in HTTPS, I have a strange behavior:

opening a page containing various iframes each pointing to some of two Sharepoint backends, I find it address calls toward the wrong connection (backend).

I try to figure it better:

My two backends:

1) backend mssA.foo.com    IP 172.22.47.54
2) backend mssB.foo.com    IP 172.22.47.52

Extract of my https virtual host:

ProxyPass                  /Runtime  http://mssA.foo.com:82/Runtime/
ProxyPassReverse           /Runtime/ http://mssA.foo.com:82/Runtime/

ProxyPass                  /fascicoli http://mssB.foo.com/fascicoli/
ProxyPassReverse           /fascicoli/ http://mssB.foo.com/fascicoli/


I found this entries in my error.log (debug activated):

[pid 8818:tid 140388427851520] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssA.foo.com)

[pid 8819:tid 140388642264832] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8905:tid 140388642264832] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8817:tid 140388490790656] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8819:tid 140388511770368] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8905:tid 140388610795264] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8818:tid 140388511770368] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssA.foo.com)

[pid 8905:tid 140388427851520] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8817:tid 140388621285120] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8819:tid 140388343932672] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

[pid 8905:tid 140388417361664] proxy_util.c(2623): AH00962: HTTP: connection complete to 172.22.47.54:80 (mssB.foo.com)

As you can see, I have :  

to 172.22.47.54:80 (mssA.foo.com)
but it should be 
to 172.22.47.52:82 (mssA.foo.com)

Both IP address and port are wrong, they are taken from the other backend (mssB.foo.com)

So, definetely, sometimes it use a wrong connection, causing a lot on 404 errors.

Can you help me ?

Thanks 

Vittorio
Comment 20 Yann Ylavic 2014-06-26 14:18:39 UTC
Created attachment 31751 [details]
Handle aside connections in mod_proxy (2.4.x)

(In reply to Vittorio Guglielmo from comment #19)
> opening a page containing various iframes each pointing to some of two
> Sharepoint backends, I find it address calls toward the wrong connection
> (backend).

Thanks for testing.

The previous patch did not work correctly whenever multiple requests on the same client connection address different backend.
The only backend connection associated with the client one was reused, whether or not the worker has changed.

This new patch addresses this by handling a hashtable of backend connections (one per worker) for one client connection.

It is quite different from the previous one, by allowing any (proxy) module to use the new :

+/**
+ * Acquire a connection from worker connection pool, or associated with
+ * the given conn_rec or request_rec.
+ * @param proxy_function calling proxy scheme (http, ajp, ...)
+ * @param conn    acquired connection
+ * @param worker  worker used for obtaining connection
+ * @param s       current server record
+ * @param baton   associated baton (or conn_rec if @acquire is NULL)
+ * @param acquire function (if not NULL) used to acquire aside connections,
+ *				  given @baton, @conn, @worker, and @s
+ * @param release function (if not NULL) used to release acquired connection,
+ *				  given @baton, @conn, @worker, and @s, as callback
+ * @return        OK or HTTP_XXX error
+ * @note If the connection limit has been reached, the function will
+ * block until a connection becomes available or the timeout has
+ * elapsed.
+ * @note If the given @baton is not NULL and @acquire is NULL, the baton is
+ * considered a conn_rec to which a connection will be attached (by worker).
+ */
+PROXY_DECLARE(int) ap_proxy_acquire_connection_ex(const char *proxy_function,
+                                                  proxy_conn_rec **conn,
+                                                  proxy_worker *worker,
+                                                  server_rec *s, void *baton,
+                                                  ap_proxy_acquire_fn acquire,
+                                                  ap_proxy_release_fn release);
+

For now, only mod_proxy_http and mod_proxy_ajp have been changed to do so, since the others seem not to reuse connections, but maybe it is worth using a quick way to acquire/release a connection (ie. no reslist locks) for those too, especially if it is always closed at the end (no recycling).

Can you give this new patch a try?
Comment 21 Yann Ylavic 2014-06-26 14:30:18 UTC
(In reply to Yann Ylavic from comment #20)
> +/**
> + * Acquire a connection from worker connection pool, or associated with
> + * the given conn_rec or request_rec.

This description is not correct (bad copy/paste), should be something like :

/**
 * Acquire a connection using the given @acquire function and register
 * a @release callback for it on cleanup, or use the worker's connections
 * pool if the associated @baton is NULL.
 * ...
 */
Comment 22 Yann Ylavic 2014-06-26 14:35:23 UTC
Also, please not that the SetEnv is now "proxy-aside-c" for this to be enabled, since the previous "proxy-1for1-connection" is quite misleading now.
Comment 23 Yann Ylavic 2014-06-26 14:53:30 UTC
Created attachment 31752 [details]
Handle aside connections in mod_proxy (2.4.x)

Fix from comment #21 above.
Comment 24 Yann Ylavic 2014-06-26 14:56:59 UTC
Created attachment 31753 [details]
Handle aside connections in mod_proxy (2.4.x)

Oups, I forgot to set the new conn in the hashtable (can't work).
Fixed now.
Comment 25 Yann Ylavic 2014-06-26 15:08:34 UTC
Created attachment 31754 [details]
Handle aside connections in mod_proxy (2.4.9)

Same (latest) patch for 2.4.9 (resolves minor conflicts).
Comment 26 Vittorio Guglielmo 2014-07-02 13:27:15 UTC
Many thanks to Yann, his last patch works perfectly.

It is now a must-have for people working in a Microsoft environment dealing
with NTLM and/or Windos Integrated Authentication.

Vittorio
Comment 27 ka.hing.chan 2014-07-16 01:24:32 UTC
Am I missing something? I have 
ProxyPass http://url/
ProxyPassReverse http://url/
SetEnv "proxy-aside-c" inside the location where it is doing the reverse proxy. But I am still getting Error code: ERR_INVALID_AUTH_CREDENTIALS. Is there other configurations I need to do?
Comment 28 Christoffer Bruun 2014-08-01 23:28:53 UTC
I had issues with mod_proxy and NTLM like the ones described in this bug. This happened after a server upgrade (debian) where I had installed apache-mpm-worker.

Changing to the apache-mpm-prefork package made the problem go away.

- this may be a workaround for people in the same situation
Comment 29 jkaluza 2014-09-22 07:49:03 UTC
Created attachment 32040 [details]
Handle aside connections in mod_proxy (trunk)
Comment 30 jkaluza 2014-09-22 07:49:57 UTC
I've rebased the patch against httpd-trunk.

Yann, do you plan to commit/propose this patch or have you already tried and were there some problems with it?
Comment 31 Yann Ylavic 2014-09-25 12:07:55 UTC
Jan,

no I didn't try to propose the patch, based on the comment(s) in this PR stating that NTLM is stateful and hence not HTTP (RFC) compliant.

On the other hand this patch is not NTLM specific (see comment #16 about issues I encountered with some load-balancers), I use the different versions I proposed here since a while now, with no problem.

It is less performant than the "normal" connections' reuse implementation, but optional still.

Now we are two maybe it has more chance to be accepted.
Should I propose it on dev@, or do you plan to?
Comment 32 Yann Ylavic 2014-09-25 12:13:39 UTC
(In reply to ka.hing.chan from comment #27)
> Am I missing something? I have 
> ProxyPass http://url/
> ProxyPassReverse http://url/
> SetEnv "proxy-aside-c" inside the location where it is doing the reverse
> proxy. But I am still getting Error code: ERR_INVALID_AUTH_CREDENTIALS. Is
> there other configurations I need to do?

No there shouldn't. Can you provide traces (debug log, maybe pcap on both sides) so that we can analyse what's going on?
Comment 33 jkaluza 2014-09-30 06:23:04 UTC
Yann,

as you are the author and see more possible use-cases, it would be probably better for you to propose it. I will join the discussion :).
Comment 34 Yann Ylavic 2014-10-01 09:52:44 UTC
Discussion initiated here: http://www.mail-archive.com/dev@httpd.apache.org/msg60721.html
Comment 35 William A. Rowe Jr. 2018-11-07 21:08:06 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.