Bug 47167

Summary: Authenticated sessions being switched by reverse proxy
Product: Apache httpd-2 Reporter: Duncan Fletcher <duncan.fletcher>
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: normal CC: rvandolson
Priority: P2 Keywords: MassUpdate
Version: 2.2.11   
Target Milestone: ---   
Hardware: PC   
OS: Windows XP   

Description Duncan Fletcher 2009-05-07 21:56:12 UTC
We've got a Subversion server (mod_dav_svn under Apache on Windows Server 2003) set up behind another Apache server configured as a reverse proxy (also on Windows Server 2003).  We recently upgraded the reverse-proxy box from Apache httpd 2.0.54 to httpd 2.2.11 and found that when two users access the Subversion repository at the same time, their usernames (or session tags maybe?) are being switched internally by Apache.  i.e. if the two users are simultaneously each trying to access an area that the other doesn't have access to, they get back 401 Authorization Required errors, with the wrong usernames appearing in the log (further explanation below).  The Subversion server has been running Apache 2.2.11 for months without trouble, its just the reverse-proxy we've changed.  We've isolated this error down to one of either mod_proxy or mod_proxy_http (details below), so even though the problem comes up when we use Subversion (including a custom authorization module), we're pretty sure the bug lies in Apache.

At the moment we're fine sticking with Apache 2.0.63 (which happens to be a lot faster than 2.2.6 as a reverse proxy, about 2x speed difference), but if anyone can look into this issue that'd be good.  Thanks!


Example scenario of problem/symptoms:
- userA is doing a large (500KB, 30+ files) commit to http://svnserver/repos/sandboxes/userA/private/
- userB is doing a large (500KB, 30+ files) commit to http://svnserver/repos/sandboxes/userB/private/
- one user kicks off their commit while the other commit is already running, i.e. in the middle of setting up the transaction & transferring the files - or both are kicked off at the same time - doesn't matter really, as long as both sessions are in progress simultaneously
- At that point, one OR both users will get a 401 error (there is no particular pattern we've seen, other than that the user who kicks off second is more likely to get the 401 while the other user's transaction runs to completion, but about 1/3 of the time they both die)
- Looking at the access.log of the subversion server, it shows that (for example) userB was denied access to userA's sandbox (http://svnserver/repos/sandboxes/userA/private/fileXYZ), even though userB was actually operating on their own sandbox at the time.
- Snippet of access.log pasted in at the bottom. We can reproduce this fairly easily if additional Apache logging is necessary, but unfortunately we can't supply any network-traffic dumps.


Details:
- Size of commit/transaction doesn't actually matter, its just that longer transactions are more likely to collide in testing.
- Problem only occurs if accessing the svn repository via the reverse proxy. i.e. Accessing the repository directly has no problems.
- Provided the commit is large enough that there is proper overlap, this happens every time, its not intermittent. The only thing that changes between tests is how many files the two commits get through before dying, but it always dies in less than a second.
- No related messages in error.log (just the usual startup and shutdown notices).
- Tested against different versions of Apache httpd and found some interesting results:
	Apache 2.0.54	- okay
	Apache 2.0.63	- okay
	Apache 2.2.2	- problem
	Apache 2.2.3	- problem
	Apache 2.2.4	- problem
	Apache 2.2.6	- okay!!
	Apache 2.2.8	- problem
	Apache 2.2.9	- problem
	Apache 2.2.10	- problem
	Apache 2.2.11	- problem
	Apache 2.2.11 with mod_proxy.so and mod_proxy_http.so copied in from Apache 2.2.6 - okay!!
- i.e.the problem is in either mod_proxy.so or mod_proxy_http.so and affects every release of Apache 2.2 except for 2.2.6!
- Authentication of users is via mod_auth_sspi, i.e. Windows Domain authentication, at the Subversion server.
- Apache installation is from the binaries downloaded from http://apache.16degrees.com.au/httpd/binaries/win32/ and http://archive.apache.org/dist/httpd/binaries/win32/.
- Tried with and without the openssl build, no difference.
- For testing, both the subversion server and the reverse-proxy server were put on the same LAN subnet so that test-clients could access either/both (to confirm the problem only happened when the reverse proxy was in the loop)
- Just for info completeness, we're running svn 1.5.4 but have also checked svn 1.6.1: no difference in the problem.
- We'd been running the reverse-proxy with Apache 2.0.54 for over a year with no trouble until the upgrade to 2.2.11; using the same httpd.conf as below (except for the version-specific changes like ServerRoot and mod_access vs mod_authz_host).
- Reverse-proxy to the svnserver is the only job the problematic Apache installation is doing (I've listed below all of the modules it has loaded).

Relevant conf lines from the reverse-proxy server:
--vv------<snip>---------vv--
LoadModule authz_host_module modules/mod_authz_host.so LoadModule dir_module modules/mod_dir.so LoadModule env_module modules/mod_env.so LoadModule headers_module modules/mod_headers.so LoadModule include_module modules/mod_include.so LoadModule log_config_module modules/mod_log_config.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_html_module modules/mod_proxy_html.so LoadModule setenvif_module modules/mod_setenvif.so ProxyRequests off ProxyVia On <Location /repos >
	ProxyPass http://svnserver.ourdomain/repos
      ProxyPassReverse /repos/
      ProxyPassReverse http://svnserver/repos
      ProxyPassReverse http://svnserver.ourdomain/repos
      <Limit OPTIONS PROPFIND GET REPORT MKACTIVITY PROPPATCH PUT CHECKOUT MKCOL MOVE COPY DELETE LOCK UNLOCK MERGE>
        Order Deny,Allow
        Allow from all
        Satisfy Any
      </Limit>
</Location>
--^^------<snip>---------^^--


Relevant conf lines from subversion server ("svnserver") (note these are not the only modules this server has loaded):
--vv------<snip>---------vv--
LoadModule sspi_auth_module modules/mod_auth_sspi.so 
LoadModule dav_svn_module modules/mod_dav_svn.so 
LoadModule permissions_svn_module modules/mod_permissions_svn.so 
<Location /repos>
    DAV svn
    SVNPath //fileserver/svnrepos/repos
    AuthType SSPI
    AuthName "Subversion repository by OurDomain"
    Require valid-user
    SSPIAuth On			
    SSPIAuthoritative On	
    SSPIDomain OurDomain
    SSPIOfferBasic On		
    PermissionsSVNAccessFile //fileserver/svnrepos/conf/svs_permissions.conf
</Location>
--^^------<snip>---------^^--


mod_permissions_svn.so is an in-house simple-ish authorization module based on mod_authz_svn that hooks into httpd core as follows:
--vv------<snip>---------vv--
static void registerHooks(apr_pool_t *p)
{
    ap_hook_access_checker(checkAccess, NULL, NULL, APR_HOOK_LAST);
    ap_hook_auth_checker(checkAuthorisation, NULL, NULL, APR_HOOK_FIRST); 
}

module AP_MODULE_DECLARE_DATA permissions_svn_module = {
    STANDARD20_MODULE_STUFF,
    createConfigRecord,              /* config creater per dir */
    NULL,                            /* dir merger --- default is to override */
    NULL,                            /* server config */
    NULL,                            /* merge server config */
    MODULE_ARGUMENTS,                /* command apr_table_t */
    registerHooks                    /* register hooks */
};
--^^------<snip>---------^^--
where checkAccess(...) and checkAuthorisation(...) implement our custom rules using fields from the request_rec parameter such as user, uri, etc.  I only mention this to help convince anyone reading this that this mod_permissions_svn is not the problem.


And finally a snippet of the access.log of the subversion server...
--vv------<snip>---------vv--
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PROPFIND /repos/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.cpp HTTP/1.1" 207 776
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PROPFIND /repos/!svn/vcc/default HTTP/1.1" 207 455
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PROPFIND /repos/!svn/bc/52575/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.cpp HTTP/1.1" 207 535
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "CHECKOUT /repos/!svn/ver/52575/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.cpp HTTP/1.1" 201 307
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "PUT /repos/!svn/wrk/11125f68-8c91-644c-85a7-b9a056d3b561/sandboxes/coopern/private/testing_commit/tool/test/test_csvfile.cpp HTTP/1.1" 204 -
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PUT /repos/!svn/wrk/11125f68-8c91-644c-85a7-b9a056d3b561/sandboxes/coopern/private/testing_commit/tool/tool_marshal_containers.h HTTP/1.1" 401 401
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "PROPFIND /repos/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.h HTTP/1.1" 401 401
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "DELETE /repos/!svn/act/10089f74-f41f-484e-b1a0-783dafcf2155 HTTP/1.1" 204 -
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "DELETE /repos/!svn/act/11125f68-8c91-644c-85a7-b9a056d3b561 HTTP/1.1" 204 -
--^^------<snip>---------^^--
Comment 1 William A. Rowe Jr. 2009-05-07 22:11:22 UTC
Please confirm;

2.0.63 reverse proxy to 2.2.11 dav_svn server is flawless.

2.0.54 reverse proxy to 2.2.11 dav_svn server results in crossed logins.

If that is correct, this bug would not be addressed; the 2.0.63 version
is the current 2.0 release and the ASF does not address bugs in previous
versions, although some vendors will depending on their service agreements.
Comment 2 Duncan Fletcher 2009-05-07 23:33:04 UTC
No, the 2.0.54 reverse proxy to 2.2.11 dav_svn server is flawless too.  It's the 2.2 series that has the problem.  i.e:

2.0.X reverse proxy to 2.2.11 dav_svn server is flawless (tested 2.0.54 and 2.0.63)

2.2.X reverse proxy to 2.2.11 dav_svn server results in crossed logins
  (except for 2.2.6, that one is okay for some reason; tested all released 2.2.X versions)

To be clear, 2.2.11 has this issue on our setup; it is not an "old problem with an old release" issue, its a current-release issue.
Comment 3 Nick Kew 2009-05-08 01:29:49 UTC
That's a very precise chronology!

Just checking the change log, 2.2.6 introduced PR 43472 - breaking keep-alive in proxied HTTP backend connections.  This was fixed in 2.2.8.  This looks a likely candidate for your observation that 2.2.6 doesn't exhibit this bug.

Can you try 2.2.11 with that fix reversed?  If this fixes your bug, we can start to think about a fix for both!

Please try 2.2.11 with the following patch (there may be an offset, since this is actually against svn 2.2 branch):

--- modules/proxy/proxy_util.c  (revision 772881)
+++ modules/proxy/proxy_util.c  (working copy)
@@ -2189,7 +2189,7 @@
         else
             return 0;
     }
-    else if (APR_STATUS_IS_EAGAIN(status) || APR_STATUS_IS_TIMEUP(status)) {
+    else if (APR_STATUS_IS_EAGAIN(status)) {
         return 1;
     }
     return 0;
Comment 4 Ruediger Pluem 2009-05-08 02:13:18 UTC
I guess this is caused by the fact that your custom authentication module makes wrong assumptions about the HTTP protocol. HTTP is *stateless*. Because of this wrong assumption it is well known that NTLM doesn't work any longer over 2.2.x reverse proxies because it assumes that HTTP is a *statefull* protocol. You have to keep in mind that the reverse proxies eventually maps the same keepalive connection from the client to the reverse proxy to different backend connections for each request.
Comment 5 Duncan Fletcher 2009-05-10 22:01:30 UTC
Okay, we just ran another test, this time using "AuthType Basic" instead of "AuthType SSPI" and mod_auth_sspi (we are using v1.0.4 from http://sourceforge.net/projects/mod-auth-sspi).  The problem went away under Apache 2.2.11.
So it looks like Ruediger is correct and that the problem is in an assumption that mod_auth_sspi is (incorrectly) making about keep-alives being synonymous with sessions.

We've been using pre-compiled binaries so although we can break out the compiler to try Nick's suggestion, it'll take time.  i.e. I'd like confirmation that it will (likely) add valuable information before we chase that rabbit.

In the meantime, we'll look into using mod_authnz_ldap for Windows AD authentication as an alternative to mod_auth_sspi and also track down a copy of the latter's source code to see if its feasible to fix that.
Comment 6 William A. Rowe Jr. 2018-11-07 21:08:46 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.