Bug 34716

Summary: ProxyPass results in Bad Request
Product: Apache httpd-2 Reporter: edwin.meester
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED INVALID    
Severity: normal CC: dan, kikov
Priority: P2    
Version: 2.0.54   
Target Milestone: ---   
Hardware: PC   
OS: Windows Server 2003   
Attachments: httpd-conf of non working proxypass setup

Description edwin.meester 2005-05-02 23:52:32 UTC
After Upgrading from 1.3.31 to 2.0.54 I can't get ProxyPass to work.
It's seems the proxy sends a Request to internal server with some bad headers.

A ProxyPass to the same server, results in a 400 error.

Bad Request

Your browser sent a request that this server could not understand.
Request header field is missing ':' separator.

GET / HTTP/1.1

Apache/2.0.54 (Win32) PHP/5.0.4 Server
Comment 1 Nick Kew 2005-05-17 15:58:53 UTC
GET / HTTP/1.1  (on its own) *is* a bad request.

There's insufficient information in this report to find a potential bug.
Please ask in a user support forum (with details of the problem),
or provide details of apache doing something wrong.
Comment 2 edwin.meester 2005-05-17 21:45:37 UTC
(In reply to comment #1)
> GET / HTTP/1.1  (on its own) *is* a bad request.
> 
> There's insufficient information in this report to find a potential bug.
> Please ask in a user support forum (with details of the problem),
> or provide details of apache doing something wrong.

the complete header sent by the ProxyPass directive:

GET / HTTP/1.1
Host: www.mydomain.com:80
Accept: */*
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR
1.1.4322)
Max-Forwards: 10
X-Forwarded-For: 80.69.81.37
X-Forwarded-Host: www.proxydomain.com
X-Forwarded-Server: www.proxydomain.com

But for some reason the Apache server doesn't like this Header.
Comment 3 Nick Kew 2005-05-18 10:34:55 UTC
> Your browser sent a request that this server could not understand.
> Request header field is missing ':' separator.
> 
> GET / HTTP/1.1

What that's actually saying is that it thinks the above is being sent as a
request header - where it is invalid (just as the message says).  Is that
happening on the proxy or the backend?  What is generating the original request
to the proxy?
Comment 4 edwin.meester 2005-05-18 18:55:14 UTC
It happens on the server behind the apache proxy.

example:

Internet Explorer -> Apache 2.0.54 As Proxy -> another server.

Tried 2 servers as backend: (Wanadoo.nl and the same apache server on another
virtual host.
Comment 5 edwin.meester 2005-12-06 22:56:04 UTC
After some more investication it seems Apache puts the header twice:

GET / HTTP/1.1\r\n
Host: www.mydomain.com:80\r\n
Accept: */*\r\n
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR
1.1.4322)\r\n
Max-Forwards: 10\r\n
X-Forwarded-For: 80.69.81.37\r\n
X-Forwarded-Host: www.proxydomain.com\r\n
X-Forwarded-Server: www.proxydomain.com\r\n
GET / HTTP/1.1\r\n
Host: www.mydomain.com:80\r\n
Accept: */*\r\n
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR
1.1.4322)\r\n
Max-Forwards: 10\r\n
X-Forwarded-For: 80.69.81.37\r\n
X-Forwarded-Host: www.proxydomain.com\r\n
X-Forwarded-Server: www.proxydomain.com\r\n\r\n

Which ofcourse results in:

A ProxyPass to the same server, results in a 400 error.

Bad Request

Your browser sent a request that this server could not understand.
Request header field is missing ':' separator.

GET / HTTP/1.1

Apache/2.0.54 (Win32) PHP/5.0.4 Server
Comment 6 Ruediger Pluem 2005-12-06 23:55:59 UTC
Several changes have been made to the proxy code from 2.0.54 to 2.0.55. Please
check if the error still exists with 2.0.55.
Comment 7 edwin.meester 2005-12-07 00:01:20 UTC
The same bug is in Apache 2.0.55 and Apache 2.2.0 
Comment 8 Ruediger Pluem 2005-12-07 00:04:44 UTC
*** Bug 37092 has been marked as a duplicate of this bug. ***
Comment 9 Ruediger Pluem 2005-12-07 00:07:36 UTC
Have you tried removing php from the httpd server running the proxy?
I am not quite sure if I remember myself correctly, but I think there had been
some nasty sideeffects between mod_proxy and php.
Comment 10 edwin.meester 2005-12-07 00:14:23 UTC
> Have you tried removing php from the httpd server running the proxy?
> I am not quite sure if I remember myself correctly, but I think there had been
> some nasty sideeffects between mod_proxy and php.

Just tried disabling PHP under Apache 2.2.0, without luck.
Still the header is send to the backend server Twice.

Comment 11 Dan Letecky 2005-12-07 00:24:12 UTC
(In reply to comment #9)
> Have you tried removing php from the httpd server running the proxy?
> I am not quite sure if I remember myself correctly, but I think there had been
> some nasty sideeffects between mod_proxy and php.

I am also not using PHP and the problem still appears. As I noted under bug
37092 the problem didn't appear on Windows Server 2003 (just on Windows XP). I
had a working installation on Windows Server 2003 which I copied exactly to WXP
and it stopped working (it started to sending the header twice).
Comment 12 Ruediger Pluem 2005-12-07 00:32:32 UTC
Strange. Just checked on my Linux box and it works perfectly. Next ideas I have is

1. A configuration issue.
2. A platform issue.

To address 1. please attach your configuration to this report.
For 2.: Can any developer reproduce this on windows? I cannot help as I am Unix guy.
Comment 13 Ruediger Pluem 2005-12-07 00:36:08 UTC
(In reply to comment #11)

[..cut..]

> I am also not using PHP and the problem still appears. As I noted under bug
> 37092 the problem didn't appear on Windows Server 2003 (just on Windows XP). I
> had a working installation on Windows Server 2003 which I copied exactly to WXP
> and it stopped working (it started to sending the header twice).

Ahh, many thanks for that hint. I just lost sight of it. So it seems to be a
platform issue with windows XP. Time to call the windows guys in and for me to
bail out.

Comment 14 edwin.meester 2005-12-07 00:46:32 UTC
Created attachment 17162 [details]
httpd-conf of non working proxypass setup

This a copy of my httpd-conf 
Apache 2.0.55 on Windows 2003 Server
Comment 15 edwin.meester 2005-12-07 00:48:52 UTC
>> I am also not using PHP and the problem still appears. As I noted under bug
>> 37092 the problem didn't appear on Windows Server 2003 (just on Windows XP). 
I
>> had a working installation on Windows Server 2003 which I copied exactly to 
WXP
>> and it stopped working (it started to sending the header twice).

>Ahh, many thanks for that hint. I just lost sight of it. So it seems to be a
>platform issue with windows XP. Time to call the windows guys in and for me to
>bail out.

I've the problem on Windows 2003 Server. So it could still be an config problem.
The working windows 2003 version of Apache was which version ?? 
Comment 16 William A. Rowe Jr. 2005-12-07 01:00:24 UTC
1)  Are you runnning on WinXP SP1, or vanilla/original XP?

2)  Are you testing/encountering this issue on the loopback (localhost, 127.0.0.1)
    device, on external connections, or both?  [Loopback by MS is a regularly
    buggy beast.]

Comment 17 edwin.meester 2005-12-07 01:05:07 UTC
1) Windows 2003 Server WEB SP1

2) Internal and External
Comment 18 Nick Kew 2005-12-07 01:26:08 UTC
Sounds like it might be interacting with something on your network.  Since 
windows is awash with security and associated products that tamper with the 
networking, perhaps those of you who suffer this behaviour can list what 
you've got installed, and maybe some common factor will appear. 
Comment 19 edwin.meester 2005-12-07 19:10:14 UTC
I found an old Apache 2.0.52 package.
Which seems to work.
So maybe samewhere betweeon 2.0.52 and 2.0.54 there is something changed which
causes this behaviour.
Comment 20 Dan Letecky 2005-12-07 19:28:32 UTC
(In reply to comment #16)
> 1)  Are you runnning on WinXP SP1, or vanilla/original XP?
> 
> 2)  Are you testing/encountering this issue on the loopback (localhost, 127.0.0.1)
>     device, on external connections, or both?  [Loopback by MS is a regularly
>     buggy beast.]
> 
> 

My configuration is 
1) working on:
Windows Server 2003
no SP1
firewall turned off
running on VMWare

2) not working on:
Windows XP
SP2
firewall turned on
running on a PC
Comment 21 edwin.meester 2005-12-07 19:48:05 UTC
(In reply to comment #19)
> I found an old Apache 2.0.52 package.
> Which seems to work.
> So maybe samewhere betweeon 2.0.52 and 2.0.54 there is something changed which
> causes this behaviour.

2.0.53 Also seems to work

Comment 22 edwin.meester 2005-12-07 20:17:07 UTC
(In reply to comment #20)
> 
> My configuration is 
> 1) working on:
> Windows Server 2003
> no SP1
> firewall turned off
> running on VMWare
> 
> 2) not working on:
> Windows XP
> SP2
> firewall turned on
> running on a PC
> 
So concluding:

The problem appears on

1. Apache versions higher than 2.0.53
2. Windows 2003 SP1 / Windows XP SP2  (same security/network fixes)

Comment 23 William A. Rowe Jr. 2005-12-07 22:21:34 UTC
You are all clear in the understanding that the various 'firewall' software
products, anti-virus 'internet protection', and other add ins all insert 
themselves into the network socket stack?

Some of the vendors of this software haven't implemented (correctly) the
entire WinSock2 advanced API.  You can test this by completely disabling
the product and rebooting the machine, and if that fails, uninstalling the
product, rebooting and retesting.

Also there are several options which cripple Apache to use older API's that
are less likely to demonstrate these bugs in 3rd party socket stack drivers,
you can try adding these directives globally to your server, one at a time,
and report back which directive did resolve this double-headers bug;

  Win32DisableAcceptEx
  EnableSendfile off
  EnableMMap off

I'm not excluding the possibility of a bug, but trying to narrow down under
what conditions the bug might exhibit itself, and at least getting your
servers up and operating.

Those others who see this flaw, what extra software (MS Firewall, AV products,
etc) are you running on the sockets?


Comment 24 edwin.meester 2005-12-07 23:08:09 UTC
(In reply to comment #23)
> You are all clear in the understanding that the various 'firewall' software
> products, anti-virus 'internet protection', and other add ins all insert 
> themselves into the network socket stack?
> 
> Some of the vendors of this software haven't implemented (correctly) the
> entire WinSock2 advanced API.  You can test this by completely disabling
> the product and rebooting the machine, and if that fails, uninstalling the
> product, rebooting and retesting.
> 
> Also there are several options which cripple Apache to use older API's that
> are less likely to demonstrate these bugs in 3rd party socket stack drivers,
> you can try adding these directives globally to your server, one at a time,
> and report back which directive did resolve this double-headers bug;
> 
>   Win32DisableAcceptEx
>   EnableSendfile off
>   EnableMMap off
> 
> I'm not excluding the possibility of a bug, but trying to narrow down under
> what conditions the bug might exhibit itself, and at least getting your
> servers up and operating.
> 
> Those others who see this flaw, what extra software (MS Firewall, AV products,
> etc) are you running on the sockets?
> 
> 
Ok thanks for the hints.
Found the problem, my virusscanner (NOD32 2.50.25 mod: IMON) was messing up the
headers. Apache (mod_proxy) is working perfect now.

Thanks Again.



Comment 25 Dan Letecky 2005-12-07 23:20:48 UTC
(In reply to comment #23)
> You are all clear in the understanding that the various 'firewall' software
> products, anti-virus 'internet protection', and other add ins all insert 
> themselves into the network socket stack?

Hmm. I am also using Nod32/IMON. I guess you can close the bug now. Although I
haven't tried turning it off yet I'm sure Nod32 is causing this problem. I
already had problems with Nod32 causing very hard-to-find network errors on
software communicating via .NET Remoting.

Thanks!
Comment 26 Ruediger Pluem 2005-12-07 23:33:28 UTC
As the error seems to be caused by a virus scanner product installed on the
boxes I mark this report as invalid.