The situation is a backend which is not apache httpd and where a successful
response splitting attack can be done (It can be because of an application flaw
or because of a bad implementation of something in the http protocol), and
mod_proxy is used as a reverse proxy for this backend.

I'd like mod_proxy to implement a basic detection of a bad behavior from the
backend by closing the keep-alive connection to this backend.

The current situation with Apache httpd 2.4.12 and before is that mod_proxy
trust the backend and does not even detect obvious problems, like having new 
http responses coming back from a backend before sending new requests.

Note that this was discussed previously in the security mailing list. The
problem is not on httpd side, so it's not a security problem, it is simply an
hardening feature for mod_proxy to reduce impacts of bad things when they
happen.

For example, here is what happen if the response splitting attacks generates two
responses when mod_proxy sends one request to the backend.


     HTTP Client    | mod_proxy       |  Compromised backend
                    |                 |
      query1 -------|--> query1       |
                    |      \----------|--> sending back 2 responses
                    |                 |   response1+BAD response
                    |  read only  <---|---------/
                    |  response1 [*]  |
                    |      /          |
      receive  <----|------           |
     response1      |                 |
                    |    [ BAD response is still there, stacked
                    |     in the mod_proxy<->backend connection
                    |     if the keep-alive status is maintained ]
                    |                 |
      query2 -------|--> query2  [**] |
                    |      \----------|--> sending back
                    |                 |      response2
                    |      read <-----|------/
                    |    BAD response | <======= here is the problem
      receive  <----|----/            |
       BAD response |

This can be tested also by sending a pipeline of two queries, where the first
query contains a splitting attack for the backend. mod_proxy will send back the
splitted response for the second pipelined query (for tests you can also use a
php script sending two http responses instead of one, as described below).

If the splitting attack can generate 100 http responses, theses 100 responses
will be present in the socket connection to the backend. The next 100 queries
from mod_proxy directed to the same backend and reusing this tainted connection
from the pool will read theses false 100 responses, one by one, and send the
results to the wrong users (or to the wrong caches entries).

The main problem here is not in httpd, it's the compromised backend, but shit
happens.

What mod_proxy could detect, is that this backend was sending http responses
without any http request; This could be detected in [*] or [**], i.e. when the
1st response is extracted from the brigade or when the connection was requested
from the connection pool, before sending a new query in this connection.
**The backend socket should be empty** when we have read the response or when we
want to send a request.

The only way to have something in the socket would be using pipelines on the
backend communications, and AFAIK mod_proxy is not using pipelines, even when
pipelined queries are received mod_proxy is splitting each query in a classical
1-request/1-response mode. And I don't think the expected-100-continue mode
could impact that. I'm pretty sure nothing should allow contents in this
connection before we send the request, but I'm maybe wrong.

The new way of handling the reverse proxy job could be expressed this way:

You have a socket opened in keepalive mode with a backend:
 1 - You send only one query, never a pipeline of queries (done)
 2 - you read 1 response even if several responses came back (done)
 3 - you send the unique response to the initial requester (done)
 4 - when receiving a new request (or for the next one the pipeline)
     you ask the pool for a new connection to the backend and the pool ensure
     that this connection is empty (no response waiting in the socket).
 5 - you send the next query (back to step 1)

Closing the backend connection if the socket is not empty is the most secure
thing, but you could also just flush the socket content.

Finally, adding such protection in mod_proxy would not prevent all http
splitting attacks from poisoning httpd's cache or mod_proxy request/responses
matching. Combined with the right timing a splitting attack could prevent the
extra-responses to be send too early and could successfully wait until new
requests are made by mod_proxy in this same connection (the socket would be
empty at step 4).
But adding temporisation in an http spitting attack is a complex task, and is
not always available. And even if this protection is not absolute it would at
least remove a very comfortable situation for the attacker where a big number
of http responses can be stacked to mod_proxy right after the end of the first
response.

How to reproduce
-----------------

This is an example, tested on apache 2.4.12:

- port 8888 apache webserver with mod_proxy
- port 80 apache webserver (or nginx, anything) with php support (php-fpm in m
  y case), PHP is used here to mimic a compromised application.

    <VirtualHost *:8888>
        ServerAdmin webmaster@dummy-host2.example.com
        DocumentRoot "/opt/apache2/htdocs"
        ServerName proxy-host.example.com
        ErrorLog "logs/proxy.example.com-error_log"
        CustomLog "logs/proxy.example.com-access_log" common
        ProxyPass / http://www.dummy-host.example.com:80/
        ProxyPassReverse / http://www.dummy-host.example.com:80/
        LogLevel trace8 proxy:trace8
    </VirtualHost>

On port 80 we add support for PHP

    <VirtualHost *:80>
        DocumentRoot "/opt/apache2/docs/dummy-host.example.com"
        ServerName dummy-host.example.com
        ServerAlias www.dummy-host.example.com
        DirectoryIndex index.php
        ProxyPassMatch ^/.*\.php(/.*)?$ fcgi://127.0.0.1:9000/opt/apache2/docs/dummy-host.example.com
        <Directory /opt/apache2/docs/dummy-host.example.com>
            Require all granted
        </Directory>
    </VirtualHost>

And finally the /opt/apache2/docs/dummy-host.example.com/index.php file:

    <?
    # I know that writing such script is killing your
    # own server. But PHP is not the only way of sending
    # 2 responses for one query. that's a test script.
    # 1st regular HTTP response ---------
    header("Content-Length: 17");
    
    echo "this may be ok.\r\n";
    # 2nd HTTP response ---------
    echo "HTTP/1.1 200 OK\r\n";
    echo "Server: Fake\r\n";
    echo "Date: Tue, 17 Mar 2015 21:33:18 GMT\r\n";
    echo "Content-Type: text/html\r\n";
    echo "Content-Length: 28\r\n";
    echo "Connection: keep-alive\r\n\r\n";
    echo "THIS SHOULD NEVER BE SEEN!\r\n";
    exit(0);

By requesting the index.php file via mod_proxy several times you should get the 
"THIS SHOULD NEVER BE SEEN!" result incoming.