Bug 49417 - mod_proxy crosses connections when apache times out
Summary: mod_proxy crosses connections when apache times out
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.2.15
Hardware: PC Windows Server 2003
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL: http://www.apache.org/dist/httpd/patc...
Keywords: FixedInTrunk
Depends on:
Blocks:
 
Reported: 2010-06-09 11:26 UTC by Loren Anderson
Modified: 2010-06-11 16:58 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Loren Anderson 2010-06-09 11:26:10 UTC
As part of our software security remediations, we upgraded our site to use apache 2.2.13 from 2.2.2.

We found a critical issue with Apache 2.2.13 (and reproduced on 2.2.15). This issue does not exist in apache 2.2.2.

The basics of this problem are that when you use mod_proxy to connect to a backend tomcat server and apache is configured to timeout before tomcat, a connection to the backend tomcat server is placed back into the pool of existing connections with what ever data that was destined to the original caller waiting to be retrieved by the next caller to take this connection from the pool. In our application which is statefull, this lead to customer's sessions crossing and customer A saw customer B's data.

Setting DisableReuse=true or changing the timeout of apache to be larger that the tomcat server avoids this issue.

I was able to recreate the issue with a simple TOMCAT echo servlet that takes an incoming URL which contains a random number and returning that value in the body of an HTML response. The apache and tomcat servers run on a single Windows 7 box (I have also tried this on Windows Server 2003) configured to connect via localhost.

The client driver needs to have KEEPALIVE on (since the front end connection and back end connections seem to be linked from a life-span perspective). My sample client driver was written in ruby. It generates a random number and then checks that the response it gets is what it send. 

Apache is configured to have 5 pooled connections (max=5, min=5) and set to timeout in 10 seconds. TOMCAT is configured to timeout in 5 minutes. 

The client ruby code sleeps a random amount of time between 1 and 15 seconds. This leads to about 1/3 of the clients timeing out. I run 20 clients. Within a few seconds, the problem is reproduced (client does not see the number it send it).

Here is my apache configuration:

=== BEGIN httpd.conf ===
PidFile httpd.pid

Timeout 600
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 300
ServerTokens ProductOnly

FileETag -INode MTime Size

ServerSignature Off

ThreadsPerChild 1000
MaxRequestsPerChild  0

Listen 81

LoadModule authz_host_module    "modules/mod_authz_host.so"
LoadModule proxy_module         "modules/mod_proxy.so"
LoadModule proxy_http_module    "modules/mod_proxy_http.so"
LoadModule dir_module           "modules/mod_dir.so"
LoadModule log_config_module    "modules/mod_log_config.so"
LoadModule mime_module          "modules/mod_mime.so"
LoadModule rewrite_module       "modules/mod_rewrite.so"
LoadModule headers_module       "modules/mod_headers.so"

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

TypesConfig "conf/mime.types"

AddDefaultCharset   ISO-8859-1
MaxMemFree 1000000

ErrorLog "apache-error.log"
LogLevel info
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %A %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "apache-access-BK-PEWEB01-B1_APACHE.log" common
DocumentRoot "root"

<Directory "root">
    Options FollowSymLinks Includes
    AllowOverride None
    Order allow,deny
    Allow from all
    DirectoryIndex index.html
</Directory>

<Proxy *>
     AddDefaultCharset Off
     Order deny,allow
     Allow from all
</Proxy>

ProxyPass /test http://localhost:9090/test min=5 max=5 timeout=10
ProxyPassReverse /test http://localhost:9090/test
=== END httpd.conf ===

Here is the TOMCAT configuration:

=== BEGIN server.xml ===
<Server
  port="8005"
  shutdown="SHUTDOWN"
  debug="0"
>
    <Service name="Tomcat">
        <Connector
            port="9090"
            maxThreads="1000"
            minSpareThreads="25"
            maxSpareThreads="75"
            enableLookups="false"
            redirectPort="8443"
            acceptCount="100"
            debug="0"
            connectionTimeout="300000"
            disableUploadTimeout="true"
            server="THX1138"
            proxyName="pe1.barra.com"
            proxyPort="443"
        />
        <Engine
          name="Standalone"
          defaultHost="localhost"
          debug="0"
        >
            <Host
              name="localhost"
              debug="0"
              appBase="webapps"
              unpackWARs="true"
            >
                <Context
                  useHttpOnly="true"
                  reloadable="false"
                  path="/test"
                  docBase="c:/workspace/webapps/test"
                  debug="0"
                  privileged="true"
                />
            </Host>
        </Engine>
    </Service>
</Server>
=== END server.xml ===

Here is my simple eacho servlet:

=== BEGIN EchoServlet.java ===
import com.sun.mail.util.UUEncoderStream;
import org.apache.derby.iapi.util.StringUtil;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Random;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class EchoServlet extends javax.servlet.http.HttpServlet implements javax.servlet.Servlet {
    private static Random rand = new Random(System.nanoTime());
    public EchoServlet() {
        super();
    }

    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        doPost(request, response);
    }

    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        response.setContentType("text/html");

        String uri = request.getRequestURI();

        if(uri.startsWith("/test/")) {
            String num = uri.replace("/test/", "");
            long rid = Long.parseLong(num);
            PrintWriter writer = response.getWriter();
            try {
                Thread.sleep(1000 * rand.nextInt(17));
            } catch(InterruptedException e) {}
            writer.print("<html>");
            writer.print("<body>");
            writer.print(rid);
            writer.print("</body>");
            writer.print("</html>");
            writer.close();
        }
    }
}
=== END EchoServlet.java ===

Here is my client ruby driver:

=== BEGIN echoclient.rb ===
require "socket"
require "cgi"

$printHeader = false
class Response
  attr_reader :socket, :header, :content

  def initialize socket
    @socket = socket
    @header = @content = ""
    contentlength = 0

    while (line = @socket.gets) != nil
      break if line == "\r\n"
      puts line if $printHeader
      @header << line
      case line
      when /^content-length: (\d+)/i
        contentlength = $1.to_i
      end
    end
    if contentlength != 0
      @content = @socket.read contentlength
    end
  end
end

socket = nil
connected = false

while true
    if socket == nil || socket.closed?
      connected = false
      socket = nil
      print '='
      $stdout.flush
    end
    while !connected
       begin
            socket = TCPSocket.new('localhost', 81)
            connected = true
        rescue
            puts "Got error, sleeping"
            sleep 5
            next
        end
    end
    num = rand(2000000000)
    #puts "Input: #{num}"
    req = "GET /test/#{num} HTTP/1.1\r\nConnection: Keep-Alive\r\nHost: 127.0.0.1\r\n\r\n"
    socket.write req
    socket.flush

    resp_num = 0
    response = Response.new socket
    if response.content.length == 0
      print '&'
      $stdout.flush
    end
    if response.content =~ /<HTML><BODY>([0-9]+)<\/BODY><\/HTML>/io
            resp_num = $1.to_i
    end
    if resp_num == 0
      $printHeader = true
      print '0'
    elsif num != resp_num
      $printHeader = false
      puts num
      puts resp_num
      print '-'
    else
      $printHeader = false
      print '*'
    end
    $stdout.flush
end
=== END echoclient.rb ===
Comment 1 Ruediger Pluem 2010-06-09 11:49:06 UTC
Can you please provide the error log file that is created during your tests?
Comment 2 Ruediger Pluem 2010-06-09 11:50:03 UTC
Ah forgot. Please set the loglevel to debug before.
Comment 3 Rainer Jung 2010-06-09 12:13:11 UTC
... and finally: if the problem really is triggered by Apache having a shorter timeout than Tomcat, I would expect it to be much more likely that the source of the problem would be a resued buffer in Tomcat.

So could you please also provide your Tomcat version and whether you are using tcnative (also known as Tomcat Native Connector or APR connector).
Comment 4 Loren Anderson 2010-06-09 12:29:08 UTC
The problem is that TOMCAT does not know that apache has placed this connection back in the pool. There is nothing done to the connection to inform TOMCAT that the connection is going to be reused by another client request. The TOMCAT version is 2.2.27. The provided tomat server.xml shows which connector we are using (coyote).
Comment 5 Rainer Jung 2010-06-10 05:15:50 UTC
(In reply to comment #4)
> The problem is that TOMCAT does not know that apache has placed this connection
> back in the pool. There is nothing done to the connection to inform TOMCAT that
> the connection is going to be reused by another client request. The TOMCAT
> version is 2.2.27. The provided tomat server.xml shows which connector we are
> using (coyote).

You mean Tomcat version 5.5.27? There is no version 2.2.x for Tomcat.

Concerning the connector: the configuration doesn't tell us which implementation it is. If Tomcat finds the tcnative DLL during startup it will automatically launch the APR connector instead of the Java Blocking IO connector.

So please double check:

APR during startup:

10.06.2010 11:10:38 org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080

Bocking IO:

10.06.2010 11:13:43 org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080

Note: "Http11AprProtocol" vs. "Http11BaseProtocol".

Please do also respond to Rüdiger's questions.

Thanks.
Comment 6 William A. Rowe Jr. 2010-06-11 16:58:47 UTC
This is resolved in svn for 2.2.16 and 2.3.6-alpha releases.  Thank you for
the thorough details and research that went into your report!

Note this affected also Netware and OS2 ports, but not *nix'es.

The simplest workaround is to globally configure;

SetEnv proxy-nokeepalive 1