Bug 63795 - mod_proxy_balancer: disabling balancer member often results in "AH01114: HTTP: failed to make connection to backend"
Summary: mod_proxy_balancer: disabling balancer member often results in "AH01114: HTTP...
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy_balancer (show other bugs)
Version: 2.4.25
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-01 09:16 UTC by jamesstocks
Modified: 2020-04-01 17:36 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description jamesstocks 2019-10-01 09:16:05 UTC
~~ Bug Description ~~

In the situation whereby an HTTP balancer is setup using mod_proxy_balancer and mod_proxy_http and proxy-initial-not-pooled is enabled, an disabling a balancer member can on occasion put the balancer member in an errored state and result in an "AH01114: HTTP: failed to make connection to backend" error written to the error.log.

We have seen this issue in our production environment and successfully reproduced it in our lab environment.

From examining multiple traffic captures, it appears the following is happening:

1) The balancer receives a HTTP request to be proxied
2) The balancer elects a balancer member and establishes a TCP connection with the elected balancer member
3) Before the balancer proxies any HTTP traffic over the established TCP connection, the balancer manager receives a request disable the elected balancer member
4) The balancer closes the established TCP connection and puts the balancer member into an error state rather than the expected disabled state, logging "AH01114: HTTP: failed to make connection to backend"
5) The balancer elects another balancer member and proxies the HTTP request as normal. The client making the request does not experience an error.

Our monitoring scrapes the HTTP balancer manager page each minute and alerts if any of the balancer members are in an error state. We run a continuous delivery pipeline and balancer members are disabled in the balancer whilst our software is deployed to them. We have 4 Apache2 HTTP balancers and 12 balancer members. Yesterday 71 deploys were executed on our pipeline, which means each balancer member was disabled, allowed to drain, deployed to and re-enabled 71 times. This resulted in 30 alerts from our monitoring for a balancer member being in an error state.

We had not observed this issue prior to upgrading to Apache 2.4.


~~ Expected Behaviour ~~

mod_proxy_balancer should not put an elected balancer member into an errored state when the established TCP connection is closed before the HTTP request is proxied because the balancer manager has recieved a POST request to disable the elected balancer member. In this situation mod_proxy_balancer should put the elected balancer member in a disabled state.


~~ Steps to reproduce ~~

Set up an apache2 http balancer (full configuraton below) with mod_proxy_balancer and mod_proxy_http. 4 balancer members each running apache2 server in stock Debian configuration with libapache2-mod-php installed. Each member server is serving up a php page an embedded random sleep (0 - 2 seconds):

	cat /var/www/html/index.php
	<html>
	<?php
			sleep(rand(0, 2));
	?>
			<head>
					<title>httpserver01.cw.lab says hello</title>
			</head>
			<body>
			  <h1>This page lovingly served by httpserver01</h1>
			  <h2>You're welcome!</h2>
			</body>
	</html>

ab (Apache Bench) is continuously making 50 concurrent requests targeted at the balancer:

	#!/bin/bash
	while : ; do 
		ab -c50 -n50000 http://wwwexample.cw.lab/ 2>&1 | grep -P '(Failed requests|Non-2xx responses):\s+\d+' | grep -v 'Failed requests:        0' | tee -a fails 
	done

Continuously cycle through the balancer members, disabling them, waiting 5 seconds and then re-enabling them:

	#!/bin/bash
	while : ; do
	  nonce=$(curl -s http://httpbalancer.cw.lab/viphttpbalancer/__balancer-manager | grep --colour -oPm 1 '\w{8}-(?:\w{4}-){3}\w{12}') || ( echo "Failed to get nonce" ; exit 1)

	  for i in {1..4} ; do
		echo Removing httpserver0$i from httpbalancer balancer
		curl -sS -w '%{http_code}' -o /dev/null 'http://httpbalancer.cw.lab/viphttpbalancer/__balancer-manager' --data 'b=vip-http-balancer' --data 'w_lf=10' --data 'w_ls=0' --data 'w_wr' --data 'w_rr' --data 'w_status_I=0' --data 'w_status_N=0' --data 'w_status_D=1' --data 'w_status_H=0' --data 'w_status_S=0' --data "w=http://192.168.13.7$i" --data 'b=vip-http-balancer' --data "nonce=$nonce"; echo

		sleep 5

		echo Re-adding httpserver0$i to httpbalancer balancer
		curl -sS -w '%{http_code}' -o /dev/null 'http://httpbalancer.cw.lab/viphttpbalancer/__balancer-manager' --data 'b=vip-http-balancer' --data 'w_lf=10' --data 'w_ls=0' --data 'w_wr' --data 'w_rr' --data 'w_status_I=0' --data 'w_status_N=0' --data 'w_status_D=0' --data 'w_status_H=0' --data 'w_status_S=0' --data "w=http://192.168.13.7$i" --data 'b=vip-http-balancer' --data "nonce=$nonce"; echo

		sleep 1
	  done
	done


~~ HTTP Balancer Environment/Configuration details ~~

Enabled MPM: worker
List of enabled modules:
  proxy_balancer (enabled by unknown)
   filter (enabled by maintainer script)
   alias (enabled by maintainer script)
   setenvif (enabled by maintainer script)
   dir (enabled by maintainer script)
   headers (enabled by unknown)
   lbmethod_byrequests (enabled by unknown)
   slotmem_shm (enabled by unknown)
   status (enabled by maintainer script)
   proxy (enabled by unknown)
   proxy_html (enabled by unknown)
   ssl (enabled by unknown)
   access_compat (enabled by maintainer script)
   deflate (enabled by maintainer script)
   env (enabled by maintainer script)
   proxy_http (enabled by unknown)
   authn_file (enabled by maintainer script)
   authz_core (enabled by maintainer script)
   authn_core (enabled by maintainer script)
   mime (enabled by maintainer script)
   authz_user (enabled by maintainer script)
   negotiation (enabled by maintainer script)
   rewrite (enabled by unknown)
   xml2enc (enabled by unknown)
   mpm_worker (enabled by unknown)
   socache_shmcb (enabled by unknown)
   auth_basic (enabled by maintainer script)
   authz_host (enabled by maintainer script)
List of enabled configurations:
   cw-lab-all-hosts (enabled by unknown)
   charset (enabled by maintainer script)
   serve-cgi-bin (enabled by maintainer script)
   localized-error-pages (enabled by maintainer script)


Package: apache2
Version: 2.4.25-3+deb9u7*

-- System Information:
Debian Release: 9.11
  APT prefers oldstable-updates
  APT policy: (500, 'oldstable-updates'), (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-11-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apache2 depends on:
ii  apache2-bin          2.4.25-3+deb9u7
ii  apache2-data         2.4.25-3+deb9u7
ii  apache2-utils        2.4.25-3+deb9u7
ii  dpkg                 1.18.25
ii  init-system-helpers  1.48
ii  lsb-base             9.20161125
ii  mime-support         3.60
ii  perl                 5.24.1-3+deb9u5
ii  procps               2:3.3.12-3+deb9u1

Versions of packages apache2 recommends:
pn  ssl-cert  <none>

Versions of packages apache2 suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
pn  www-browser                                      <none>

Versions of packages apache2-bin depends on:
ii  libapr1                  1.5.2-5
ii  libaprutil1              1.5.4-3
ii  libaprutil1-dbd-sqlite3  1.5.4-3
ii  libaprutil1-ldap         1.5.4-3
ii  libc6                    2.24-11+deb9u4
ii  libldap-2.4-2            2.4.44+dfsg-5+deb9u3
ii  liblua5.2-0              5.2.4-1.1+b2
ii  libnghttp2-14            1.18.1-1+deb9u1
ii  libpcre3                 2:8.39-3
ii  libssl1.0.2              1.0.2s-1~deb9u1
ii  libxml2                  2.9.4+dfsg1-2.2+deb9u2
ii  perl                     5.24.1-3+deb9u5
ii  zlib1g                   1:1.2.8.dfsg-5

Versions of packages apache2-bin suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
pn  www-browser                                      <none>

Versions of packages apache2 is related to:
ii  apache2      2.4.25-3+deb9u7
ii  apache2-bin  2.4.25-3+deb9u7

-- Configuration Files:
/etc/apache2/mods-available/mpm_worker.conf:
	<IfModule mpm_worker_module>
			StartServers             2
			MinSpareThreads          25
			MaxSpareThreads          75
			ThreadLimit              64
			ThreadsPerChild          25
			MaxRequestWorkers        400
			MaxConnectionsPerChild   0
	</IfModule>

/etc/apache2/mods-available/cw_lab_all_hosts.conf:
ProxyPreserveHost On
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled 1
SetEnv proxy-sendchunked 1

/etc/apache2/sites-available/viphttpbalancer.conf
<VirtualHost viphttpbalancer:80>
        ServerAdmin infrastructure@codeweavers.net

        DocumentRoot /var/www

        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>

        <Directory /var/www/>
                Options FollowSymLinks
                AllowOverride None
                Require all granted
        </Directory>

        <Location /__server-status>
                SetHandler server-status
        </Location>

        <Location /__balancer-manager>
                SetHandler balancer-manager
        </Location>

        <Location /__heartbeat>
                SetHandler server-status
        </Location>

        #This is not inherited, so activate each individually
        RewriteEngine On
        #Don't proxy things that start with __ !!
        RewriteRule ^/__(.*)$ /__$1 [L]

        <Proxy balancer://vip-http-balancer>
                BalancerMember http://192.168.13.71 loadfactor=10
                BalancerMember http://192.168.13.72 loadfactor=10
                BalancerMember http://192.168.13.73 loadfactor=10
                BalancerMember http://192.168.13.74 loadfactor=10
                ProxySet lbmethod=byrequests
        </Proxy>

        RewriteRule ^(.*)$ balancer://vip-http-balancer/$1 [P,QSA]
</VirtualHost>

/etc/apache2/sites-available/local_eth1.conf
<VirtualHost local_eth1:80>
        ErrorLog ${APACHE_LOG_DIR}/local_eth1.redir.error.log
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/local_eth1.redir.access.log combined

        SetOutputFilter proxy-html;INFLATE;DEFLATE

        ProxyPass /viphttpbalancer/ http://viphttpbalancer/
        <Location /viphttpbalancer/>
                ProxyHTMLURLMap /__balancer-manager /viphttpbalancer/__balancer-manager R
        </Location>
</VirtualHost>

/etc/apache2/sites-available/000-default.conf
<VirtualHost vip1:80>
        ServerName wwwexample.cw.lab
        ServerAlias wwwexample
        ServerAlias vip1

        ServerAdmin webmaster@localhost

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        RewriteEngine On
        RewriteRule ^/__(.*)$ /__$1 [L]

        RequestHeader set X-Forwarded-Proto "http"

        RewriteCond %{HTTP_HOST} ^.*\.cw\.lab$ [NC]
        RewriteRule ^(.*)$ http://viphttpbalancer/$1 [P,QSA]
</VirtualHost>


~~ Footnotes ~~

*2.4.25-3+deb9u8 is broken for our environment as per https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941202#5 / https://bz.apache.org/bugzilla/show_bug.cgi?id=63688
Comment 1 Codeweavers Infrastructure 2019-10-09 15:04:38 UTC
Reproduced on Debian 10 (2.4.38-3+deb10u1)