Bug 46682

Summary: mod_proxy EAI_AGAIN DNS failure
Product: Apache httpd-2 Reporter: David Multer <david>
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: normal Keywords: MassUpdate
Priority: P2    
Version: 2.2.11   
Target Milestone: ---   
Hardware: Macintosh   
OS: Mac OS X 10.4   

Description David Multer 2009-02-09 09:26:39 UTC
I have an installation HTTPD running on a Mac OS X 10.5 system that uses mod_proxy to front a Tomcat 6 installation.

Here's my httpd.conf excerpt for the virtual host:
    ProxyPass /hudson/ ajp://sub.domain.com:8009/app/
    ProxyPassReverse /hudson/ ajp://sub.domain.com:8009/app/

The proxy works fine, but after some amount of time (days), I start getting a "502 GET /app/ error reporting a DNS failure for sub.domain.com". Once it gets in this state, I must stop and then restart Apache (a restart won't fix it). While in this state, Tomcat is running fine as I can get directly to it with a port 8080 reference.

I inserted log messages directly into the code and have determined that APR is getting an EAI_AGAIN return value from getaddrinfo() after running for a while. I inserted the messages into these files:
- modules/proxy/proxy_util.c
- srclib/apr/network_io/unix/sockaddr.c

I've verified that DNS calls are actually still working in other processes via "host sub.domain.com", etc. There are no other problems seen on this server when it gets into this state.

Note that I've compiled and run Apache on Mac OS X 10.5.6 (latest). I've been running Apache for many years on this server with no issues until I started using mod_proxy with Tomcat.

My Apache ./configure is as follows. I built and run Apache on the same Mac system.
./configure \
--prefix=/usr/local/apache2 \
--enable-mods-shared=all \
--enable-proxy-balancer \
--enable-proxy-ajp \
--enable-proxy-http \
--enable-proxy-ftp \
--enable-proxy-connect \
--enable-proxy \
--enable-ssl \
--with-included-apr \
--enable-dav \
--with-berkeley-db=/usr/local/bdb \
--with-dbm=db4 \
--enable-so

It appears to me that Apache/mod_proxy is leaking memory somehow. I have not been able to track down the leak, or identify a test case that will reliably reproduce the problem.
Comment 1 Jon Auman 2009-03-20 08:38:28 UTC
I had the EXACT same issue. Name resolution would completely break for all my virtual hosts every 4 - 10 hours it seemed. It did not matter whether is was a database connection or ProxyPass directive in the virtual host. DNS would break for *all* virtual hosts.

I'm also on the latest Leopard version:  OS X 10.5.6
We are running Apache 2.2.11 compiled from source, 32-bit mode (for compatibility with some modules)

./configure --prefix=/usr/local/apache2 --enable-layout=Darwin --enable-mods-shared=all --enable-cache --enable-disk-cache --enable-mem-cache --enable-ssl --enable-proxy --enable-dav --enable-dav-fs --enable-dav-lock --enable-so --enable-rewrite


I found a workaround. It appears to be a keepalive issue. I wrote a LaunchDaemon that would curl one of my proxies every couple of minutes. If the http status code was anything but 200, the script would restart apache.

The funny thing is, apache has not been restarted in days since I launched the daemon. that makes me think it is a keepalive issue.

I have not tried it yet, but adjusting some of the apache environment variables for mod_proxy may help:

force-proxy-request-1.0, proxy-nokeepalive, proxy-sendchunked, proxy-sendcl, proxy-chain-auth, proxy-interim-response, proxy-initial-not-pooled

We have had so many issues with Apache on OS X, though, that we are looking to move to Linux in the near future.
Comment 2 David Multer 2009-05-21 18:42:01 UTC
I too created a cron job to curl one of my virtual host Tomcat apps (via mod_proxy) every 15 minutes and the problem has gone away for me too. Very interesting... I played with the various Apache proxy settings, but wasn't able to avoid the problem through those options. I does appear to only manifest if the site is idle for long periods of time.

Mac OS X continues to be a great LAMP platform for me otherwise, and I have a ton of open source packages running with no issue. I hope someone on the Apache dev team takes notice and is able to help on this.
Comment 3 kilaru 2009-05-27 06:48:09 UTC
Hi

Have you tried starting apache in system startup context? With that you might be able to avoid ping apache every couple of minutes. 

--Satish
Comment 4 David Multer 2009-05-27 07:02:11 UTC
Pinging Apache is only a way to avoid the problem, it's not the root issue. Apache is already started at system startup. If you have more questions, I would suggest email rather than bug comments.
Comment 5 William A. Rowe Jr. 2018-11-07 21:09:52 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.