Bug 51511 - apr_memcache_server_create creates connection pool with invalid TTL
Summary: apr_memcache_server_create creates connection pool with invalid TTL
Alias: None
Product: APR
Classification: Unclassified
Component: APR (show other bugs)
Version: HEAD
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache Portable Runtime bugs mailinglist
Depends on:
Reported: 2011-07-14 21:54 UTC by Tim Whittington
Modified: 2012-09-27 20:14 UTC (History)
0 users

Patch to correct memcache pool ttl (799 bytes, application/octet-stream)
2011-07-14 21:54 UTC, Tim Whittington

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Whittington 2011-07-14 21:54:50 UTC
Created attachment 27285 [details]
Patch to correct memcache pool ttl

The ttl parameter to apr_memcache_server_create is documented as:

"time to live in seconds of a client connection"

Subversion (in my case 1.6.11, but still the same in trunk), calls this with the value 50.

This parameter is passed directly to apr_reslist_create without modification, however the ttl parameter to apr_reslist_create is a 		apr_interval_time_t microseconds value.

The net result of this is that connections to memcache are closed immediately after they are returned to the pool by the Subversion process. 
It seems that the pool is using a 50 microsecond TTL value, and closing any connection pretty much immediately after use.

On our Subversion server, we observe port exhaustion, with > 30,000 TCP sockets to the memcache port in TIME_WAIT (unless we turn on very aggressive time_wait reuse/recycle options).

A tcpdump capture on that server reveals every connection is closed with a memcache quit command, and tracing back through the code this is only called from the conn_clean pool cleanup handler.

Patching apr_memcache_server_create to multiply the TTL by 1,000,000 fixes the issue on our server (as observed by the number of TIME_WAIT sockets and connections to memcached).

Patch against 1.3.9 (which should apply cleanly against trunk) is attached.
Comment 1 Jeff Trawick 2012-09-26 14:49:46 UTC
committed to apr trunk and apr-util 1.5.x and 1.4.x branches
Comment 2 Jeff Trawick 2012-09-27 20:14:27 UTC
The change has been reverted.

The documentation had been previously fixed to indicate that ttl is in microseconds instead of seconds.