Bug 58529

Summary:	Segmentation fault after modifying a proxy BalancerMember
Product:	Apache httpd-2	Reporter:	Antoine Prevosto <antoine.prevosto>
Component:	mod_proxy_http	Assignee:	Apache HTTPD Bugs Mailing List <bugs>
Status:	NEW ---
Severity:	major	CC:	sygoulmy
Priority:	P2
Version:	2.4.16
Target Milestone:	---
Hardware:	Other
OS:	Linux

Description Antoine Prevosto 2015-10-23 16:14:57 UTC

With are using Apache 2.4 as an HTTP reverse proxy, and have activated the BalancerPersist option to preserve balancer member status update when doing graceful restart.

If we modify an existing balancer member definition (modifying the port number or the protocol), the child process crashes with a segmentation fault when proxifying a request

This behavior may be easily reproduced.

Comment 1 Antoine Prevosto 2015-10-23 16:26:04 UTC

Apache is 2.4.16.

Our vhost contains this directives :

<Proxy balancer://tst_antoine_um_unmanaged>
    ProxySet lbmethod=bybusyness stickysession=ASID|asid growth=1
    BalancerMember http://http1qvit-15.xxx:80 lbset=0 connectiontimeout=5 retry=60 ttl=10 timeout=60 route=a1s1
</Proxy>

Then the balancer-manager displays this balancer as follow :

************************************************
Server Version: Apache/2.4.16 (Unix) OpenSSL/1.0.1e-fips
Server Built: Oct 6 2015 15:58:34
Balancer changes will be persisted on restart.
Balancers are NOT inherited from main server.
ProxyPass settings are inherited from main server.
LoadBalancer Status for balancer://tst_antoine_um_unmanaged [p7cf634bb_tst_antoine_um_unmanaged]

MaxMembers	StickySession	DisableFailover	Timeout	FailoverAttempts	Method	Path	Active
2 [1 Used]	ASID | asid	Off	0	0	bybusyness	/	Yes

Worker URL	Route	RouteRedir	Factor	Set	Status	Elected	Busy	Load	To	From
http://http1qvit-15.xxxx a1s1		1	0	Init Ok	1	0	0	208	1.4K
************************************************

Now, we change the port number into the configuration file, so the BalancerMember directive is now :

BalancerMember http://http1qvit-15.xxxx:8080 lbset=0 connectiontimeout=5 retry=60 ttl=10 timeout=60 route=a1s1


Then, we reload this configuration (graceful restart). After reloading, the balancer-manager shows that a new balancer member has been added (this seems strange to me) :

************************************************
Server Version: Apache/2.4.16 (Unix) OpenSSL/1.0.1e-fips
Server Built: Oct 6 2015 15:58:34
Balancer changes will be persisted on restart.
Balancers are NOT inherited from main server.
ProxyPass settings are inherited from main server.
LoadBalancer Status for balancer://tst_antoine_um_unmanaged [p7cf634bb_tst_antoine_um_unmanaged]

MaxMembers	StickySession	DisableFailover	Timeout	FailoverAttempts	Method	Path	Active
2 [2 Used]	ASID | asid	Off	0	1	bybusyness	/	Yes

Worker URL	Route	RouteRedir	Factor	Set	Status	Elected	Busy	Load	To	From
http://http1qvit-15.xxxx:8080	a1s1		1	0	Init Ok	0	0	0	0	0
http://http1qvit-15.xxxx	a1s1		1	0	Init Ok	1	0	0	208	1.4K
************************************************

Finally, when sending a request to this virtual host, the child process crashes when trying to proxify the request to the new balancer member (in this case, no server is listnening on port 8080).

Here is the error log :

************************************************
[Fri Oct 23 15:45:46.848308 2015] [mpm_worker:notice] [pid 25396:tid 140519901787968] AH00292: Apache/2.4.16 (Unix) OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Fri Oct 23 15:45:46.848329 2015] [core:notice] [pid 25396:tid 140519901787968] AH00094: Command line: '/tech/apache/runtime/httpd-2.4.16/bin/httpd -D inst-antoine-back-tomcat7_a -d /tech/apache/a090966 -f /tech/apache/a090966/etc/main.conf -T -D connector -D hapi -D qos -D cache -D ssl -D standard -D proxy -D arch-64 -D env-dev -D site-tls -D habile-r5 -D target-any -D localIP'
[Fri Oct 23 15:45:46.848347 2015] [mpm_worker:warn] [pid 25396:tid 140519901787968] AH00291: long lost child came home! (pid 27493)
[Fri Oct 23 15:45:46.848359 2015] [mpm_worker:warn] [pid 25396:tid 140519901787968] AH00291: long lost child came home! (pid 27494)
[Fri Oct 23 15:45:46.848376 2015] [mpm_worker:warn] [pid 25396:tid 140519901787968] AH00291: long lost child came home! (pid 27495)

[Fri Oct 23 15:49:03.631823 2015] [proxy:error] [pid 22019:tid 140519652980480] (111)Connection refused: AH00957: HTTP: attempt to connect to 10.60.88.70:8080 (http1qvit-15.xxxx) failed
[Fri Oct 23 15:49:03.631858 2015] [proxy:error] [pid 22019:tid 140519652980480] AH00959: ap_proxy_connect_backend disabling worker for (http1qvit-15.xxxx) for 60s
[Fri Oct 23 15:49:03.631868 2015] [proxy_http:error] [pid 22019:tid 140519652980480] [client 10.70.130.50:52700] AH01114: HTTP: failed to make connection to backend: http1qvit-15.xxxx
[Fri Oct 23 15:49:04.031590 2015] [core:notice] [pid 25396:tid 140519901787968] AH00051: child pid 22019 exit signal Segmentation fault (11), possible coredump in /tech/apache/a090966/var/logs/antoine-back-tomcat7_a
************************************************

Comment 2 Antoine Prevosto 2015-10-23 16:31:13 UTC

Here is the main information from rhe core file :

Core was generated by `/tech/apache/runtime/httpd-2.4.16/bin/httpd -D inst-antoine-back-tomcat7_a -d /'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fcd5221e954 in ap_proxy_acquire_connection (proxy_function=0x7fcd5200e1fe "HTTP", conn=0x7fcd47ffeac8, worker=0x2269438, s=0x227c258) at proxy_util.c:2124
2124        if (worker->s->hmax && worker->cp->res) {
Missing separate debuginfos, use: debuginfo-install AF-httpd-2.4.16-0-el6.4.x86_64
(gdb) bt
#0  0x00007fcd5221e954 in ap_proxy_acquire_connection (proxy_function=0x7fcd5200e1fe "HTTP", conn=0x7fcd47ffeac8, worker=0x2269438, s=0x227c258) at proxy_util.c:2124
#1  0x00007fcd5200b879 in proxy_http_handler (r=0x7fcd1c004980, worker=0x2269438, conf=0x229a638, url=0x7fcd1c006e20 "http://http1qvit-15.xxxx/", proxyname=0x0, proxyport=0) at mod_proxy_http.c:1957
#2  0x00007fcd52214f12 in proxy_run_scheme_handler (r=0x7fcd1c004980, worker=0x2269438, conf=0x229a638, url=0x7fcd1c006e20 "http://http1qvit-15.xxxx/", proxyhost=0x0, proxyport=0) at mod_proxy.c:2789
#3  0x00007fcd52219fa3 in proxy_handler (r=0x7fcd1c004980) at mod_proxy.c:1161
#4  0x000000000044b210 in ap_run_handler ()
#5  0x000000000044f38e in ap_invoke_handler ()
#6  0x000000000046201a in ap_process_async_request ()
#7  0x000000000046217f in ap_process_request ()
#8  0x000000000045e4ce in ap_process_http_connection ()
#9  0x0000000000456020 in ap_run_process_connection ()
#10 0x00007fcd54e8bf2a in process_socket (thd=0x224bb90, dummy=<value optimized out>) at worker.c:619
#11 worker_thread (thd=0x224bb90, dummy=<value optimized out>) at worker.c:978
#12 0x00007fcd55a379d1 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fcd557848fd in clone () from /lib64/libc.so.6
(gdb) info threads
  27 Thread 0x7fcd4cd6c700 (LWP 22023)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  26 Thread 0x7fcd4d76d700 (LWP 22022)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  25 Thread 0x7fcd4e16e700 (LWP 22021)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  24 Thread 0x7fcd3a3e9700 (LWP 22046)  0x00007fcd55784ef3 in epoll_wait () from /lib64/libc.so.6
  23 Thread 0x7fcd3adea700 (LWP 22045)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  22 Thread 0x7fcd3b7eb700 (LWP 22044)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21 Thread 0x7fcd3c1ec700 (LWP 22043)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20 Thread 0x7fcd3cbed700 (LWP 22042)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 Thread 0x7fcd3d5ee700 (LWP 22041)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 Thread 0x7fcd3dfef700 (LWP 22040)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17 Thread 0x7fcd3e9f0700 (LWP 22039)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  16 Thread 0x7fcd3f3f1700 (LWP 22038)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15 Thread 0x7fcd3fdf2700 (LWP 22037)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14 Thread 0x7fcd407f3700 (LWP 22036)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  13 Thread 0x7fcd411f4700 (LWP 22035)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12 Thread 0x7fcd41bf5700 (LWP 22034)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11 Thread 0x7fcd425f6700 (LWP 22033)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 Thread 0x7fcd42ff7700 (LWP 22032)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 Thread 0x7fcd439f8700 (LWP 22031)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 Thread 0x7fcd443f9700 (LWP 22030)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 Thread 0x7fcd44dfa700 (LWP 22029)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x7fcd457fb700 (LWP 22028)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 0x7fcd461fc700 (LWP 22027)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x7fcd46bfd700 (LWP 22026)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 0x7fcd475fe700 (LWP 22025)  0x00007fcd55a3b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 Thread 0x7fcd56d47740 (LWP 22019)  0x00007fcd55a3e75d in read () from /lib64/libpthread.so.0
* 1 Thread 0x7fcd47fff700 (LWP 22024)  0x00007fcd5221e954 in ap_proxy_acquire_connection (proxy_function=0x7fcd5200e1fe "HTTP", conn=0x7fcd47ffeac8, worker=0x2269438, s=0x227c258) at proxy_util.c:2124