Bug 56261

Summary: Load balancing method bybusiness not working properly
Product: Apache httpd-2 Reporter: Enrico Bonato <enribnt>
Component: mod_proxy_balancerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED INVALID    
Severity: normal    
Priority: P2    
Version: 2.2.21   
Target Milestone: ---   
Hardware: Other   
OS: Linux   

Description Enrico Bonato 2014-03-14 10:39:06 UTC
We have set up this conifguration on an HTTPD reverse proxy:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

Header add Set-Cookie "APACHE_ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://AppCluster>
  BalancerMember https://node1.service.int route=1
  BalancerMember https://node2.service.int route=2
  BalancerMember https://node3.service.int route=3
  BalancerMember https://node4.service.int route=4
  ProxySet stickysession=APACHE_ROUTEID
  ProxySet lbmethod=bybusyness
</Proxy>

We expect requests to be balanced equally across all nodes; but the first node is overloaded
Here is the output of the balancer-manager

StickySession	Timeout	FailoverAttempts	Method
APACHE_ROUTEID	0	3 	bybusyness

Worker URL	Route	RouteRedir	Factor	Set	Status	Elected	To	From
https://node1.service.int	1		1	0	Ok	2380166	3.5G	892M
https://node2.service.int	2		1	0	Ok	203860	292M	78M
https://node3.service.int	3		1	0	Ok	203250	294M	79M
https://node4.service.int	4		1	0	Ok	351205	511M	132M


There is only one kind of request made, always of the same dimension
Comment 1 Eric Garreau 2014-03-14 11:03:07 UTC
hello

by-business is supposed to compare current input FIFO lengths of the balancer members prior to select the best choice for new sessions

it means that the distribution is likely to be "strange" in different use-cases:
 - your nodes may not be busy enough, meaning that the first one looks cool
 - your four nodes may have hidden extra load due to other internal tasks
 - requests processing of your business traffic might be "short" and "long",
    regardless of the real data size (by-business looks for queue lengths)
 - long-life sessions go where the sticky cookie says, and thus the first node
    may have to assume traffic which has been chosen in an unloaded state

are you able to gather the input FIFO size of your backend nodes and monitor that their input invocations are balanced ?

(FYI, I'm not a developer of mod_security_balancer but I already had to understand the same kind of pseudo-issue in the past)

try to compare how your backend nodes behave when you use the by-request policy (the console should look normal, but you might get different CPU loads on the nodes ?)

I hope this helps,
eric
Comment 2 Enrico Bonato 2014-03-18 09:38:42 UTC
Eric, thank you for the response. 

BTW, apply the byrequest method solved our issue; I can say it was just our lack of understanding about the bybusiness behaviour