Bug 61140

Summary: Add hot spares to load balancers
Product: Apache httpd-2 Reporter: Jim Riggs <jhriggs>
Component: mod_proxy_balancerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Severity: enhancement    
Priority: P2    
Version: 2.5-HEAD   
Target Milestone: ---   
Hardware: All   
OS: All   
Attachments: hot spare functionality
patch v2
patch v3
callback patch
callback patch v2

Description Jim Riggs 2017-05-31 17:33:59 UTC
As discussed at ApacheConNA 2017 in Miami a couple of weeks ago and followed up on the mailing list (http://marc.info/?l=apache-httpd-dev&m=149554902825812), here is a patch to add a new type/status to balancer members: R - hot spare. These workers will be treated as drop-in replacements for unusable workers without all workers needing to be unavailable (as for H - hot standby). This allows for maintaining a consistent number of available workers in the balancer.

"Normal" workers always take precedence over spares. For each "normal" worker that is unusable (failed, stopped, disabled, unreachable, etc.), though, a spare will be added to the list of available workers to try to maintain the desired number of members.

The find_best_by(requests|busyness|traffic) functions were 99% duplicated code. I combined all of that logic into a single ap_proxy_balancer_usable_workers() function in proxy_util.c. (Is that the best place for it?) That function contains all of the logic for iterating through lbsets and returning an array of the available workers that the lbmethod should choose from, whether they be normal workers, spares, or hot standbys. Thus, the lbmethods only need to worry about their specific heuristics for choosing the best worker, not determining which workers are available and which is best.

Balancer-manager management is implemented. AND, I even updated the xml documentation(!!!).

I did not implement hot spare support in mod_lbmethod_heartbeat, as it seems all but dead...and maybe should be deprecated and removed due to hcheck? I do have some code stubs if we really want to add it, though.
Comment 1 Jim Riggs 2017-05-31 17:38:01 UTC
Created attachment 35021 [details]
hot spare functionality

For those who like to see a Github-type view, see https://github.com/jhriggs/httpd/pull/1/files.
Comment 2 Jim Riggs 2017-06-01 20:38:11 UTC
Created attachment 35022 [details]
patch v2

Fix superfluous ap_proxy_retry_worker_fn
Comment 3 Jim Riggs 2017-06-01 20:46:58 UTC
Created attachment 35023 [details]
patch v3

Grr...missed the prototype.
Comment 4 Jim Riggs 2017-06-02 03:19:59 UTC
Created attachment 35024 [details]
callback patch

- use a callback and baton to the lbmethod in
  ap_proxy_balancer_get_best_worker() (previously
  ap_proxy_balancer_usable_workers()) so that we can find the best worker in
  flight while iterating rather than returning an array of usable workers that
  has to then in turn be iterated

- consider a draining worker unusable and suitable for replacement by a spare

(I continue to push these to my fork on github, so visual diff/PR can still be seen @ https://github.com/jhriggs/httpd/pull/1/files.)
Comment 5 Jim Riggs 2017-06-02 03:22:39 UTC
Created attachment 35025 [details]
callback patch v2

Previous callback patch was mistakenly not against trunk.
Comment 6 Jim Riggs 2018-04-11 12:36:43 UTC
Committed to trunk (https://svn.apache.org/r1828890) and proposed for backport to 2.4 (https://svn.apache.org/r1828893).