Making mod_jk replication aware (Clustering Support)

At the moment mod_jk makes it's load-balancing decisions to workers according to
the following parameters:

- session-stickyness (choose the "only" right worker)
- locality (one worker is preferred)
- weight-based

What is missing, is any information about secondary cluster nodes.

Use Case 1: Horizontal scalability and performance of tomcat cluster

Tomcat cluster does only allow session replication to all nodes in the cluster.
Once you work with more than 3-4 nodes there is too much overhead and risk in
replicating sessions to all nodes. We split all nodes into clustered groups. I
introduced a new worker attribute "domain" to let get_most_suitable_worker in
mod_jk know, to which other nodes a session gets replicated (all workers with
the same value in the domain attribute). So a load balancing worker knows, on
which nodes the session is alive. If a node fails or is being taken down
administratively, mod_jk chooses another node that has a replica of the session.

Use Case 2: Prevent thread count explosion

Once mod_jk connects an apache process to a tomcat instance, the tomcat jk
connector will need one thread for this connection as long as the apache process
is alive, independent from the number of requests apache actually sends to tomcat.

Now assume you have multiple Apaches and Tomcats. The Tomcats are clustered and
mod_jk uses sticky sessions. Now you are going to shut down (maintenance) one
tomcat. All Apache will start connections to all tomcats. You end up with all
tomcats getting connections from all apache processes, so the number of threads
needed inside the tomcats will explode.

If you group the tomcats to domain as in use case 1, the connections normally
will stay inside the domain and you will need much less threads. 

Implementation

The implementation uses and additional worker attribute domain, and if a
session has a jvmRoute, is sticky and the correct worker is in error state
another worker with the same domain attribute is chosen.

I have an implementation based on mod_jk 1.2.6 running successfully in
production, but I would have to adapt to the 1.2.7 code changes. The
implementation would only concern common/jk_util.c for the additional
configuration worker attribute "domain" and common/jk_lb_worker.c for using the
attribute 
   - with stickyness in case the primary worker is down to first seek for some
other worker with the same domain attribute. 
   - without stickyness to decide to which workers an existing session is
allowed to be balanced (all workers with the same domain as the worker given in
the session id)

I could provide the code, if you are interested.

Here is a more concrete example:

Enterprise application with redundant internet connections A and B.
A consists of two Apache instances A.a1 und A.a2, B of B.a1 and B.a2.
Behind are 4 Tomcat A.t1, A.t2, B.t1, B.t2.

A.t1 and A.t2 are clustered, B.t1 and B.t2 are clustered. mod_jk uses load
balancing with sticky sessions.

All Apaches can connect to any Tomcat, but A.t1 is local for A.a1, A.t2
for A.a2, B.t1 for B.a1 and B.t2 for B.a2:

A.a1   A.a2   B.a1   B.a2
 ||  X  ||  X  ||  X  ||
A.t1---A.t2   B.t1---B.t2

A.t1 and A.t2 are put into the same "domain" "A" in workers.properties,
B.t1 and B.t2 are put into the same "domain" "B" in workers.properties.

Now if you shutdown e.g. tomcat B.t1 for service/update (or if it breaks)
All apaches will know from the domain configuration, that sticky requests
for B.t1 have to go to B.t2. This is important, since only on that tomcat
the replicated sessions from B.t1 will exist.

Without domains you have to put all the Tomcats in one cluster. But then
all sessions are replicated to all tomcats. We have a production side
using 3x3=9 tomcats and a cluster with 9 nodes would mean too much
overhead.