|Summary:||Making mod_jk replication aware (Clustering Support)|
|Product:||Tomcat Connectors||Reporter:||Rainer Jung <rainer.jung>|
|Component:||Common||Assignee:||Tomcat Developers Mailing List <dev>|
Add hierarchical routing to load balancer
See former patch - now produced with diff -u
Description Rainer Jung 2004-11-19 13:53:50 UTC
Making mod_jk replication aware (Clustering Support) At the moment mod_jk makes it's load-balancing decisions to workers according to the following parameters: - session-stickyness (choose the "only" right worker) - locality (one worker is preferred) - weight-based What is missing, is any information about secondary cluster nodes. Use Case 1: Horizontal scalability and performance of tomcat cluster Tomcat cluster does only allow session replication to all nodes in the cluster. Once you work with more than 3-4 nodes there is too much overhead and risk in replicating sessions to all nodes. We split all nodes into clustered groups. I introduced a new worker attribute "domain" to let get_most_suitable_worker in mod_jk know, to which other nodes a session gets replicated (all workers with the same value in the domain attribute). So a load balancing worker knows, on which nodes the session is alive. If a node fails or is being taken down administratively, mod_jk chooses another node that has a replica of the session. Use Case 2: Prevent thread count explosion Once mod_jk connects an apache process to a tomcat instance, the tomcat jk connector will need one thread for this connection as long as the apache process is alive, independent from the number of requests apache actually sends to tomcat. Now assume you have multiple Apaches and Tomcats. The Tomcats are clustered and mod_jk uses sticky sessions. Now you are going to shut down (maintenance) one tomcat. All Apache will start connections to all tomcats. You end up with all tomcats getting connections from all apache processes, so the number of threads needed inside the tomcats will explode. If you group the tomcats to domain as in use case 1, the connections normally will stay inside the domain and you will need much less threads. Implementation The implementation uses and additional worker attribute domain, and if a session has a jvmRoute, is sticky and the correct worker is in error state another worker with the same domain attribute is chosen. I have an implementation based on mod_jk 1.2.6 running successfully in production, but I would have to adapt to the 1.2.7 code changes. The implementation would only concern common/jk_util.c for the additional configuration worker attribute "domain" and common/jk_lb_worker.c for using the attribute - with stickyness in case the primary worker is down to first seek for some other worker with the same domain attribute. - without stickyness to decide to which workers an existing session is allowed to be balanced (all workers with the same domain as the worker given in the session id) I could provide the code, if you are interested. Here is a more concrete example: Enterprise application with redundant internet connections A and B. A consists of two Apache instances A.a1 und A.a2, B of B.a1 and B.a2. Behind are 4 Tomcat A.t1, A.t2, B.t1, B.t2. A.t1 and A.t2 are clustered, B.t1 and B.t2 are clustered. mod_jk uses load balancing with sticky sessions. All Apaches can connect to any Tomcat, but A.t1 is local for A.a1, A.t2 for A.a2, B.t1 for B.a1 and B.t2 for B.a2: A.a1 A.a2 B.a1 B.a2 || X || X || X || A.t1---A.t2 B.t1---B.t2 A.t1 and A.t2 are put into the same "domain" "A" in workers.properties, B.t1 and B.t2 are put into the same "domain" "B" in workers.properties. Now if you shutdown e.g. tomcat B.t1 for service/update (or if it breaks) All apaches will know from the domain configuration, that sticky requests for B.t1 have to go to B.t2. This is important, since only on that tomcat the replicated sessions from B.t1 will exist. Without domains you have to put all the Tomcats in one cluster. But then all sessions are replicated to all tomcats. We have a production side using 3x3=9 tomcats and a cluster with 9 nodes would mean too much overhead.
Comment 1 Peter Rossbach 2004-11-21 09:53:35 UTC
Very fine idea, please send the patches! regards peter
Comment 2 Rainer Jung 2004-12-05 22:27:28 UTC
Created attachment 13650 [details] Add hierarchical routing to load balancer The attached patch concerns jk_lb_worker.c and jk_util.c. It introduces the following features: - additional property "domain" for workers in workers.properties - Make routing decision hierarchical: a) sticky worker if not in error b) else worker from same domain c) else local worker d) else local domain (worker inside domain of one of the local workers) e) any other worker - erroneous non-sticky workers get chosen after recovery time only if their lb_value is maximum. Avoids continously decrementing lb_value for erroneous workers. Patch was transformed from a patch for version 1.2.6. I did not yet have the time to compile and test, but I know that at least Peter waits for it, so I submit it untested. It's predecessor for 1.2.6 has been tested and in production for several weeks now.
Comment 3 Mladen Turk 2004-12-06 09:35:58 UTC
Rainer, Could you please run unified diff ('diff -u')? I can not use the patch otherwise. Thanks.
Comment 4 Rainer Jung 2004-12-06 20:41:16 UTC
Created attachment 13658 [details] See former patch - now produced with diff -u Now in diff -u format. Had to update Solaris diff to gnu diff.
Comment 5 Mladen Turk 2004-12-07 13:26:02 UTC
Commited with small cosmetic modifications. Thanks Rainer! It would be great if you could write some docs or configuration examples
Comment 6 Rainer Jung 2004-12-07 18:19:15 UTC
Thank's a lot for accepting the patch. I hope it will not make too much trouble and I'm sure it will be helpful for really useful bigger cluster designs. Documentation and examples config: I will provide in the next 5 days. Code without documentation is not very useful.