This article explains how to set up simple cluster using couple of Tomcat instances and the load balancer. It’s divided into 3 sections:
- Environment setup (how to run load balancer and Tomcat instances on the separate machines)
- Creating simple JEE web application, that can demonstrate operation of the cluster underneath
- Benchmark results done during on of the trainings couple of weeks ago
Stage one: environment setup
For the demo-setup we will use 3 Linux machines. Perfect configuration would be to set up 3 descent PCs, connected with the wire (smallest ping). Of course the same config will work with virtual machines as well, wifi or tunnel connection.
I used Fedora/Red Hat machines for both load balancer and the devices hosting Tomcat. Of course setup on Windows/Mac/other distros is also possible, but it’s not an object of this tutorial. Here is a sketch of the environment setup:
LB – machine running load balancer, T1, T2 – machines running Tomcat #1 and #2 instances. . For the clarity, those node names and their IP addresses will be referred through all the article (if you have different ones, just adopt on your own).
Ok, let’s summarise the must-have:
- 3 Linux machines (can be virtual); I’m using 64 bit Fedora, kernel 3.9.x
- Sudo on this machines
- Machines must be able to “see” each other (necessary ports must be open in between them, preferably all PCs should be in the same subnetwork)
- You’ll need to compile some sources, so some additional tools are necessary (gcc, gcc-c++, automake, linux-headers, etc.)
- No fear using linux command line, basic java and Tomcat knowledge
If you don’t feel safe in Linux command line – don’t use your “main” pc for experiments – creating virtual machines is faster then reinstalling everything from scratch and less scary than loosing all your data :).
Setup: Load balancer
Load balancer (LB) is the machine which will split the traffic in between our two other nodes (T1,T2).
Installation steps (for LB) (all steps must be done successfully, especially the connector compilation!)
- Install Apache httpd server and APR libraries.
yum install httpd apr apr-util
- Download Tomcat sources (you’ll need it for mod_jk compilation)
- Compile mod_jk (it’s Apache Tomcat Connector). Connector can be found in: [TOMCAT_SOURCE]/bin/tomcat-native-1.1.27-src/jni/native/srclib/apr/. Cd to this directory and execute:
sudo ./configure --with-apr=/usr/bin/apr-1-config sudo make sudo make install
–wit-apr= switch is necessary on some system (it points to the APR config installation, you can locate it by typing ‘locate apr-1-config’ if it’s somewhere else)
- Once we have mod_jk in our system, we will configure HTTPD to enable load balancing. Let’s edit /etc/httpd/httpd.conf (alter the file by adding the following section – comments will explain you each line)
# Load just compiled mod_jk LoadModule jk_module modules/mod_jk.so # Tell mod_jk where the workers file is placed (keep this location) JkWorkersFile /etc/httpd/conf/workers.properties # Log and process files (must be writable by apache) JkShmFile /var/log/httpd/mod_jk.shm JkLogFile /var/log/httpd/mod_jk.log # Log level and format JkLogLevel info JkLogStampFormat "[%a %b %d %H:%M:%S %Y] " # Balance all queries to the 'bal1' JkMount /* bal1
Note it’s bal-one, not ball. bal1 is the name of the load balancer, it will be referenced in the next step.
- Create /etc/httpd/conf/workers.properties file, with the following content:
# Coma sperated list of load balancers worker.list = bal1 # first node (our T1) worker.tomcat1.type = ajp13 worker.tomcat1.host = 192.168.10.2 worker.tomcat1.port = 8009 worker.tomcat1.lbfactor = 10 # second node (our T2) worker.tomcat2.type = ajp13 worker.tomcat2.host = 192.168.10.3 worker.tomcat2.port = 8009 worker.tomcat2.lbfactor = 10 worker.bal1.type = lb # sticky session is enabled worker.bal1.sticky_session = 1 worker.bal1.balance_workers = tomcat1,tomcat2
- Now – start httpd.
sudo service httpd start
If you HTTPD starts fine (starting… [ok]) your setup should be correct. Otherwise check the logs – probably you made a typo somewhere.
Setup: Tomcats for load balancing (T1,T2)
This must be done on each of the machines running our servlet containers (in our case twice).
Installation steps (for T1,T2,…)
- Install Java, setup JDK_HOME
- Download and install Tomcat (using yum or download from the depending on your configuration)
- Now, alter server.xml file (it’s in [TOMCAT_HOME]/server). Alter the following sections:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"/>
<Connector port="8009" protocol="AJP/1.3" />
– so the redirect port is not necessary anymore.
<!-- <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1"> -->
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
, so uncomment this section, and assign unique jvmRoute for each of the machines
- Save the file, start the tomcat
- This step is optional, but we will use it in the next section to determine which machine is handling the request. Let’s setup the global variable which will be unique for the node. It’s name is nodeName, and should be unique per node. Add it to the server.xml, under GlobalNamingResources:
<GlobalNamingResources> <!-- ... ---> <Environment name="nodeName" type="java.lang.String" value="Tomcat 1"/> <!-- ... ---> </GlobalNamingResources>
- Repeat the steps for the T2
That’s all, start your Tomcats. Now, when you go to 192.168.10.1, you should see Tomcat welcome page or 404 when no ‘ROOT’ application is being deployed on the servers (of course 404 should come from Tomcat in that case). Load balancer works, so queries can be directed to T1 or T2.
Stage two: Web application
Now we will deploy the application, that will demonstrate how the traffic is being balanced. It must be deployed on each of the Tomcat instances within the cluster.
- Download web application. You can grab WAR file: [tomcat.war] or download the sources and compile on your own: [tomcat.zip] (it’s maven-spring simple servlet)
- Deploy the application on the Tomcats
- Query the load balancer. With the configuration described, it should be 192.168.10.1/clustering
This shows some basic machine-specific data. The red variable on the top is the before mentioned “nodeName” variable, configured in the 5th step of the previous section.
Refreshing the page will demonstrate, how the simple balancing works – queries will be directed to the T1 and T2 nodes so this page will change.
Stage three: Performance benchmark
Now let’s prove that despite balancing really works it also boosts the performance. During one of the trainings I conducted lately, I was able to test the solution using 4 nodes (4 laptops connected wirelessly together).
Tests have been made using the Jmeter tool. During the tests, web application had to serve 200 parallel threads making 5 requests each.
Results table (time in ms for the various tests)
|Node||Tomcat||HTPD||LB (single)||LB (4)||LB (4, fail)||LB (3)|
Node – name of the node (“Node 1” means that Node 1 was the LB, and nodes 2,3,4 were the Tomcats. “Node 2” – LB on the node 2 and Tomcats on 1,3,4 and so on.)
The tests made:
- Tomcat – single tomcat instance (no load balancing)
- HTTPD – Tomcat plus Apache HTTPD (communication over AJP protocol)
- LB (single) – HTTPD + Load Balancer, but single Tomcat instance behind (so traffic is always rooted to the same Tomcat)
- LB (4) – Load balancing with 4 nodes (1 LB and 3 tomcats)
- LB (3) – Load balancing with 3 nodes (1 LB and 2 tomcats)
- LB (4,FAIL) – Load balancing with 4 nodes (1 LB and 3 tomcats). One node is stopped to simulate failure.
We can clearly see the boost when load balancer is engaged (4 or 3 nodes on the graph above). When one of the nodes fails, performance drastically drops down. But after several retries and failures node is being excluded and performance comes back to normal.
Hope it helps. Good luck building your own cluster!