Linux Virtual Server with Direct Routing
and High Availability load balancing
for Windows 2000 Terminal Servers
<jeremy AT jheslop.com>
This work is licensed under a Creative Commons License.
Revision 0.3 2004-05-01 Revised by: jah
Added Creative Commons License to document. Share to all!
Revision 0.2 2004-01-30 Revised by: jah
Updated dates and misc editing.
Revision 0.1 2004-01-09 Revised by: jah
Initial document creation.
This document describes how to build, install, and configure Linux Virtual Server Direct Routing (LVS-DR) with High Availability (HA) to load balance a Windows 2000 Terminal Server (Win2kTS) cluster. Short names in the previous parenthesis will be used throughout the document.
This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY, either expressed or implied. While every effort has been taken to ensure the accuracy of the information documented herein, the author(s)/editor(s)/maintainer(s)/contributor(s) assumes NO RESPONSIBILITY for any errors, or for any damages, direct or consequential, as a result of the use of the information documented herein.
Table of Contents
What is LVS?
What is DR?
Why do I want LVS?
What is HA?
Why do I want HA?
Setting up LVS
Compiling the kernel with LVS
Build and Install Ipvsadm
Setting up HA
Compiling heartbeat with ldirectord
Configuring HA: ha.cf, haresources, authkeys, ldirectord.cf
Setting up the Windows 2000 Terminal Servers
This is an attempt to collect knowledge needed to setup two (or more) LVS-DR/HA nodes for a Win2kTS cluster. The entire process of getting, compiling, installing, and configuring this process will be covered. Pointers to LVS-DR, HA, and other projects will be included as needed. This version of the HowTo is for IPVS Netfilter 1.1.8 <http://www.linuxvirtualserver.org/software/ipvs.html> for the 2.6 linux kernel, ipvsadm 1.24, and heartbeat 1.0.4. HowTo will be Gentoo specific, but will have alternate distribution information as needed.
Chapter 2. What is an LVS?
A Linux Virtual Server (LVS) is a cluster of servers which appears to be one server to an outside client. In this situation it allows for many Windows 2000 Terminal Servers to be accessed by using one IP address and to balance the load across them. For more detailed information please consult the LVS HowTo <http://www.austintek.com/LVS/LVS-HOWTO/>
2.1 What is DR?
You can choose from three setups when using LVS: NAT, TUN, or DR. DR stands for Direct Routing. To find out more about the other two setups consult the LVS documentation <http://www.linuxvirtualserver.org/Documents.html> We will use DR because it allows higher bandwidth. Once the client makes a connection to the LVS server and gets redirected to a Win2KTS the client directly communicates with the terminal server and therefore does not use the LVS bandwidth. There are some limitations and advantages to all three setups so please make sure which one will work for you best.
2.2 Why do I want LVS?
In this setup LVS will balance out the load between a cluster of Windows 2000 Terminal Servers. Typically when a user has a list of IPs or computer they choose the last one they connected to or the first in the list. This leads to having an over abundance of clients on one or two servers with the other servers being relatively open. With LVS and it's weighted least-connection scheduling we will be able to connect clients to servers that have the least load thus increasing clients productivity because server response will be better.
3. What is HA?
4. Setting up LVS
Setting up LVS is a straight forward procedure. We will need to assume a few things before we can continue: A working Linux computer such as Gentoo Linux 1.4, all computers including the terminal servers on the same network, and the appropriate hardware on all the computers (NIC cards for each machine).
the kernel with LVS
The appropriate code we need to setup LVS is included in the Linux kernel 2.6.0 and above. There are patches for 2.2 and 2.4 kernels here: <http://www.linuxvirtualserver.org>
Do this in gentoo to get the latest 2.6.0 sources:
You will need to download a Linux 2.6.0 kernel from <http://www.kernel.org> uncompress it and then type in either:
make menuconfig make xconfig
kernel is out of the scope of this document. There are plenty of
resources available on the web that explain it in more detail.
Here are the kernel compile options needed for LVS:
Code maturity level options ---> [*] Prompt for development and/or incomplete code/drivers Device Drivers ---> Networking support ---> Networking options ---> [*] Network packet filtering (replaces ipchains) ---> [ ] Network packet filtering debugging IP: Netfilter Configuration ---> IP: Virtual Server Configuration ---> <M> IP virtual server support (EXPERIMENTAL) [*] IP virtual server debugging (12) IPVS connection table size (the Nth power of 2) --- IPVS transport protocol load balancing support [*] TCP load balancing support [*] UDP load balancing support [ ] ESP load balancing support [ ] AH load balancing support --- IPVS scheduler <M> round-robin scheduling <M> weighted round-robin scheduling <M> least-connection scheduling scheduling <M> weighted least-connection scheduling <M> locality-based least-connection scheduling <M> locality-based least-connection with replication scheduling <M> destination hashing scheduling <M> source hashing scheduling --- IPVS application helper <M> FTP protocol helper This will include all the LVS modules you need for this setup. Then compile your kernel:
make && make modules_install
If you have lilo or grub under Gentoo you can do a make install and get the kernel copied into /boot. Just remember to make sure you mount /boot first. Also make sure to reflect the changes of your new kernel's name and location under
/etc/lilo.conf or /boot/grub/grub.conf
restart your computer.
4.2 Build and Install Ipvsadm
This is relatively easy under gentoo, but because we are using the 2.6 kernel we will need to make sure we use the latest ebuild.
cd /usr/portage/sys-cluster/ipvsadm emerge ipvsadm-1.24.ebuild
For those without
gentoo head on over to the LVS main website and download the latest
version of ipvasdm then follow the LVS instructions on building it
for your system.
You should now have a fully functional LVS system without HA functionality.
5. Setting up
Setting up HA has a few steps. We need to compile and install the heartbeat and ldirectord programs, configure the ha.cf, haresource, and ldirectord.cf files and make sure we can get a heartbeat of some kind between the HA nodes. Remember each HA node needs to be mostly identical in setup. That is each needs LVS setup and HA setup as well to function.
5.1 Compiling heartbeat with ldirectord
Again Gentoo makes this easy with an ebuild, but make sure we have ldirectord built in:
USE='ldirectord' emerge heartbeat
The only thing I found that gentoo did not properly setup was the /var/lig/heartbeat directory. So lets make sure we create that:
For other Linux distributions please check the HA website, but it is only a matter of compiling and installing the heartbeat program.
Configuring HA: ha.cf, haresources, authkeys,
This was the hardest part of the setup for me even though I had done it one time in the past. First we will copy the example files into the /etc/ha.d/ folder.
cd /usr/share/doc/heartbeat-1.0.4/ cp ha.cf haresources /etc/ha.d
Here are the options I have in ha.cf with comments stripped out:
logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 120 ucast plip0 192.168.0.1 node lvs1 node lvs2
To get a more detailed explanation of each line please consult the HA documentation. I had to make sure I had the correct IPs for my lvs1 and lvs2 nodes. The ucast option will be the IP address for the other node in the parallel port network. I also made references to the plip IPs for readability. Here is what I put in /etc/hosts:
127.0.0.1 localhost 10.140.20.7 lvs1 10.140.20.8 lvs2 10.140.20.9 lvs # plip entries 192.168.0.1 localplip 192.168.0.2 remoteplip
The above /etc/hosts file was from the lvs1 node. You will need to switch the plip ips in the lvs2 node's /etc/hosts file. I used parallel cables to create the heartbeat between the two nodes and setup a private network for just these two computers. You will need to use another setup if you plan on using more than two. I could have put extra NIC cards and used crossover cables as well, but there where no extra NIC cards at the time. I would advise using extra NIC cards above this solution and putting the extra NIC cards on a private network using a Hub or switch.
I then setup the heartbeat program to do ucast which will send out udp pings to each node over the parallel link. I configured the parallel port network referring to this site: <http://www.tldp.org/HOWTO/Net-HOWTO/x1650.html> It was for kernel 2.0, but I was able to adapt it to my configuration. Here is what I had to setup to get the parallel port network working in kernel 2.6 and gentoo:
Added this to
options parport_pc io=0x378 irq=7
Make sure you put in the correct io address and irq found in your bios. Then issue modules_update command to regenerate the /etc/modules.conf file.
I then added these lines to /etc/conf.d/local.start:
/sbin/ifconfig plip0 localplip pointopoint remoteplip \ netmask 255.255.255.0 /sbin/route add remoteplip plip0
And these lines to /etc/conf.d/local.stop:
/sbin/route del remoteplip plip0 /sbin/ifconfig plip0 down
I'm sure there are other ways to do this, but this is how I did it. I
now have a working heartbeat private network. So the computers can
talk to each other and when one goes down the other one takes over
Here is the only line I have in the haresources file. Remember this file has to be the same on all HA nodes.
lvs1 10.140.20.9 ldirectord
This lets us know that lvs1 is the primary HA node and will answer to
this virtual IP address and will start ldirectord when taking over HA
For the authkeys file I just used the simple crc flag because I was using parallel port networking. If you are using a public connection for the heartbeat you will want to use a better authentication. Here is the contents of my authkeys file on both nodes:
auth 1 1 crc
Remember that the authkeys file must be 600 permission:
chmod 600 /etc/ha.d/authkeys
Here is the ldirectord.cf file:
We will log to standard syslog output,
set our virtual ip to the same as the one in haresources with the
port we want (terminal services), setup our real servers with direct
routing (gate) and turn on the weighted least-connections scheduler
(wlc). All of these files should be under /etc/ha.d.
6. Setting up the Windows 2000 Terminal Servers
We will need to make sure that we have the Windows 200 Terminal Servers setup properly so that we don't encounter any arp issues with the virtual IP address. We need to setup the Windows machines like so:
Add the Microsoft Loopback Adapter by going to the Control Panel and Add/Remove Hardware. Then add a new networking device. Go under Microsoft and select the Loopback Adapter device.
Go into the new network configuration by going into the Control Panel and then Network and Dialup Connections.
Go to the properties of the new Local Area Connection
Configure it with your virtual IP address (ie 10.140.20.9)
Un-check Client for Microsoft Networks
Un-check File and Printer Sharing for Microsoft Networks.
Advanced and set interface metric to 254
completes the setup. You should now have two or more computers that
act as an HA cluster to serve a LVS cluster of Windows 2000 Terminal
Servers to load balance the connections from your clients.
Official LVS Site <http://www.linuxvirtualserver.org>
Official HA Site <http://www.linux-ha.org>
LVS HowTo and Mini-HowTo <http://www.austintek.com/LVS/LVS-HOWTO/>
This documents location <http://footon.jheslop.com/howto/>
Official Linux Kernel <http://www.kernel.org>
PLIP Information <http://www.tldp.org/HOWTO/Net-HOWTO/x1650.html>