This work is licensed under a Creative Commons License.
Linux Virtual Server with Direct Routing
and High Availability load balancing
for Windows 2000 Terminal Servers
Jeremy Heslop
This
work is licensed under a Creative
Commons License.
Revision History:
Revision 0.3 2004-05-01 Revised by: jah
Added Creative Commons License to document. Share to all!
Revision 0.2 2004-01-30 Revised by: jah
Updated dates and misc editing.
Revision 0.1 2004-01-09 Revised by: jah
Initial document creation.
This document describes how to build, install, and configure Linux Virtual Server Direct Routing (LVS-DR) with High Availability (HA) to load balance a Windows 2000 Terminal Server (Win2kTS) cluster. Short names in the previous parenthesis will be used throughout the document.
This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY, either expressed or implied. While every effort has been taken to ensure the accuracy of the information documented herein, the author(s)/editor(s)/maintainer(s)/contributor(s) assumes NO RESPONSIBILITY for any errors, or for any damages, direct or consequential, as a result of the use of the information documented herein.
Table of Contents
Introduction
Latest Version
Disclaimer
Authors
What is LVS?
What is DR?
Why do I want LVS?
What is HA?
Why do I want HA?
Setting up LVS
Compiling the kernel with LVS
Build and Install Ipvsadm
Setting up HA
Compiling heartbeat with ldirectord
Configuring HA: ha.cf, haresources, authkeys, ldirectord.cf
Setting up the Windows 2000 Terminal Servers
Links
1. Introduction
This is
an attempt to collect knowledge needed to setup two (or more)
LVS-DR/HA nodes for a Win2kTS cluster. The entire process of getting,
compiling, installing, and configuring this process will be covered.
Pointers to LVS-DR, HA, and other projects will be included as
needed. This version of the HowTo is for IPVS Netfilter 1.1.8
<http://www.linuxvirtualserver.org/software/ipvs.html>
for the 2.6 linux kernel, ipvsadm 1.24, and heartbeat 1.0.4. HowTo
will be Gentoo specific, but will have alternate distribution
information as needed.
All previous versions of IPVS, ipvadm, and heartbeat are either obsolete or not guaranteed. This document makes no promises as to the success of getting any of the previous versions working.
The main purpose of this setup is to connect Microsoft Windows clients using Remote Desktop Connection tool to Windows 2000 Terminal Servers. You can alter use this information for other purposes as well.
1.1. Latest Version
The latest versions of this document will be kept in this location:
<http://www.opustechsystems.com/howto/>
1.2. Disclaimer
This document is distributed in
the hope that it will be useful, but WITHOUT ANY WARRANTY, either
expressed or implied. While every effort has been taken to ensure the
accuracy of the information documented herein, the
author(s)/editor(s)/maintainer(s)/contributor(s) assumes NO
RESPONSIBILITY for any errors, or for any damages, direct or
consequential, as a result of the use of the information documented
herein.
1.3. Authors
List of everyone who has put words into
this file.
[mailto:jeremy.heslop@opustechsystems.com_NOSPAM]
Jeremy Heslop
Please notify the HowTo maintainer if you
believe you should be listed above.
Chapter 2. What is an LVS?
A
Linux Virtual Server (LVS) is a cluster of servers which appears to
be one server to an outside client. In this situation it allows for
many Windows 2000 Terminal Servers to be accessed by using one IP
address and to balance the load across them. For more detailed
information please consult the LVS HowTo
<http://www.austintek.com/LVS/LVS-HOWTO/>
2.1 What is DR?
You can choose from three setups when using LVS: NAT, TUN, or DR. DR stands for Direct Routing. To find out more about the other two setups consult the LVS documentation <http://www.linuxvirtualserver.org/Documents.html> We will use DR because it allows higher bandwidth. Once the client makes a connection to the LVS server and gets redirected to a Win2KTS the client directly communicates with the terminal server and therefore does not use the LVS bandwidth. There are some limitations and advantages to all three setups so please make sure which one will work for you best.
2.2 Why do I want LVS?
In
this setup LVS will balance out the load between a cluster of Windows
2000 Terminal Servers. Typically when a user has a list of IPs or
computer they choose the last one they connected to or the first in
the list. This leads to having an over abundance of clients on one or
two servers with the other servers being relatively open. With LVS
and it's weighted least-connection scheduling we will be able
to connect clients to servers that have the least load thus
increasing clients productivity because server response will be
better.
3. What is HA?
High Availability Cluster (HA)
is a cluster that allows a host (or hosts) to become Highly
Available, that means if one node goes down (or a service on that
node goes down) another node can pick up the service or node and take
over from the failed machine. <http://linux-ha.org>
3.1
Why do I want HA?
The
HA cluster will allow your LVS cluster to continue to function even
if the main HA cluster node goes down. This means a high amount of
redundancy is introduced into the equation allowing for more uptime
if a hardware or software component fails.
4. Setting up LVS
Setting up LVS is a straight forward procedure. We will need to assume a few things before we can continue: A working Linux computer such as Gentoo Linux 1.4, all computers including the terminal servers on the same network, and the appropriate hardware on all the computers (NIC cards for each machine).
4.2 Compiling
the kernel with LVS
The appropriate code we need to setup
LVS is included in the Linux kernel 2.6.0 and above. There are
patches for 2.2 and 2.4 kernels here:
<http://www.linuxvirtualserver.org>
Do this in gentoo to get the latest 2.6.0 sources:
emerge sys-kernel/gentoo-dev-sources
You will need to download a Linux 2.6.0 kernel from <http://www.kernel.org> uncompress it and then type in either:
make menuconfig make xconfig
or
make gconfig
Configuring the
kernel is out of the scope of this document. There are plenty of
resources available on the web that explain it in more detail.
Here are the kernel compile options needed for LVS:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Device Drivers --->
Networking support --->
Networking options --->
[*] Network packet filtering (replaces ipchains) --->
[ ] Network packet filtering debugging
IP: Netfilter Configuration --->
IP: Virtual Server Configuration --->
<M> IP virtual server support (EXPERIMENTAL)
[*] IP virtual server debugging
(12) IPVS connection table size (the Nth power of 2)
--- IPVS transport protocol load balancing support
[*] TCP load balancing support
[*] UDP load balancing support
[ ] ESP load balancing support
[ ] AH load balancing support
--- IPVS scheduler
<M> round-robin scheduling
<M> weighted round-robin scheduling
<M> least-connection scheduling scheduling
<M> weighted least-connection scheduling
<M> locality-based least-connection scheduling
<M> locality-based least-connection with replication scheduling
<M> destination hashing scheduling
<M> source hashing scheduling
--- IPVS application helper
<M> FTP protocol helper
This will include all the LVS modules you need for this setup. Then compile your kernel:
make && make modules_install
If you have lilo or grub under Gentoo you can do a make install and get the kernel copied into /boot. Just remember to make sure you mount /boot first. Also make sure to reflect the changes of your new kernel's name and location under
/etc/lilo.conf or /boot/grub/grub.conf
Then
restart your computer.
4.2 Build and Install
Ipvsadm
This is relatively easy under gentoo, but because we are using the 2.6 kernel we will need to make sure we use the latest ebuild.
cd /usr/portage/sys-cluster/ipvsadm emerge ipvsadm-1.24.ebuild
For those without
gentoo head on over to the LVS main website and download the latest
version of ipvasdm then follow the LVS instructions on building it
for your system.
You should now have a fully functional LVS
system without HA functionality.
5. Setting up
HA
Setting up HA has a few steps. We need to compile and
install the heartbeat and ldirectord programs, configure the ha.cf,
haresource, and ldirectord.cf files and make sure we can get a
heartbeat of some kind between the HA nodes. Remember each HA node
needs to be mostly identical in setup. That is each needs LVS setup
and HA setup as well to function.
5.1
Compiling heartbeat with ldirectord
Again Gentoo makes
this easy with an ebuild, but make sure we have ldirectord built in:
USE='ldirectord' emerge heartbeat
The only thing I found that gentoo did not properly setup was the /var/lig/heartbeat directory. So lets make sure we create that:
mkdir /var/lib/heartbeat
For other Linux distributions please check the HA website, but it is only a matter of compiling and installing the heartbeat program.
5.2
Configuring HA: ha.cf, haresources, authkeys,
ldirectord.cf
This was
the hardest part of the setup for me even though I had done it one
time in the past. First we will copy the example files into the
/etc/ha.d/ folder.
cd /usr/share/doc/heartbeat-1.0.4/ cp ha.cf haresources /etc/ha.d
Here are the options I have in ha.cf with comments stripped out:
logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 120 ucast plip0 192.168.0.1 node lvs1 node lvs2
To get a more detailed explanation of each line please consult the HA documentation. I had to make sure I had the correct IPs for my lvs1 and lvs2 nodes. The ucast option will be the IP address for the other node in the parallel port network. I also made references to the plip IPs for readability. Here is what I put in /etc/hosts:
127.0.0.1 localhost 10.140.20.7 lvs1 10.140.20.8 lvs2 10.140.20.9 lvs # plip entries 192.168.0.1 localplip 192.168.0.2 remoteplip
The above /etc/hosts file was from the lvs1 node. You will need
to switch the plip ips in the lvs2 node's /etc/hosts file. I used
parallel cables to create the heartbeat between the two nodes and
setup a private network for just these two computers. You will need
to use another setup if you plan on using more than two. I could have
put extra NIC cards and used crossover cables as well, but there
where no extra NIC cards at the time. I would advise using extra NIC
cards above this solution and putting the extra NIC cards on a
private network using a Hub or switch.
I then setup the
heartbeat program to do ucast which will send out udp pings to each
node over the parallel link. I configured the parallel port network
referring to this site:
<http://www.tldp.org/HOWTO/Net-HOWTO/x1650.html>
It was for kernel 2.0, but I was able to adapt it to my
configuration. Here is what I had to setup to get the parallel port
network working in kernel 2.6 and gentoo:
Added this to
/etc/modules.d/i386:
options
parport_pc io=0x378 irq=7
Make sure you put in
the correct io address and irq found in your bios. Then issue
modules_update command to regenerate the /etc/modules.conf file.
I then added these lines to /etc/conf.d/local.start:
/sbin/ifconfig plip0 localplip pointopoint remoteplip \ netmask 255.255.255.0 /sbin/route add remoteplip plip0
And these lines to /etc/conf.d/local.stop:
/sbin/route del remoteplip plip0 /sbin/ifconfig plip0 down
I'm sure there are other ways to do this, but this is how I did it. I
now have a working heartbeat private network. So the computers can
talk to each other and when one goes down the other one takes over
LVS duties.
Here is the only line I have in the haresources file. Remember this file has to be the same on all HA nodes.
lvs1 10.140.20.9 ldirectord
This lets us know that lvs1 is the primary HA node and will answer to
this virtual IP address and will start ldirectord when taking over HA
duties.
For the authkeys file I just used the simple crc flag because I was using parallel port networking. If you are using a public connection for the heartbeat you will want to use a better authentication. Here is the contents of my authkeys file on both nodes:
auth 1 1 crc
Remember that the authkeys file must be 600 permission:
chmod 600 /etc/ha.d/authkeys
Here is the ldirectord.cf file:
logfile="local0"
virtual=10.140.20.9:3389
real=10.140.20.11 gate
real=10.140.20.13 gate
real=10.140.20.15 gate
real=10.140.20.17 gate
real=10.140.20.19 gate
real=10.140.20.21 gate
scheduler=wlc
We will log to standard syslog output,
set our virtual ip to the same as the one in haresources with the
port we want (terminal services), setup our real servers with direct
routing (gate) and turn on the weighted least-connections scheduler
(wlc). All of these files should be under /etc/ha.d.
6. Setting up the Windows 2000 Terminal Servers
We will need
to make sure that we have the Windows 200 Terminal Servers setup
properly so that we don't encounter any arp issues with the virtual
IP address. We need to setup the Windows machines like so:
Add the Microsoft Loopback Adapter by going to the Control Panel and Add/Remove Hardware. Then add a new networking device. Go under Microsoft and select the Loopback Adapter device.
Go into the new network configuration by going into the Control Panel and then Network and Dialup Connections.
Go to the properties of the new Local Area Connection
Configure it with your virtual IP address (ie 10.140.20.9)
Un-check Client for Microsoft Networks
Un-check File and Printer Sharing for Microsoft Networks.
Go to
Advanced and set interface metric to 254
This
completes the setup. You should now have two or more computers that
act as an HA cluster to serve a LVS cluster of Windows 2000 Terminal
Servers to load balance the connections from your clients.
7.
Links
Official LVS Site <http://www.linuxvirtualserver.org>
Official HA Site <http://www.linux-ha.org>
LVS HowTo and Mini-HowTo <http://www.austintek.com/LVS/LVS-HOWTO/>
This documents location <http://footon.jheslop.com/howto/>
Official Linux Kernel <http://www.kernel.org>
PLIP Information <http://www.tldp.org/HOWTO/Net-HOWTO/x1650.html>