We are here today to setup a KVM host on CentOS 6 (or Linux variant). The host will have logical volumes backed by software RAID1 and a virtual pfSense router. With this setup you can securely and reliably host multiple applications on the same server.
I am writing these instructions for Linux users. I do not use Windows on my personal computer. You may need to run a Linux virtual machine or locate the appropriate Windows software.
The server should have a VT-x enabled proccessor, two hard drives (for software RAID1) and an uninterruptible power source. If the server is in a datacenter with reliable or redundant power you may be fine without a UPS.
Install CentOS 6.2 (or latest version) using the minimal disc (CentOS-6.2-x86_64-minimal.iso) and choose custom partitioning. If your provider pre-installed the operating system then skip the next few steps.
At the partition editor create two RAID partitions on each drive. One 500mb and a second to fill the remainder of space. This leads to four partitions.
Next, create a RAID device with mount point “/boot”, filesystem ext4, type RAID1 and select both 500mb partitions. Create another RAID device, select filesystem LVM, type RAID1 and select the two remaining partitions.
Next, create a new logical volume group and call it whatever you want. In the logical volume manager click add, set the mount point to “/”, filesystem ext4 and size of 20000MB. Click add again, set the filesystem to “swap” and make it the same size as your RAM. (I never seem to need the swap but I read it is better to have than not.)
Complete the install and reboot.
Login as root. Let’s disable some services we don’t use, “chkconfig fcoe off; chkconfig ip6tables off; chkconfig iptables off; chkconfig iscsi off; chkconfig iscsid off; chkconfig lldpad off; chkconfig netfs off; chkconfig nfslock off; chkconfig rpcbind off; chkconfig rpcgssd off; chkconfig rpcidmapd off”. If your provider pre-installed your operating system don’t disable iptables unless you have another firewall setup.
Time to setup initial networking.
eth0 is connected to our WAN. We will configure it as a bridge so our pfSense router can use it as an interface. If you have multiple network ports and/or LAN this may be different. “vi /etc/sysconfig/network-scripts/ifcfg-eth0”
DEVICE="eth0"
HWADDR="xx:xx:xx:xx:xx:xx"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
BRIDGE="br0"
Next we will set the IP on the WAN bridge. Later we will use it for SSH to setup pfSense. If you have one IP, don’t worry, we will comment it out before bringing up our router. “vi /etc/sysconfig/network-scripts/ifcfg-br0”
DEVICE=br0
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none
IPADDR=x.x.x.x
GATEWAY=x.x.x.x
NETMASK=x.x.x.x
DNS1=x.x.x.x
If you have a LAN port to use for SSH then configure that interface instead. If you would like to allow virtual machines to be on your LAN, in addition to WAN, then replicate the above two configurations for your LAN interface. Make sure to only specify the IP information (address, gateway, netmask, DNS) on one bridge.
We have to make atleast one bridge for virtual machines to connect to. I usually make two. One for the host server and virtualizations that do not run public services. The second for virtualizations that run public services. “vi /etc/sysconfig/network-scripts/ifcfg-br1”
DEVICE=br1
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none
“vi /etc/sysconfig/network-scripts/ifcfg-br2”
DEVICE=br2
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none
Change the default SSH port. Why use the default port when you don’t have to? “vi /etc/ssh/sshd_config”, uncomment and change “Port 22” to “Port xxx22” (easier to remember) or “Port xxxxx”. Restart the service, “/etc/init.d/sshd restart”.
If you are on a public network apply some iptables rules to protect yourself before or immediately after restarting the network service. Restart the network service. “/etc/init.d/network restart”
If you need a simple firewall to copy and paste, “vi /root/firewall”
#!/bin/sh
# iptables script generated 2011-01-23
# http://www.mista.nu/iptables
IPT="/sbin/iptables"
# Extra stuff
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/log_martians
echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 0 > /proc/sys/net/ipv4/ip_forward
# Flush old rules, old custom tables
$IPT --flush
$IPT --delete-chain
# Set default policies for all three default chains
$IPT -P INPUT DROP
$IPT -P FORWARD DROP
$IPT -P OUTPUT ACCEPT
# Enable free use of loopback interfaces
$IPT -A INPUT -i lo -j ACCEPT
$IPT -A OUTPUT -o lo -j ACCEPT
# All TCP sessions should begin with SYN
$IPT -A INPUT -p tcp ! --syn -m state --state NEW -s 0.0.0.0/0 -j DROP
# Accept inbound TCP packets
$IPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A INPUT -p tcp --dport XXX22 -m state --state NEW -s 0.0.0.0/0 -j ACCEPT
# Accept inbound ICMP messages
$IPT -A INPUT -p ICMP --icmp-type 8 -s 0.0.0.0/0 -j ACCEPT
# $IPT -A INPUT -p ICMP --icmp-type 11 -s 0.0.0.0/0 -j ACCEPT
Then, “chmod +x /root/firewall; /root/firewall”.
We are connected to the internet! Let’s update the packages, “yum update -y”.
Next, install KVM, libvirt, ntpd and wget, “yum install kvm libvirt ntp wget -y”.
Enable ntpd on boot, “chkconfig ntpd on”.
Disable SELinux, “vi /etc/selinux/config” and change “SELINUX=enforcing” to “SELINUX=disabled”.
Remove the default libvirt networking, “rm -f /etc/libvirt/qemu/networks/autostart/default.xml; rm -f /etc/libvirt/qemu/networks/default.xml”.
The next numbered items are optional and may improve performance.
1. “vi /etc/fstab”, change the entry for root(/) from “defaults” to “defaults,relatime”.
2. “vi /boot/grub/menu.lst”, at the end of the kernel line add “elevator=deadline”.
3. “vi /etc/sysctl.conf”, at the bottom add “vm.swappiness=0” and on another line “vm.zone_reclaim_mode=0”
The idea is to increase the availability of writing to the disk. Read about it on IBM’s website.
(Another good idea, if you are using LVMs, is to disable caching on your virtual hard drives. Keep this in mind for later.)
Add a line to “/etc/rc.local” (above the last line) for your firewall script. For example, “/root/firewall”. You may also want to add a line to turn off TCP segementation offloading. For example, “ethtool -K eth0 tso off”.
Type “reboot now”.
On your personal computer, create a key pair using “ssh-keygen” and copy the public key to your server using “ssh-copy-id”.
The easiest way to start making virtual machines is to install virt-manager on your personal computer, “sudo yum (or apt-get) install virt-manager”.
Open virt-manager and click “add connection” in the file menu. Tick “connect to remote host” and set hostname to “domain(or IP):port”. If you copied the key it will connect without a password.
Right click on the host entry and click on details. Click on the storage tab. Click on the + at the bottom left corner. Name it whatever, select type “logical: LVM Volume Group” and click next. Set the target path to “/dev/(whatever you named it during installation)”, leave the rest blank and click finish. You will see the name on the left with the percentage of free space.
Open a terminal and SSH into the server. Change directories to the default image location, “cd /var/lib/libvirt/images”. Visit the pfSense mirror page and download the iso, “wget mirror/pfSense-2.0.1-RELEASE-i386.iso.gz”. Decompress it, “gunzip pfSense-2.0.1-RELEASE-i386.iso.gz”.
In virt-manager, right click on the host and click new. Call it whatever, select local installation media and click forward. Browse for the iso you just downloaded (“/var/lib/libvirt/images/pfSense-2.0.1-RELEASE-i386.iso”) and click choose volume. Set the OS type to Unix, version to FreeBSD 8.x and click forward. Allocate atleast 512mb of memory, 1 or 2 CPUs and click forward. Select managed or existing storage and click browse. Select your volume group, click new volume, set the max capcity between 1000 and 4000mb, click ok, click choose volume and click forward. Click advanced options, select eth0/br0 and click finish.
While the pfSense iso is booting push “i” when prompted to start the installer. Install with default options and SMP kernel. Reboot. Before the virtual machine turns back on force the power off.
Go to the virtual machine details. Click add hardware, select network, select host device “eth1/br1”, select device model “e1000” and click finish. Repeat this for each bridge. Write down the MAC assigned to each bridge. Click on IDE cd-rom and remove it. Click boot options, tick “start virtual machine on host boot up” and click apply. Start the virtual machine.
When asked about VLANs choose setup later. Assign the interfaces their appropriate designation.
Go to your terminal that is connected to the host server. In the default image directory download a live CD. I like to use Ubuntu 10.04.4. Here is the mirror list.
Go to virt-manager, right click on the host and click new. Setup a virtual machine to boot from the live CD. The amount of storage doesn’t matter. Under advanced options, select the bridge you assigned to the LAN in pfSense.
On the live desktop open Firefox and go to “http://192.168.1.1”. The default login is admin and password is pfsense. This can be changed under system, user manager.
Go to system, advanced and change the default port. (optional)
Go to system, general settings, set your hostname and domain. You can use your own DNS servers or set them to 8.8.4.4, 8.8.8.8, 208.67.222.222 and 208.67.220.220. Set your time zone and hit save.
Go to interfaces, set the details for each interface and save but do not hit apply. If you have one public IP it is very important that you do not click apply!
Go to services and setup your DHCP servers as desired.
Go to firewall, NAT and click on the + icon. Set the destination port range to whatever you changed it to earlier in system/advanced, redirect target IP to the LAN IP of your router, redirect target port to the same value as destination port and click save.
Click the plus icon, again. Set the destination port to the SSH port you setup earlier, redirect target IP to the LAN IP of your host server (to be set in the next step), redirect target port to the same as destination port and click save.
Go to diagnostics, halt system and click yes.
Open the terminal which is connected to your server and open the br0 config file. Comment or delete the address, gateway, netmask, DNS details and save. Open the br1 (assuming you used br1 for LAN) config file and use the same format as br0 to set a private IP. For example, the router is x.x.x.1 so I made the host server x.x.x.2. Don’t forget to set the gateway, netmask and DNS!
Reboot the host server. When it comes back up the router will autostart. You can visit the web interface and SSH using the WAN IP!
A few more things before you are on your way…
Install grub on the second drive’s master boot record. When the first drive fails use the second to boot. Go to this link and scroll down to “Install Grub on new hard drive MBR”. You can follow exactly what the person wrote there. Test it by rebooting and booting from the second drive!
To monitor your virtual machines’s resource consumption install Host sFlow. It will add and remove virtual machines without manual intervention. Combine with Ganglia for web based reporting. I have written a guide here.
In pfSense, setup firewall rules on the OPT* interfaces to allow internet access. Look at the LAN rules to get started. Also, add rules to block traffic from OPT* interfaces to your LAN, etc.
When rebooting the host set a 2 minute timer (instead of now) and halt the router. I have had problems with the router when resuming from a suspended state.
I hope this helped you setup something cool!! Please leave a comment if you have a suggestion or question.