Setup a KVM VPS Host, LVM on Software RAID1 and a Virtual pfSense Router

We are here today to setup a KVM host on CentOS 6 (or Linux variant). The host will have logical volumes backed by software RAID1 and a virtual pfSense router. With this setup you can securely and reliably host multiple applications on the same server.

I am writing these instructions for Linux users. I do not use Windows on my personal computer. You may need to run a Linux virtual machine or locate the appropriate Windows software.

The server should have a VT-x enabled proccessor, two hard drives (for software RAID1) and an uninterruptible power source. If the server is in a datacenter with reliable or redundant power you may be fine without a UPS.

Install CentOS 6.2 (or latest version) using the minimal disc (CentOS-6.2-x86_64-minimal.iso) and choose custom partitioning. If your provider pre-installed the operating system then skip the next few steps.

At the partition editor create two RAID partitions on each drive. One 500mb and a second to fill the remainder of space. This leads to four partitions.

Next, create a RAID device with mount point “/boot”, filesystem ext4, type RAID1 and select both 500mb partitions. Create another RAID device, select filesystem LVM, type RAID1 and select the two remaining partitions.

Next, create a new logical volume group and call it whatever you want. In the logical volume manager click add, set the mount point to “/”, filesystem ext4 and size of 20000MB. Click add again, set the filesystem to “swap” and make it the same size as your RAM. (I never seem to need the swap but I read it is better to have than not.)

Complete the install and reboot.

Login as root. Let’s disable some services we don’t use, “chkconfig fcoe off; chkconfig ip6tables off; chkconfig iptables off; chkconfig iscsi off; chkconfig iscsid off; chkconfig lldpad off; chkconfig netfs off; chkconfig nfslock off; chkconfig rpcbind off; chkconfig rpcgssd off; chkconfig rpcidmapd off”. If your provider pre-installed your operating system don’t disable iptables unless you have another firewall setup.

Time to setup initial networking.

eth0 is connected to our WAN. We will configure it as a bridge so our pfSense router can use it as an interface. If you have multiple network ports and/or LAN this may be different. “vi /etc/sysconfig/network-scripts/ifcfg-eth0”

DEVICE="eth0"
HWADDR="xx:xx:xx:xx:xx:xx"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
BRIDGE="br0"

Next we will set the IP on the WAN bridge. Later we will use it for SSH to setup pfSense. If you have one IP, don’t worry, we will comment it out before bringing up our router. “vi /etc/sysconfig/network-scripts/ifcfg-br0”

DEVICE=br0
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none
IPADDR=x.x.x.x
GATEWAY=x.x.x.x
NETMASK=x.x.x.x
DNS1=x.x.x.x

If you have a LAN port to use for SSH then configure that interface instead. If you would like to allow virtual machines to be on your LAN, in addition to WAN, then replicate the above two configurations for your LAN interface. Make sure to only specify the IP information (address, gateway, netmask, DNS) on one bridge.

We have to make atleast one bridge for virtual machines to connect to. I usually make two. One for the host server and virtualizations that do not run public services. The second for virtualizations that run public services. “vi /etc/sysconfig/network-scripts/ifcfg-br1”

DEVICE=br1
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none

“vi /etc/sysconfig/network-scripts/ifcfg-br2”

DEVICE=br2
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Bridge
STP=on
DELAY=0
BOOTPROTO=none

Change the default SSH port. Why use the default port when you don’t have to? “vi /etc/ssh/sshd_config”, uncomment and change “Port 22” to “Port xxx22” (easier to remember) or “Port xxxxx”. Restart the service, “/etc/init.d/sshd restart”.

If you are on a public network apply some iptables rules to protect yourself before or immediately after restarting the network service. Restart the network service. “/etc/init.d/network restart”

If you need a simple firewall to copy and paste, “vi /root/firewall”

#!/bin/sh

# iptables script generated 2011-01-23
# http://www.mista.nu/iptables

IPT="/sbin/iptables"

# Extra stuff
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/log_martians
echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 0 > /proc/sys/net/ipv4/ip_forward

# Flush old rules, old custom tables
$IPT --flush
$IPT --delete-chain

# Set default policies for all three default chains
$IPT -P INPUT DROP
$IPT -P FORWARD DROP
$IPT -P OUTPUT ACCEPT

# Enable free use of loopback interfaces
$IPT -A INPUT -i lo -j ACCEPT
$IPT -A OUTPUT -o lo -j ACCEPT

# All TCP sessions should begin with SYN
$IPT -A INPUT -p tcp ! --syn -m state --state NEW -s 0.0.0.0/0 -j DROP

# Accept inbound TCP packets
$IPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A INPUT -p tcp --dport XXX22 -m state --state NEW -s 0.0.0.0/0 -j ACCEPT

# Accept inbound ICMP messages
$IPT -A INPUT -p ICMP --icmp-type 8 -s 0.0.0.0/0 -j ACCEPT
# $IPT -A INPUT -p ICMP --icmp-type 11 -s 0.0.0.0/0 -j ACCEPT

Then, “chmod +x /root/firewall; /root/firewall”.

We are connected to the internet! Let’s update the packages, “yum update -y”.

Next, install KVM, libvirt, ntpd and wget, “yum install kvm libvirt ntp wget -y”.

Enable ntpd on boot, “chkconfig ntpd on”.

Disable SELinux, “vi /etc/selinux/config” and change “SELINUX=enforcing” to “SELINUX=disabled”.

Remove the default libvirt networking, “rm -f /etc/libvirt/qemu/networks/autostart/default.xml; rm -f /etc/libvirt/qemu/networks/default.xml”.

The next numbered items are optional and may improve performance.
1. “vi /etc/fstab”, change the entry for root(/) from “defaults” to “defaults,relatime”.
2. “vi /boot/grub/menu.lst”, at the end of the kernel line add “elevator=deadline”.
3. “vi /etc/sysctl.conf”, at the bottom add “vm.swappiness=0” and on another line “vm.zone_reclaim_mode=0”
The idea is to increase the availability of writing to the disk. Read about it on IBM’s website.
(Another good idea, if you are using LVMs, is to disable caching on your virtual hard drives. Keep this in mind for later.)

Add a line to “/etc/rc.local” (above the last line) for your firewall script. For example, “/root/firewall”. You may also want to add a line to turn off TCP segementation offloading. For example, “ethtool -K eth0 tso off”.

Type “reboot now”.

On your personal computer, create a key pair using “ssh-keygen” and copy the public key to your server using “ssh-copy-id”.

The easiest way to start making virtual machines is to install virt-manager on your personal computer, “sudo yum (or apt-get) install virt-manager”.

Open virt-manager and click “add connection” in the file menu. Tick “connect to remote host” and set hostname to “domain(or IP):port”. If you copied the key it will connect without a password.

Right click on the host entry and click on details. Click on the storage tab. Click on the + at the bottom left corner. Name it whatever, select type “logical: LVM Volume Group” and click next. Set the target path to “/dev/(whatever you named it during installation)”, leave the rest blank and click finish. You will see the name on the left with the percentage of free space.

Open a terminal and SSH into the server. Change directories to the default image location, “cd /var/lib/libvirt/images”. Visit the pfSense mirror page and download the iso, “wget mirror/pfSense-2.0.1-RELEASE-i386.iso.gz”. Decompress it, “gunzip pfSense-2.0.1-RELEASE-i386.iso.gz”.

In virt-manager, right click on the host and click new. Call it whatever, select local installation media and click forward. Browse for the iso you just downloaded (“/var/lib/libvirt/images/pfSense-2.0.1-RELEASE-i386.iso”) and click choose volume. Set the OS type to Unix, version to FreeBSD 8.x and click forward. Allocate atleast 512mb of memory, 1 or 2 CPUs and click forward. Select managed or existing storage and click browse. Select your volume group, click new volume, set the max capcity between 1000 and 4000mb, click ok, click choose volume and click forward. Click advanced options, select eth0/br0 and click finish.

While the pfSense iso is booting push “i” when prompted to start the installer. Install with default options and SMP kernel. Reboot. Before the virtual machine turns back on force the power off.

Go to the virtual machine details. Click add hardware, select network, select host device “eth1/br1”, select device model “e1000” and click finish. Repeat this for each bridge. Write down the MAC assigned to each bridge. Click on IDE cd-rom and remove it. Click boot options, tick “start virtual machine on host boot up” and click apply. Start the virtual machine.

When asked about VLANs choose setup later. Assign the interfaces their appropriate designation.

Go to your terminal that is connected to the host server. In the default image directory download a live CD. I like to use Ubuntu 10.04.4. Here is the mirror list.

Go to virt-manager, right click on the host and click new. Setup a virtual machine to boot from the live CD. The amount of storage doesn’t matter. Under advanced options, select the bridge you assigned to the LAN in pfSense.

On the live desktop open Firefox and go to “http://192.168.1.1”. The default login is admin and password is pfsense. This can be changed under system, user manager.

Go to system, advanced and change the default port. (optional)

Go to system, general settings, set your hostname and domain. You can use your own DNS servers or set them to 8.8.4.4, 8.8.8.8, 208.67.222.222 and 208.67.220.220. Set your time zone and hit save.

Go to interfaces, set the details for each interface and save but do not hit apply. If you have one public IP it is very important that you do not click apply!

Go to services and setup your DHCP servers as desired.

Go to firewall, NAT and click on the + icon. Set the destination port range to whatever you changed it to earlier in system/advanced, redirect target IP to the LAN IP of your router, redirect target port to the same value as destination port and click save.

Click the plus icon, again. Set the destination port to the SSH port you setup earlier, redirect target IP to the LAN IP of your host server (to be set in the next step), redirect target port to the same as destination port and click save.

Go to diagnostics, halt system and click yes.

Open the terminal which is connected to your server and open the br0 config file. Comment or delete the address, gateway, netmask, DNS details and save. Open the br1 (assuming you used br1 for LAN) config file and use the same format as br0 to set a private IP. For example, the router is x.x.x.1 so I made the host server x.x.x.2. Don’t forget to set the gateway, netmask and DNS!

Reboot the host server. When it comes back up the router will autostart. You can visit the web interface and SSH using the WAN IP!

A few more things before you are on your way…

Install grub on the second drive’s master boot record. When the first drive fails use the second to boot. Go to this link and scroll down to “Install Grub on new hard drive MBR”. You can follow exactly what the person wrote there. Test it by rebooting and booting from the second drive!

To monitor your virtual machines’s resource consumption install Host sFlow. It will add and remove virtual machines without manual intervention. Combine with Ganglia for web based reporting. I have written a guide here.

In pfSense, setup firewall rules on the OPT* interfaces to allow internet access. Look at the LAN rules to get started. Also, add rules to block traffic from OPT* interfaces to your LAN, etc.

When rebooting the host set a 2 minute timer (instead of now) and halt the router. I have had problems with the router when resuming from a suspended state.

I hope this helped you setup something cool!! Please leave a comment if you have a suggestion or question.

9 Replies to “Setup a KVM VPS Host, LVM on Software RAID1 and a Virtual pfSense Router”

  1. Hey thanks for the tut. Can a windows vps be set up on this rig? anything to be worried when doing so? Would the windows vps have to go in as a guest inside the host and would the KVM be slow?

    1. Yes, windows can run on KVM just fine. Depending on the version and virtual hardware added to the VM, you may need to install the appropriate virtio drivers, etc.

  2. Hi HUSSEIN,

    I have been googling around for tutorial to set up host machine under linux centos 7 (or 6.5), the host will have some VMs Guest, and will be used for production environment. So far this tutorial of yours is the most complete i found. Since i am a beginner, i am still not sure enough in digesting of every step in this, but i will do it.

    I have question, is it applicable for Centos 7 as the OS for the Host?, which one will be better using centOS 6.x or CentOS 7?

    I know that this tutorial is 2 yeras old or more, do you have update it? for example when using CentOS 7

    Regards,
    Bun

    1. Thank you, Bun.

      Package and config file names don’t usually change between versions, so I think you should be okay with CentOS 7. I personally haven’t tried this with CentOS 7 and I don’t use CentOS as much as I used to.

      This setup was good in 2012, 4 years ago, but there is probably better and more convenient ways to manage your host machine. I have been using Promox (https://www.proxmox.com/en/proxmox-ve) for 2-3 years now and am very happy with it. However, it is Debian based.

      If you are going to use CentOS, consider checking out the official KVM page which lists a bunch of management tools (http://www.linux-kvm.org/page/Management_Tools).

      To answer your question, I don’t see why this wouldn’t work with CentOS 7. Feel free to update us with your findings! Thanks!

      EDIT: After reading the article again, I set swappiness to one instead of zero (vm.swappiness=1) now.

    2. Hi HUSSEIN,

      I have installed kvm host on centos 7 minimal install, i also can connect to the host via putty with X11 forwarding, i can start virt-manager from putty with this warning “** (virt-manager:34569): WARNING **: Couldn’t connect to accessibility bus: Fail ed to connect to socket /tmp/dbus-zDYuKSmbbz: Connection refused”

      in virt manager windows show this error message:
      “Unable to connect to libvirt.
      You need to install openssh-askpass or similar
      to connect to this host.”

      i tried to generate the ssh-key with putty key gen, i also make this command to copy and paste the public-key, (follow a tutorial)

      mkdir -p ~/.ssh
      touch ~/.ssh/authorized_keys
      vi ~/.ssh/authorized_keys
      then copy the public key inside the file save and close it

      when i connect the putty with auth ssh-private-key file can not connect and always timeout

      i check to the server in myuser account nothing is there (no file also no directory), in /home/my_user/ also nothing there. (so i am lost the file i create above dont know going to where actually.

      i am sorry dont know where the resultded folder from this command “mkdir -p ~/.ssh”

      Could you please share and help so i can connect and start create the vm in the host

      Thanks,
      Bun

      1. Sorry – I just saw this comment, for some reason I didn’t get a notification for it.

        The reason you cannot see the folder you made is because files and folders that start with a period are hidden. Try “ls -al” while connected with putty.

        As for running virt-manager with X11 forwarding, sorry I cannot help you with this because I have never done it. IIRC I used to use the SSH option that was integrated into virt-manager.

        EDIT: It looks like there is a Windows client available: https://virt-manager.org/download/ …Why don’t you try that instead of using X11 forwarding?

  3. Hi HUSSEIN,

    Thank you for your replies.
    It’s me again, i have installed the kvm host and can connect with virt-manager into it via ssh, ready to deploy the router, VM, and application into the created VM. For my better capture of your set-up in this tutorial about the network configuration so i can evaluate whether will be suitable for my planned setup, and then do it correctly in my setup, would you please visualize the configuration with picture of pfsense router, VM machines, VM host, LAN, WAN (internet).

    I am sorry for bothering you too much, but it seems your set-up almost same with my plan here.

    Thank you and regards,
    Bun Hin

    1. Hi Bun Hin,

      The set-up will be like a typical hardware router configuration…

      WAN => router => LAN: VM1, VM2, VMn…

      Remember this: The network bridges assigned to your software router kind of act like the ethernet cables connected to a hardware router.

      The WAN port being the network bridge br0 that you created on the host machine’s interface which is connected to the internet. If you want to setup multiple IP addresses, you can set them up all on one router with a single bridge or with multiple software router VMs on the same bridge.

      The LAN port being either a dummy interface or a physical interface which connnects to your physical LAN. If you want to setup multiple subnets and/or isolate traffic for different virtual machine networks then you can use multiple dummy interface without VLANs or physical interface bridges with VLANs.

Leave a Reply

Your email address will not be published. Required fields are marked *