Configure KVM Private NAT With Guest Gateway

This tutorial arose out of necessity. Hetzner only allows you a certain amount of "public" IPs to each of your dedicated servers. This means that if you follow the simple/traditional route of giving each of your guests their own public IP (like if you used DigitalOcean hosting etc), then you will quickly run out of IPs and won' be able to host very many services on your server.

What we need to do is create a private network for the majority of our guests to use, and set up a single guest as a gateway that they all connect to, in order to connect with the outside world. We will also be setting up a reverse proxy and port forwarding rules on this gateway so that our guests can be reached from the outside world, not just the other way around.

Steps

Create Private Network

The first thing we need to do is set up a private internal network that all of the guests will be connected to such that they can communicate with each other. This is unlike the default KVM network, which does not allow KVM guests to communicate with each other. This network will use the standard private IP address set, so we don't need to worry about running out of IP addresses.

Add/append this to your KVM host's network/interfaces file (don't change any existing blocks):

auto privatebr0
iface privatebr0 inet static
        address 192.168.0.1
        netmask 255.255.0.0
        pre-up    brctl addbr privatebr0
        post-down brctl delbr privatebr0

Now use virsh to define the private network that relates to the block we just added to the interfaces file:

echo '<network> <name>privatenet</name> <bridge name="privatebr0" /> </network>' >> /tmp/net.xml
virsh net-define /tmp/net.xml
virsh net-start privatenet
virsh net-autostart privatenet

Now if you run

virsh net-list --all

You should see the new network named privatenet.

 Name         State    Autostart   Persistent
-----------------------------------------------
 default      active   yes         yes
 privatenet   active   yes         yes

Create NAT/Gateway Guest

Next, we need to create the NAT/gateway guest.

If you haven't already deployed a Debian guest to fulfill this role, do so now.

I will assume that they have already been set up against a normal bridged setup with a public IP address.

Run the following command to add a network interface to the guest that is connected to this new private network:

virsh attach-interface \
  --domain $GUEST_ID \
  --type network \
  --source privatenet  \
  --model virtio \
  --config \
  --live

If the guest is not running, then just remove the --live part.

Edit the guest in order to see the mac addresses that are assigned to it. If you need to manually set one to a whitelisted MAC address, now is the time to do it.

sudo virsh edit $GUEST_ID

Log into the guest

sudo virsh console $GUEST_ID

Use the following command to get details of your interfaces:

ip addr

Make a note of the MAC address to interface name pairs. We need to make sure we use the name of the interface that is connected to the whitelisted MAC address, for the public IP, and the other interface name for the internal private network that the other KVM guests will be on.

Now, using the names of the networks, set your /etc/network/interfaces file accordingly. Here is mine:

source-directory /etc/network/interfaces.d

auto enp0s2
allow-hotplug enp0s2

# Interface for public IP that can access the outside world
iface enp0s2 inet static
        address xxx.xxx.xxx.xxx
        netmask 255.255.255.255
        point-to-point xxx.xxx.xxx.xxx
        gateway xxx.xxx.xxx.xxx
        dns-nameservers 8.8.8.8 8.8.4.4


# internal private network interface
auto enp1s0
iface enp1s0 inet static
        address 192.168.0.1
        netmask 255.255.0.0
        gateway 192.168.0.1
        dns-nameservers 8.8.8.8 8.8.4.4

Your interface names are likely to be different. The important thing is to use the correct name for the public/private based on MAC address.

Forwarding

We now need to configure the guest to allow forwarding of packets from the guests on the private network, as if they were this server, so they can get through the MAC address filtering.

First we need to tell your server to accept forwarding:

echo 1 > /proc/sys/net/ipv4/ip_forward 
sudo sysctl -p

Edit the /etc/sysctl.conf file

editor /etc/sysctl.conf

...and uncomment this line so there is no # character:

#net.ipv4.ip_forward=1 

IPTables

Now we need to set up iptables rules so that the server will forward packets. Whenever you wish to allow the outside world to gain access to one of your servers, you will need to tweak these rules to forward the relevant port. The only case where you wont need to do this is for websites with ports 80/443 because we will also set this server up as a reverse proxy later.

Create a file with the following contents (tweaking the variables as necessary), and save it somewhere like /root/import-iptables.sh.

#!/bin/bash

# Specify the name of the interface that has access to the outside world
# the one with your public IP
PUBLIC_INTERFACE="enp0s2"

# Specify the name of the interface that is connected to the internal private
# LAN.
INTERNAL_INTERFACE="enp1s0"

# Flushing all rules
/sbin/iptables -F
/sbin/iptables -X

# Setting default filter policy
/sbin/iptables -P INPUT ACCEPT
/sbin/iptables -P OUTPUT ACCEPT
/sbin/iptables -P FORWARD ACCEPT

# Allow unlimited traffic on loopback
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT

/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow new incoming SSH connections from anywhere for this server
/sbin/iptables -I INPUT -p tcp --dport 22    -j ACCEPT

# Allow all new incoming connections from our internal network
/sbin/iptables -I INPUT -p tcp --in-interface $INTERNAL_INTERFACE -j ACCEPT


# Example for rorwarding ftp requests
#HOST="192.168.0.1"
#DEST_SERVER="192.168.0.2"
#/sbin/iptables -t nat -A PREROUTING -d $HOST/32 -p tcp --dport 20   -j DNAT --to-destination $DEST_SERVER:20
#/sbin/iptables -t nat -A PREROUTING -d $HOST/32 -p tcp --dport 21   -j DNAT --to-destination $DEST_SERVER:21
#/sbin/iptables -A FORWARD -p tcp -d $DEST_SERVER --dport 20    -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
#/sbin/iptables -A FORWARD -p tcp -d $DEST_SERVER --dport 21    -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT


# All packets being forwarded should look like they are coming from this server.
sudo iptables --table nat --append POSTROUTING --out-interface $PUBLIC_INTERFACE -j MASQUERADE

# Accept the forwarding of all packets that came in on the internal internal 
# private network for KVM guests.
sudo iptables --append FORWARD --in-interface $INTERNAL_INTERFACE -j ACCEPT

Make sure that file gets executed on boot, either by adding the relevant cron (e.g. @reboot), or adding to your /etc/rc.local file etc.

Reverse Proxy

Finally, port forwarding only helps us when we wish to provide services that use different ports. However, I find that most services are just websites that all want to use ports 80 and port 443. Thus we need to set up a reverse proxy. Luckily I already have a post for that, which you can use.

Other Guests

Make sure that your other guests have a single network interface, and they are configured to use the KVM private network we set up earlier.

sudo virsh edit $GUEST_ID

    <interface type='network'>
      <mac address='52:xx:xx:xx:xx:xx'/>
      <source network='privatenet'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

Then be sure to assign them private static IPs. You will need to keep track of which IPs are already being used and assigned to which server. This is what you will need to set up your reverse proxy and port-forwarding rules against, otherwise we would have set up DHCP.

Below is an example /etc/network/interfaces file of such a guest:

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

auto enp1s0
iface enp1s0 inet static
        address 192.168.0.8
        netmask 255.255.0.0
        gateway 192.168.0.1
        dns-nameservers 8.8.8.8 8.8.4.4

... and below is a netplan configuration for that guest (in case you are using Ubuntu etc)

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    ens6:
        addresses:
            - 192.168.0.2/16
        gateway4: 192.168.0.1
        nameservers:
            addresses:
                - 8.8.8.8
                - 8.8.4.4

References

• Help.ubuntu.com - KVM/Networking

Appendix

IPTables Reset Rules

If you ever mess up when setting up iptables, it's good to have this to hand to perform a reset.

/usr/sbin/iptables --policy INPUT ACCEPT;
/usr/sbin/iptables --policy OUPUT ACCEPT;
/usr/sbin/iptables --policy FORWARD ACCEPT;

/usr/sbin/iptables -Z;
/usr/sbin/iptables -F;
/usr/sbin/iptables -X;