Send And Receive ZFS Snapshots

Introduction

In this tutorial, I am going to use ZFS send/recieve to sync data from one server to another (primary/slave). For this example, I am going to configure two MySQL servers to store their data in ZFS, and then synchronize them. This will provide me with off-site backups whilst using minimal CPU and network resources because ZFS will only send the differences between the snapshots.

For this tutorial, I am using two Ubuntu 16.04 servers that I installed MySQL on with the command: sudo apt install mysql-server -y. The steps should be mostly the same for other setups.

Setup

Create The ZFS Pool On Master

We start by creating our ZFS pool on just the the master server (ignore the slave for now):

sudo zpool create -f pool1 /dev/sdb
sudo zfs create pool1/mysql

Set The Record Size

For optimal performance, we are also going to change the record size from the default 64K to 16K.

sudo zfs set recordsize=16K pool1/mysql

Stop The Master MySQL Server

Now stop the mysql service.

sudo service mysql stop

Rsync Files Into Master Dataset

Now use the rsync command to copy the existing data into the dataset we created, whilst preserving permissions.

# do this as root
sudo su

rsync \
  --recursive \
  --verbose \
  --human-readable \
  --progress \
  --links \
  --copy-unsafe-links \
  --owner \
  --group \
  --perms \
  --times \
  --force \
  --delete \
  --delete-before \
  --ignore-errors \
  --sparse \
  --info=progress2 \
  /var/lib/mysql/ \
  /pool1/mysql/.

Configure MySQL To Use ZFS Dataset's Files

Now remove the existing data and change the mountpoint of the dataset to be at that location and grant the mysql user access to it.

sudo rm -rf /var/lib/mysql
sudo zfs set mountpoint=/var/lib/mysql pool1/mysql
sudo chown mysql:mysql /var/lib/mysql

Create First Snapshot

Create our first snapshot called "base" (you can choose a different name).

sudo zfs snapshot pool1/mysql@base

Optional - use the command below to list snapshots to see that it is there.

sudo zfs list -t snapshot  

Start The MySQL Server

At this point you can start the MySQL service. Any changes that are made will go into further snapshots later.

sudo service mysql start

Permissions Preparation For ZFS Sync

Now we need to sync this initial snapshot over to the other server, which will require sending all of the data.

Unfortunately, we need to be able to SSH in as root for the recieving end so set a password or SSH key login for the root user on the slave database. You may need to edit the /etc/ssh/sshd_config file and change

PermitRootLogin prohibit-password

to:

PermitRootLogin yes

Alternatively, if you enable passwordless sudo access on a non-root user account, you can just use that and apply sudo to the command below.

Sync First Snapshot

When you are able to SSH into the slave server as root, run the command below to send the first snapshot.

sudo zfs send pool1/mysql@base | \
  ssh root@10.2.0.11 zfs recv pool1/mysql

We use the dataset names rather than the mount paths. Make your life easier and just configure the remote host in your .ssh/config file. Then you can easily use keys and not have to change the command to specify a key file etc. If you had already created the dataset on the slave, the command above will fail.

Alternative - Show Progress With PV

With the command above, the shell will just sit there with no output until the job has completed which can be a bit disconcerting. If you would like to see some progress updates, such as how much data has transferred and how fast it is going, you can pipe it through PV.

sudo zfs send pool1/mysql@base | \
  pv | \
  ssh root@10.2.0.11 zfs recv pool1/mysql

You need to have installed the pv package, which can usually be installed by running sudo apt install pv.

Alternative - Pulling From Slave

Alternatively, if you need to pull from the slave:

ssh root@remote-host zfs send pool1/mysql@base | \
  sudo zfs recv pool1/mysql

Configure Slave MySQL Server

Now you will have the dataset on your slave. Now perform the necessary steps to have the mysql server use that data:

sudo service mysql stop
sudo rm -rf /var/lib/mysql
sudo zfs set mountpoint=/var/lib/mysql pool1/mysql
sudo service mysql start

Now if you log into the slave database, you should see all your data.

Steps for Future Syncs

Syncing future snapshots is slightly different. Whenever you want to sync the servers perform the following:

1. Shutdown databases on both servers: sudo service mysql stop.
2. Create a snapshot on the primary: sudo zfs snapshot [pool]/[dataset name]@[new snapshot name]
3. Rollback the slave to the last snapshot to remove any changes that were made. If there are changes, the sync command in the next step will fail: sudo zfs rollback pool1/mysql@base
4. Send the snapshot from the master to the slave server, incrementally:

sudo zfs send -i \
  [pool]/[dataset]@[previous snapshot name] \
  [pool]/[dataset]@[new snapshot name] | \
  ssh root@slave zfs recv [slave pool]/[slave dataset]

5. list snapshots on slave: sudo zfs list -t snapshot
6. Start the MySQL databases. sudo service mysql start

Conclusion

You are now able to efficiently sync your MySQL database from one host to another through incremental ZFS snapshots. This is a great way to provide off-site backups or a development/test server. I used MySQL as an example here, but you could use this technique for syncing any kind of data.