Programster's Blog

Tutorials focusing on Linux, programming, and open source

Send And Receive ZFS Snapshots

In this tutorial, I am going to use ZFS send/recieve to sync data from one server to another (primary/slave). For this example, I am going to configure two MySQL servers to store their data in ZFS, and then synchronize them. This will provide me with off-site backups whilst using minimal CPU and network resources because ZFS will only send the differences between the snapshots.

For this tutorial, I am using two Ubuntu 16.04 servers that I installed MySQL on with the command: sudo apt install mysql-server -y. The steps should be mostly the same for other setups.

Setup

We start by creating our ZFS pool on just the the master server (ignore the slave for now):

sudo zpool create -f pool1 /dev/sdb
sudo zfs create pool1/mysql

For optimal performance, we are also going to change the record size from the default 64K to 16K.

sudo zfs set recordsize=16K pool1/mysql

Now stop the mysql service.

sudo service mysql stop

Now use the rsync command to copy the existing data into the dataset we created, whilst preserving permissions.

# do this as root
sudo su

rsync \
--recursive \
--verbose \
--human-readable \
--progress \
--links \
--copy-unsafe-links \
--owner \
--group \
--perms \
--times \
--force \
--delete \
--delete-before \
--ignore-errors \
--sparse \
--info=progress2 \
/var/lib/mysql/ /pool1/mysql/.

Now remove the existing data and change the mountpoint of the dataset to be at that location and grant the mysql user access to it.

sudo rm -rf /var/lib/mysql
sudo zfs set mountpoint=/var/lib/mysql pool1/mysql
sudo chown mysql:mysql /var/lib/mysql

Create First Snapshot

Create our first snapshot called "base" (you can choose a different name).

sudo zfs snapshot pool1/mysql@base

Optional - use the command below to list snapshots to see that it is there.

sudo zfs list -t snapshot  

At this point you can start the MySQL service. Any changes that are made will go into further snapshots later.

sudo service mysql start

Now we need to sync this initial snapshot over to the other server, which will require sending all of the data. Unfortunately, we need to be able to ssh in as root for the recieving end so set a password or ssh key login for the root user on the slave database. You may need to edit the /etc/ssh/sshd_config file and change

PermitRootLogin prohibit-password

to:

PermitRootLogin yes

When you are able to ssh into the slave server as root, run the command below to send the first snapshot.

sudo zfs send pool1/mysql@base | \
ssh root@10.2.0.11 zfs recv pool1/mysql

Note: we use the dataset names rather than the mount paths.

Now you will have the dataset on your slave. If you had already created the dataset on the slave, the command above would have failed. Now perform the necessary steps to have the mysql server use that data:

sudo service mysql stop
sudo rm -rf /var/lib/mysql
sudo zfs set mountpoint=/var/lib/mysql pool1/mysql
sudo service mysql start

Now if you log into the slave database, you should see all your data.

Steps for Future Syncs

Syncing future snapshots is slightly different. Whenever you want to sync the servers perform the following:

  1. Shutdown databases on both servers: sudo service mysql stop.
  2. Create a snapshot on the primary: sudo zfs snapshot [pool]/[dataset name]@[new snapshot name]
  3. Rollback the slave to the last snapshot to remove any changes that were made. If there are changes, the sync command in the next step will fail: sudo zfs rollback pool1/mysql@base
  4. Send the snapshot from the master to the slave server, incrementally:
sudo zfs send -i \
[pool]/[dataset]@[previous snapshot name] \
[pool]/[dataset]@[new snapshot name] | \
ssh root@slave zfs recv [slave pool]/[slave dataset]
  1. list snapshots on slave : sudo zfs list -t snapshot
  2. Start the MySQL databases. sudo service mysql start

Conclusion

You are now able to efficiently sync your MySQL database from one host to another through incremental ZFS snapshots. This is a great way to provide off-site backups or a development/test server. I used MySQL as an example here, but you could use this technique for syncing any kind of data.