I recently converted my home fileserver and KVM host over to using ZFS (on Ubuntu 14.04). After the transition, I have noticed that the server appears to always be maxing out it's memory utilization as shown below:
It turns out that ZFS is designed to use all physical memory except 1GB on systems that have 4GB or more, which is causing it to push my virtual machines into swap over time, which is not ideal. Here we are going to change some of ZFS's memory utilization settings so that this does not happen.
I want to limit my ARC to 4 GiB on my 16GiB system because I am running several virtual machines, so I run the following command.
echo "options zfs zfs_arc_max=4294967296" \ | sudo tee -a /etc/modprobe.d/zfs.conf
One needs to reboot the server for the changes to take effect.
One can check the utilization statistics of their ZFS filesystem with the following command: