ZFS on Linux - Running Out Of Memory

I recently converted my home fileserver and KVM host over to using ZFS (on Ubuntu 14.04). After the transition, I have noticed that the server appears to always be maxing out it's memory utilization as shown below:

(IMAGE LOST)

It turns out that ZFS is designed to use all physical memory except 1GB on systems that have 4GB or more, which is causing it to push my virtual machines into swap over time, which is not ideal. Here we are going to change some of ZFS's memory utilization settings so that this does not happen.

I want to limit my ARC to 4 GiB on my 16GiB system because I am running several virtual machines, so I run the following command.

echo "options zfs zfs_arc_max=4294967296" \
| sudo tee -a /etc/modprobe.d/zfs.conf

One needs to reboot the server for the changes to take effect.

Checking Utilization

One can check the utilization statistics of their ZFS filesystem with the following command:

cat /proc/spl/kstat/zfs/arcstats

References

• Stack Overflow: Why isn't the arc_max setting honoured on ZFS on Linux?
• Stack Overflow: Lost memory on Linux - not cached, not buffers
• Debian / Ubuntu / Centos / Zfs on Linux: ZFS Limit Arc Cache