Programster's Blog

Tutorials focusing on Linux, programming, and open-source

ZFS - Create Disk Pools

ZFS

Once you've installed support for ZFS, you need to create a zpool on which we can store data. This tutorial will cover how to create pools with different RAID levels.

  • singular/basic (no RAID)
  • raidz
  • raidz2
  • raid 0
  • raid 10

For this tutorial, I will be using Virtualbox with fixed size disk images of to emulate physical drives attached to my server.

Update 21st October 2017

After performing some tests, I realize now that creating the pools by specifying disks like /dev/sda and /dev/sdb is somewhat dangerous. Your pool could become degraded or disappear if you plug in another drive and all the letters change. To remedy this, use paths with /dev/disk/by-id/... instead or immediately convert your pool after creation by running:

sudo zpool export [my pool name]
sudo zpool import -d [my pool name]

Also, if your pool does become degraded because of the situation described above, running those export/import commands will fix the situation, even if you have already written data since the pool became degraded.

Single Disk (No RAID)

Substitute sdb with whatever your drive is identified as.

sudo zpool create -f [new pool name] /dev/sdb

The -f prevents the error message from preventing the creation. Be careful when using this as you could overwrite existing pools/partitions.

Multiple Disk (RAID 0)

This will create a pool of storage where data is striped across all of the devices specified. Loss of any of the drives will result in losing all of your data.

sudo zpool create -f [new pool name] /dev/sdb /dev/sdc

Adding Drives

You can add drives to a pool to increase its capacity. Any new data will be dynamically striped across the pool, but existing data will not be moved in order to "balance" the pool.

sudo zpool add [existing pool name] /dev/sdd

Raid 1

To create a RAID1 pool (or mirror), simply add the command mirror when creating or adding drives. For example:

sudo zpool create  -f [new pool name] mirror /dev/sdb /dev/sdc
sudo zpool add [existing pool name] mirror /dev/sdd /dev/sde

RAID 10

Creating a RAID1 pool of two drives, and then adding another pair of mirroring drives as shown above would actually create a RAID 10 pool whereby data is striped over two mirrors. This results in better performance without sacrificing redundancy.

To create a RAID10 array in a single command, simply run:

sudo zpool create [pool name] \
  mirror disk1 disk2 \
  mirror disk3 disk4

The output of sudo zpool status would show:

  pool: myPool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        myPool      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0

RAID Z (RAID 5)

RAIDz is very similar to RAID 5 in which parity is generated and stored in order for the RAID array to be able to gracefully handle a drive failing. This means that only one drive's worth of capacity is "lost" in order to provide redundancy for the entire pool of drives. This means that you lose a lot less capacity than with using mirrors which result in you losing 50% of your storage capacity. However, parity cannot be used for scrubbing like RAID 10 can, and calculating parity for every tile transaction can slow down your IO.

To create a RAIDz pool simply run:

sudo zpool create -f [pool name] raidz /dev/sdb /dev/sdc /dev/sdd

You can't add to a RAIDz pool. If you were to run the command below:

sudo zpool add -f [pool name] /dev/sde

Then you would be creating a RAID 0 array over the raidz pool and the drive you just added. This is best shown by the ouput of zpool status:

  pool: poolz
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        poolz       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
          sde       ONLINE       0     0     0

RAIDz2

This is much like RAIDz except that two drives will store parity instead of just one. This means that your array can lose any two drives without loss of data. This means it is actually safer than RAID10 where you would lose all of your data if two drives in the same mirror died. Using RAIDz2 over RAIDz makes a lot of sense because the likelihood of losing a drive when rebuilding a degraded RAIDz array, is greatly increased. Thus losing a drive during the rebuild shouldn't be a problem.

To create a RAIDz2 pool simply run:

sudo zpool create -f [pool name] raidz2 /dev/sdb /dev/sdc /dev/sdd

You need a minimum of 3 drives

RAIDz3

Exactly the same as RAIDz3 except a third drive holds parity and the minimum number of drives is 4. Your array can lose 3 drives without loss of data.

To create a RAIDz3 pool simply run:

sudo zpool create -f [pool name] raidz3 /dev/sdb /dev/sdc /dev/sdd /dev/sde

Mixing and Matching

You may have realized by now that you can mix and match pools together. The pools are always "striped" together creating an effective RAID 0 (although you may need to rebalance/resilver if you added to an existing pool). This means you can combine any number of mirrors and RAIDz pools to create any kind of crazy configuration of your choice. For example, you could run RAID 0 over the top of a RAIDz and mirror arrays.

Pool Balancing

Since ZFS does not have an in-built tool to re-stripe existing data when a drive has been added, I wrote a tool to re-write every file in the pool in order to achieve this. Restriping in this manner results in drives containing an equal share of data, and improves IO. Best of all, this is all done internally, so you do not need to export all of your data and then write it back again like others have suggested.

To use the tool, simply clone the repository and make sure you have PHP installed before running the tool with a path to where your files are kept.

sudo apt-get install php5-cli git -y
git clone https://github.com/programster/zfs-balancer.git
cd zfs-balancer
php main.php /path/to/zfs/pool

Destroying Pools

sudo zpool destroy [pool name]

Conclusion

Now you've setup your storage pools, you probably want to learn the various commands for interfacing with your pool and possibly take snapshots.

References

Last updated: 21st June 2021
First published: 16th August 2018