ZFS

Enable extended attributes for Linux

Without this enabled extended attributes are stored in a hidden subdirectory. This hurts performance, and clutters things.

zfs set xattr=sa tank tank2 tank3

Exporting ZFS over NFS

On Linux, remote mounted ZFS over NFS filesystems don't follow umask unless POSIX ACLs are enabled. This is done with:

zfs set acltype=posixacl tank tank2 tank3

It's also possible to manage NFS sharing directly from ZFS properties rather than exports(5) entries.

Some notes:

Don't need to bind mount filesystem in /export/
Children datasets inherit this property, but can override it in their own properties
You can manually toggle sharing with zfs share and zfs unshare (-a for all filesystems)

zfs set sharenfs=sec=krb5,rw=@192.168.1.0/24,rw=@10.12.0.0/24,async,no_subtree_check,root_squash storage_pool

Other common tasks with ZFS

List pools, check status:

zfs list
zpool list
zpool status

Mounting, unmounting, mount all

zfs mount tank
zfs unmount tank
zfs mount -a

Prepare pool for another machine

To make a pool ready for importing elsewhere mark its members inactive by exporting:

zpool export tank

Automatic import may use generic device names. I found it preferable to use the device id because it identifies the disk better. Import a pool using specific device names (like id, uuid, etc.).

zpool import -d /dev/disk/by-id tank

Scan and import existing pools:

systemctl start zfs-import-scan.service

Display a history of a pool/all pools:

zpool history tank
zpool history

List block devices that are ZFS members:

lsblk -f|grep zfs

Display more info than you want to know about your pools:

zdb -U /etc/zfs/zpool.cache

Remove a device from the pool:

zpool remove tank <device_name>

Replace a device with another in the pool:

zpool replace tank <old_device_name> <new_device_name>

I/O Statistics, and with individual disks:

zpool iostat
zpool iostat -v

Considerations for an external drive pool

TODO

always export before disconnecting. If you don't, and need to import it on another machine you'll need to force it:

zpool import -f tank

scrubbing won't happen on a regular schedule so you'll have to manage it.

import with appropriate label (will search and import any pools)Hide

zpool import -d /dev/disk/by-id/ video_pool

OR

import with specific device names if you wantHide

zpool import -d /dev/disk/by-id/usb-WD_Elements_25-0\:0 -d /dev/disk/by-id/usb-WD_Elements_25-0\:0 video_pool

Do your work

unmount and exportHide

zfs unmount video_pool
zpool export video_pool

Remove usb device (udiskie, udisksctl power-off).

Moving a pool to new drives (example with my video_pool mirror)

This was my process for moving my video mirror from a pair of full internal disks (2x8TB) to a set of larger external disks (2x12TB), but keeping the same data.

May be worth noting that zfs send/recv on a local machine is probably slower than using cp/rsync. Take care to preserve attributes (file dates, etc.) if you use one of these.

1. get history, so we can create the new pool with same settingsHide

sudo zpool history video_pool|grep -v import|grep -v scrub

2. init new pool, configure pool.Hide

sudo zpool create -f new_video_pool mirror \
      /dev/disk/by-id/usb-WD_Elements_25-0\:0 \
      /dev/disk/by-id/usb-WD_Elements_25-0\:0
sudo zfs set xattr=sa new_video_pool

3. snapshot old poolHide

sudo zfs snapshot video_pool@migrate12tb

4. send old pool (takes a long time), copies propertiesHide

sudo -c 'zfs send -vR video_pool@migrate12tb | zfs recv -F new_video_pool'

5. rename old poolHide

sudo zfs unmount video_pool
sudo zpool export video_pool
sudo zpool import video_pool old_video_pool

6. change old pool mountpointHide

sudo zfs set mountpoint=/mnt/old_video_pool old_video_pool

7. rename new pool to old nameHide

sudo zfs unmount new_video_pool
sudo zpool export new_video_pool
sudo zpool import -d /dev/disk/by-id/ new_video_pool video_pool

8. define mountHide

sudo zfs set mountpoint=/mnt/video_pool video_pool

9. finish initializing unallocated space (wait for finish before returning)Hide

sudo zpool initialize video_pool -w

.

10. (optional) verify copy. (note: I tried zfs diff, but it only compares snapshots of the same pool)Hide

diff -qrN /mnt/video_pool/ /mnt/old_video_pool/

My zpool history

An abridged (scrubs and automatic imports removed) zpool history of my pools:

zpool historyHide

History for 'zbackup' (4x 2TB raidz):
zpool create -f -m /var/backups/ zbackup raidz /dev/disk/by-id/ata-WDC_WD20EFRX_WD-1 /dev/disk/by-id/ata-WDC_WD20EFRX_WD-2 /dev/disk/by-id/ata-WDC_WD20EFRX_WD-3 /dev/disk/by-id/ata-WDC_WD20EFRX_WD-4
zpool upgrade zbackup

History for 'zhome' (2x 240GB SSD in mirror + spare):
zpool create zhome mirror ata-SATA_SSD_1 ata-SATA_SSD_2
zpool add zhome spare ata-SATA_SSD_3
zfs set mountpoint=/home zhome
zpool upgrade zhome
zpool trim zhome ata-SATA_SSD_1
zpool trim zhome ata-SATA_SSD_2
zpool set autotrim=on zhome
zpool set autotrim=off zhome
zfs set xattr=sa zhome

History for 'storage_pool' (4x 4TB raidz2 + optane cache):
zpool create storage_pool raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd
zfs set mountpoint=/path/to/storage storage_pool
zpool import -f storage_pool
zpool import storage_pool
zpool export storage_pool
zpool import -d /dev/disk/by-id/ storage_pool
zpool add storage_pool cache /dev/disk/by-id/nvme-INTEL_MEMPEK1W016GA
zpool upgrade storage_pool
zfs set xattr=sa storage_pool
zfs set acltype=posixacl storage_pool

History for 'storage_pool' (2x 4TB mirror, lz4 compression, NFS share):
zpool create -m /path/to/storage -o feature@lz4_compress=enabled -O compression=on -O acltype=posixacl -O xattr=sa storage_pool mirror /dev/disk/by-id/ata-WDC_WD40EFRX_WD-1 /dev/disk/by-id/ata-WDC_WD40EFRX_WD-2
zfs receive -F storage_pool
zfs set sharenfs=sec=krb5,rw=@192.168.1.0/24,rw=@10.12.0.0/24,async,no_subtree_check,root_squash storage_pool

History for 'video_pool' (2x 8TB mirror):
zpool create -f -m /mnt/videotemp video_pool mirror /dev/disk/by-id/ata-WDC_WD80EMAZ-1 /dev/disk/by-id/ata-WDC_WD80EMAZ-2
zfs set mountpoint=/mnt/video_pool video_pool
zpool import video_pool
zpool clear video_pool
zpool upgrade video_pool
zfs set xattr=sa video_pool zbackup

SSD cache drives

SSD cache drives probably aren't worth it unless your data exceeds RAM.

For example I built this document, which touches at least a few hundred megs of images, and gains were only 0.5s on 48s.

Setup was two 8TB WD WD80EMAZ, and a Kowin 256GB M.2 SATA SSD.

make clean-html; time make html:
MIRROR, NO CACHE
    real  0m47.518s 0m47.493s 0m47.559s 0m47.846s 0m47.977s
    user  3m23.100s 3m23.030s 3m22.193s 3m22.782s 3m22.827s
    sys   0m13.766s 0m13.356s 0m13.298s 0m13.390s 0m13.323s

MIRROR, CACHE (256GB SATA SSD)
    real  0m46.994s 0m46.926s 0m46.998s 0m47.110s 0m47.071s
    user  3m21.965s 3m22.086s 3m22.342s 3m22.285s 3m22.709s
    sys   0m13.348s 0m12.908s 0m12.870s 0m13.360s 0m13.254s