Borg Backup

Local backup
Remote backup with Kerberos
- Server
- Client
Automatic backup on drive attachment
Getting mail on automatic backup failure (or for any systemd service)
Migrating Attic to Borg
- Server
- Client

Borg is a de-duplicating backup solution. My setup uses separate repositories for each machine which may not be ideal for de-duplication. It is important for file ownership to be checked anytime manual changes are made to a repository because borg doesn't preserve ownership and remote clients won't be able to complete their backups running under the backup user.

Local backup

Initialize a repository

borg init /var/backups/chaos.borg
chown -R backup_<hostname>:root /var/backups/<hostname>.borg

Create a cron job for daily backups

Remote backup with Kerberos

Server

adduser backup_<hostname> --disabled-password
kadmin

kadmin commands

addprinc -policy service -randkey backup_<hostname>@MY_REALM
ktadd -k /path/to/output/new.keytab -norandkey backup_<hostname>@MY_REALM

borg init /var/backups/<hostname>.borg
chown -R backup_<hostname>:root /var/backups/<hostname>.borg

Client

Transfer keytab file to the client (via scp or other secure means): /etc/backup_<hostname>.keytab

The cron job for clients will be similar to the server one above. A kinit line is added for authentication, and the repository is modified for the server remote. Be sure to adjust backup paths and excluded paths for each individual system.

Listing 1: /etc/cron.daily/borg

#!/bin/sh
REPOSITORY=/var/backups/chaos.borg
BORG_CACHE_DIR=/var/backups/borgcache

borg create --stats                              \
    $REPOSITORY::`date +%Y-%m-%d`                 \
    /home                                         \
    /etc                                          \
    /root                                         \
    /path/to                                  \
    /var/lib                                      \
    /var/log                                      \
    /var/mail                                     \
    /var/www                                      \
    --exclude '/home/*/.Trash'                    \
    --exclude '/home/*/.backgrounds'              \
    --exclude '/home/*/.cache'                    \
    --exclude '/home/*/.calibre_library'          \
    --exclude '/home/*/.covers'                   \
    --exclude '/home/*/.dropbox-dist'             \
    --exclude '/home/*/.eve'                      \
    --exclude '/home/*/.local/share/Runic\ Games' \
    --exclude '/home/*/.GOG\ Games'               \
    --exclude '/home/*/.local/share/Steam'        \
    --exclude '/home/*/.local/share/Trash'        \
    --exclude '/home/*/.minecraft/libraries'      \
    --exclude '/home/*/.minecraft/assets'         \
    --exclude '/home/*/.thumbnails'               \
    --exclude '/home/*/.wallpaper'                \
    --exclude '/home/*/.wine'                     \
    --exclude '/home/*/.wine64'                   \
    --exclude '/home/*/Dropbox'                   \
    --exclude '/home/*/VirtualBox\ VMs'           \
    --exclude '/home/*/downloads'                 \
    --exclude /media                              \
    --exclude /path/to/video/family           \
    --exclude /path/to/share                  \
    --exclude /path/to/transfer               \
    --exclude /path/to/video/iso

# When space becomes available start archiving the iso files too.
borg prune -v $REPOSITORY --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly=1

Automatic backup on drive attachment

Borg Docs: Automated backups to local hard drive

First I gathered up my serial numbers and uuids. Something like:

Listing 2: /etc/backups/drive_list

drive           serial                          uuid
====================================================================================
name1           111111111111111111111111        11111111-1111-1111-1111-111111111111
name2           222222222222222222222222        22222222-2222-2222-2222-222222222222
name3           333333333333333333333333        33333333-3333-3333-3333-333333333333
name4           444444444444444444444444        44444444-4444-4444-4444-444444444444
name5           555555555555555555555555        55555555-5555-5555-5555-555555555555
name6           666666666666666666666666        66666666-6666-6666-6666-666666666666
name7           777777777777777777777777        77777777-7777-7777-7777-777777777777
name8           888888888888888888888888        88888888-8888-8888-8888-888888888888
name9           999999999999999999999999        99999999-9999-9999-9999-999999999999
name0           000000000000000000000000        00000000-0000-0000-0000-000000000000

Create mount directories for each drive:

for i in $(cat /etc/backups/drive_list |cut -f 1|tail -n 11); do sudo mkdir -p $i; done

Do up some fstab lines for mounting:

Listing 3: /etc/fstab

UUID=11111111-1111-1111-1111-111111111111 /mnt/backup_disks/name1 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=22222222-2222-2222-2222-222222222222 /mnt/backup_disks/name2 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=33333333-3333-3333-3333-333333333333 /mnt/backup_disks/name3 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=44444444-4444-4444-4444-444444444444 /mnt/backup_disks/name4 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=55555555-5555-5555-5555-555555555555 /mnt/backup_disks/name5 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=66666666-6666-6666-6666-666666666666 /mnt/backup_disks/name6 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=77777777-7777-7777-7777-777777777777 /mnt/backup_disks/name7 ext4 nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=88888888-8888-8888-8888-888888888888 /mnt/backup_disks/name8 ntfs nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=99999999-9999-9999-9999-999999999999 /mnt/backup_disks/name9 ntfs nofail,noauto,x-systemd.device-timeout=1ms 0 2
UUID=00000000-0000-0000-0000-000000000000 /mnt/backup_disks/name10 ntfs nofail,noauto,x-systemd.device-timeout=1ms 0 2

Make some udev rules to run a systemd unit. If the udev rule doesn't seem to fire then increase it's priority to 99. Other rules may take precedence even if they don't cause any action. Also ensure the relevant subsystem is used. Here I'm triggering from the filesystem UUID add event which is on the block subsystem, but if you trigger on the disk or usb device it may be different.

Listing 4: backup_drive_connect.rules

ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}=="11111111-1111-1111-1111-111111111111"

Create the systemd unit template:

Listing 5: automatic-backup@.service

[Unit]
Description=Run backup script with ID_FS_UUID as argument
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/etc/backups/backup.sh %i

[Install]
WantedBy=multi-user.target

Create the backup shell script to run borg. The script encrypts some drives. The passphrase needs to be inside /etc/backups/BORG_PASSPHRASE.

Listing 6: backup.sh

#!/usr/bin/env bash

UUID="$1"
ENCRYPTION=false
BORG_CACHE_DIR=/var/backups/borgcache
export BORG_CACHE_DIR
    
TEST_SET=('/home' '--exclude-from' '/etc/backups/home_excludes') 

# 2TB drives
FILE_SET1=('/' '/home' '/boot' '/boot/efi' '/path/to/video' '/path/to/music' '--exclude-from' '/etc/backups/home_excludes' '--exclude-from' '/etc/backups/root_excludes' '--exclude' '/path/to/video/iso')

# 4TB drives
FILE_SET2=('/home' '/etc' '/root' '/path/to' '/var/lib' '/var/log' '/var/mail' '/var/www' '--exclude' '/path/to/video/family' '--exclude' '/path/to/video/iso' '--exclude-from' '/etc/backups/home_excludes')

# 6TB drives
FILE_SET3=('/mnt/zvideo/iso')

# 2TB safe drive
FILE_SET4=('/path/to' '/home' '--exclude-from' '/etc/backups/home_excludes' '--exclude' '/path/to/video')

# non-backup drives
SET=()

# Mount by uuid (uses fstab entry)
mount -U "$UUID"
MOUNTPOINT="$(lsblk --noheadings --output MOUNTPOINT "/dev/disk/by-uuid/$UUID")"
REPONAME="$(basename "$MOUNTPOINT").borg"

# Determine which file set is used
case $MOUNTPOINT in
        "/mnt/backup_disks/samsung_1tb" )
                SET=("${TEST_SET[@]}");;
        "/mnt/backup_disks/ext_2tb"|"/mnt/backup_disks/car_2tb" )
                SET=("${FILE_SET1[@]}");;
        "/mnt/backup_disks/ext_4tb"|"/mnt/backup_disks/car_4tb" )
                SET=("${FILE_SET2[@]}");;
        "/mnt/backup_disks/ext_6tb"|"/mnt/backup_disks/car_6tb" )
                SET=("${FILE_SET3[@]}");;
        "/mnt/backup_disks/safe_2tb" )
                SET=("${FILE_SET4[@]}");;
        * )
                ;; # Leave SET empty
esac

case $MOUNTPOINT in
        "/mnt/backup_disks/car_2tb"|"/mnt/backup_disks/car_4tb"|"/mnt/backup_disks/car_6tb" )
                ENCRYPTION=true;;
        * )
                ;; # Leave ENCRYPTION false
esac

if [ ${#SET[@]} -gt 0 ]; then
        if [ $ENCRYPTION = true ]; then
                BORG_PASSPHRASE=$(cat /etc/backups/BORG_PASSPHRASE)
                export BORG_PASSPHRASE
        fi

        # Run backup
        ARGS=("$MOUNTPOINT/$REPONAME::$(date +%Y-%m-%d)" "${SET[@]}")
        echo borg create --one-file-system --stats "${ARGS[@]}"
        borg create --one-file-system --stats "${ARGS[@]}"
        RETURN=$?

        # Prune backup
        case $MOUNTPOINT in
                "/mnt/backup_disks/samsung_1tb" )
                        echo borg prune -v "$MOUNTPOINT/$REPONAME" --keep-weekly=4 --keep-monthly=6 --keep-yearly=1
                        borg prune -v "$MOUNTPOINT/$REPONAME" --keep-weekly=4 --keep-monthly=6 --keep-yearly=1
                        ;;
                "/mnt/backup_disks/ext_2tb" | "/mnt/backup_disks/ext_4tb" | "/mnt/backup_disks/ext_6tb" )
                        echo borg prune -v "$MOUNTPOINT/$REPONAME" --keep-weekly=4 --keep-monthly=6 --keep-yearly=1
                        borg prune -v "$MOUNTPOINT/$REPONAME" --keep-weekly=4 --keep-monthly=6 --keep-yearly=1
                        ;;
                "/mnt/backup_disks/car_2tb" | "/mnt/backup_disks/car_4tb" | "/mnt/backup_disks/car_6tb" | "/mnt/backup_disks/safe_2tb" )
                        echo borg prune -v "$MOUNTPOINT/$REPONAME" --keep-weekly=4 --keep-monthly=6 --keep-yearly=1
                        borg prune -v "$MOUNTPOINT/$REPONAME" --keep-monthly=12 --keep-yearly=2
        esac
fi

# Unmount (send a libnotify notification if you can)
umount "/dev/disk/by-uuid/$UUID" 

exit $RETURN

Before enabling the udev rules and systemd unit init the repos on each drive. This is the time to enable encryption if you want it. Just reuse the mountpoint name for the reponame with a .borg extension.

borg init --encryption none /mnt/backup_disks/$MOUNTPOINT/$REPONAME.borg
borg init --encryption repokey-blake2 /mnt/backup_disks/$MOUNTPOINT/$REPONAME.borg

In order to pull data from these backups you need to disable the udev rule or systemd service to prevent it being run on drive connection. Then you can mount the drive and borg mount manually.

TODO

Getting mail on automatic backup failure (or for any systemd service)

Add OnFailure=unit-status-mail@%n.service to the automatic-backup@.service [Unit] section.

Create a unit-status-mail service:

Listing 7: /etc/systemd/system/unit-status-mail@.service

[Unit]
Description=Unit Status Mailer Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/unit-status-mail.sh %i "Hostname: %H" "Machine ID: %m" "Boot ID: %b"

Create a mail script and make it executable:

Listing 8: /usr/local/bin/unit-status-mail.sh

#!/bin/bash
MAILTO="root"
MAILFROM="unit-status-mailer"
UNIT=$1

EXTRA=""
for e in "${@:2}"; do
  EXTRA+="$e"$'\n'
done

UNITSTATUS=$(systemctl status "$UNIT")

mail $MAILTO <<EOF
From:$MAILFROM
To:$MAILTO
Subject:Status mail for unit: $UNIT

Status report for unit: $UNIT
$EXTRA

$UNITSTATUS
EOF

echo -e "Status mail sent to: $MAILTO for unit: $UNIT"

Migrating Attic to Borg

Borg can do a one way conversion for attic repos but it cannot be performed remotely. A backup copy of the repository is made unless –inplace is passed with the upgrade command.

Server

borg upgrade -v <hostname>.attic
mv <hostname>.attic <hostname>.borg
chown -R backup_<hostname>:root /var/backups/<hostname>.borg

Client

sed -i s/attic/borg/ /etc/cron.daily/attic
mv /etc/cron.daily/attic /etc/cron.daily/borg

Borg Backup

Table of Contents

Local backup

Remote backup with Kerberos

Server

Client

Automatic backup on drive attachment

Getting mail on automatic backup failure (or for any systemd service)

Migrating Attic to Borg

Server

Client