Disaster recovery rehearsal

This runbook proves that a clean Debian VM can be converged and a real application restored from the offsite appdata backup. It is a rehearsal, not a production restore procedure. The first target is Actual Budget because its state is confined to appdata/actual and its image has a built-in health check.

Do not record application contents, credentials, repository URLs, or restored filenames as evidence. Snapshot IDs, timestamps, elapsed time, numeric ownership, file counts, byte counts, container health, and mount paths are enough.

Recovery boundaries

Data class	Recovery source	Rehearsal proof
Host configuration	Public Git repository plus the rebuild guide	Clean-VM Ansible converge and idempotence
Application data	Offsite restic repository	Verified staged restore of `appdata/actual`
Secrets	Operator-supplied recovery values	Minimal temporary restic-only environment; never the production `.env`
Media	Surviving mergerfs/SnapRAID data disks	Out of scope; media is not in the appdata backup

The timer starts when the clean VM first accepts SSH. Record two elapsed figures: host recovery to successful Ansible convergence, and application recovery to a healthy Actual container using the restored directory.

Safety gate

Before continuing, confirm all of the following:

The target is a disposable Debian 13 VM on the existing Proxmox test host, not the production host and not a production disk or mount.
The VM has no Cloudflare tunnel, public ingress, or production data mounts.
The repository is copied without .env files. Only four restic/B2 variables are entered into a root-only file under /run, which is memory-backed and disappears at reboot.
Actual mounts /mnt/cache/recovery-staging/actual, never /mnt/cache/appdata/actual.
The VM will be destroyed after the evidence is recorded.

Stop if any check is false.

1. Create and converge a clean VM

Create a minimal Debian 13 VM manually in Proxmox with a non-root operator and the workstation SSH key. VM lifecycle automation is optional and warranted only if manual creation becomes a demonstrated burden.

From the task worktree on the workstation, run the existing rehearsal. It loads the production address from the ignored private site.yml safety rail, refuses any target resolving to it, and copies no .env files. Set SITE_FILE only when the escrowed site file lives outside the task worktree:

cd ansible/tests/vm
VM=<operator>@<vm-address> ./rehearse.sh all

The full rehearsal ends with a reboot, which removes its disposable loop mounts. Recreate the known-good storage stand-ins before restoring:

VM=<operator>@<vm-address> ./rehearse.sh bootstrap

Record the rehearsal pass count, the first-converge completion time, and the second-converge changed=0 result. Do not continue after a failed proof.

2. Provide only the restore credentials

On the VM, create /run/homelab-recovery.env as root:root 0600 and enter only these assignments:

RESTIC_PASSWORD=...
RESTIC_B2_REPO=...
B2_ACCOUNT_ID=...
B2_ACCOUNT_KEY=...

Do not copy the production root, monitoring, or Actual .env files. Actual needs no application secret for this isolated check; its example environment provides the timezone and appdata path.

3. Stage and verify the Actual restore

Run the restore helper as root so restic can preserve numeric ownership. The subshell sources the root-only file without displaying any value:

sudo bash -c '
  set -a
  . /run/homelab-recovery.env
  set +a
  export RESTIC_REPOSITORY="$RESTIC_B2_REPO"
  export RESTIC_CACHE_DIR=/var/cache/restic
  exec /opt/homelab/scripts/stage-appdata-restore.sh --app actual
'

The helper selects the latest homelab snapshot tagged appdata, discovers the snapshot's changing btrfs path, restores only the actual subtree with restic verification, and refuses an existing destination. It can never target the live appdata root. Use --snapshot <exact-id> to repeat the rehearsal from a specific restore point. Restic backend errors are kept out of normal output; on failure, the helper reports the path to a mode-0600 temporary diagnostic log. Inspect that file locally and do not paste it into evidence or tickets.

Record the helper's snapshot timestamp, file count, byte count, and restore duration. A successful --verify proves that the downloaded bytes match the repository.

4. Start Actual against the staged data

The VM sync deliberately excludes every .env* file, including examples. Create the isolated two-variable environment directly; it contains no secret:

cd /opt/homelab/actual
umask 077
printf 'TZ=Europe/Brussels\nAPPDATA=/mnt/cache/recovery-staging\n' > .env.recovery
grep -qx 'APPDATA=/mnt/cache/recovery-staging' .env.recovery
test "$(stat -c '%a' .env.recovery)" = 600
docker compose --env-file .env.recovery config --quiet
docker compose --env-file .env.recovery -p recovery-actual up -d actual

Wait for the built-in health check and prove that the container is using the staged mount:

actual_id="$(docker compose --env-file .env.recovery -p recovery-actual ps -q actual)"
test -n "$actual_id"
docker inspect --format '{{.State.Health.Status}}' "$actual_id"
docker inspect --format '{{range .Mounts}}{{if eq .Destination "/data"}}{{.Source}}{{end}}{{end}}' "$actual_id"
docker exec "$actual_id" sh -c 'find /data -type f -size +0c -print -quit | grep -q .'
sudo stat -c '%u:%g %a' /mnt/cache/recovery-staging/actual

Acceptance requires healthy, the exact staging path, a zero exit status from the non-empty-state check, and usable numeric ownership. Do not publish the VM or inspect financial contents merely to strengthen the proof.

5. Record and destroy

Record:

Debian, Ansible, Docker, Compose, restic, and Actual image versions;
VM CPU, memory, and disk allocation;
selected snapshot timestamp and age;
host-convergence and application-recovery elapsed times;
restore file/byte counts, numeric ownership, mount source, and health;
every deviation, failed attempt, and manual prerequisite.

Then stop the isolated application, remove the temporary credential file, run the fixture teardown, and destroy the VM in Proxmox:

cd /opt/homelab/actual
docker compose --env-file .env.recovery -p recovery-actual down
rm -f -- .env.recovery
sudo rm -f /run/homelab-recovery.env
# Remove only root-owned, mode-0600 diagnostic logs created by this helper.
sudo find /tmp -maxdepth 1 -type f -user root -perm 0600 \
  -name 'stage-appdata-restore.*.log' -delete
cd /opt/homelab/ansible/tests/vm
sudo bash fixtures.sh teardown

The rehearsal is complete only after the VM is destroyed and the result is recorded here or in a dated acceptance record linked from this page.

Acceptance record: 2026-07-14

Result: PASS. A fresh Debian 13 VM on the existing Proxmox host converged, restored a same-day Actual Budget snapshot from the offsite restic repository, started the application against only the restored staging tree, passed every acceptance check, and was destroyed after cleanup.

Environment and timing

Item	Observed result
VM allocation	2 vCPU, 4,015,932 KiB RAM, 32 GiB virtual disk
Software	Debian 13; Ansible 2.19.4; Docker 26.1.5; Compose 2.26.1; restic 0.18.0
Application	`actualbudget/actual-server:26.7.0-alpine`
Full host/deploy rehearsal	7m43s; 84 passed, 0 failed
Idempotence	second host converge reported `changed=0`
Restore point	snapshot `d36bf7afc8d3`, 2026-07-14 04:00:57 CEST
Verified restore	4 files; 2,853,318 bytes; 4 seconds
Application startup	first healthy result 60 seconds after container start

The 7m43s measurement covers the automated rehearse.sh all run after the initial manual Debian/SSH preparation and first bootstrap; those manual steps were not timed and are therefore not claimed as a full bare-metal RTO.

Acceptance evidence

The existing harness proved clean converge, fail-closed mount/network/secret checks, Docker restart safety, transactional SSH, check mode, reboot persistence, Compose gating, and deployment with 84 passed, 0 failed.
The restore helper selected the homelab snapshot tagged appdata, resolved its changing btrfs snapshot path, restored only appdata/actual, and ran restic's post-restore byte verification.
Actual reported healthy, its live /data mount source was exactly /mnt/cache/recovery-staging/actual, and a container-side assertion found non-empty state without printing application contents or filenames.
Restored directory ownership and mode were 0:0 0700; the application could read and use that state while the unprivileged host operator could not.
The isolated Actual container and network were removed, the temporary recovery environment, restic credentials, and protected error log were deleted, the fixture mounts were torn down, and the VM plus its virtual disk were destroyed in Proxmox.
Media was neither mounted nor restored. Host configuration came from Git; the only secrets present were the four temporary restic/B2 values.

Safe failures and corrections

The first restore attempt used only a bucket name instead of the full restic repository locator. Snapshot selection failed before creating the staging destination. The exact single locator was then transferred directly from production over SSH without displaying or storing it on the workstation.
The first Compose attempt stopped before container creation because the VM sync correctly excludes all .env* files, including .env.example. This runbook now creates the minimal non-secret recovery environment directly.
The first host ownership check ran unprivileged and was denied by the restored 0700 directory. The check now uses sudo and records only numeric ownership and mode.