Ansible host-convergence behavior contract
This is the source of truth for the state Ansible must maintain on the homelab
host. It was derived from the previous install.sh workflow and a read-only
audit of the server (see the audit snapshot).
Ansible owns host convergence and Compose owns workloads. This file records the observable behavior that boundary preserves. Current Ansible configuration and verified host behavior supersede the frozen bash installer.
Implementation status
All eleven host.yml roles and the deploy.yml wrapper are implemented,
VM-rehearsed, and in use on the production server. The production cutover ran
on 2026-07-13; the next-day checks and first Renovate-driven image refresh are
recorded in the acceptance pages.
preflight,base_packages, andmaintenance_jobs: PR #83.secrets,storage_assert,network_assert,backup,monitoring_host(slice 2, PR #94). Thehomepage_configrole delivered in that slice was retired on 2026-07-13 when Grafana became the service launcher. Proofs:acceptance-slice1.md,acceptance-slice2.md.docker_host,hardening, andssh_hardening: PR #97. Proof:acceptance-slice3.md.playbooks/deploy.ymlandcompose_deploy: PR #97. Proof:acceptance-deploy.md.
Every production proof is recorded, including the daemon-initiated parity
spindown observed on 2026-07-14 after the hd-idle filter fix. With the burn-in
complete, install.sh was retired (removed from the tree; see Git history).
Line references to install.sh in the tables below are historical provenance
for where each behavior originated.
Ownership boundaries
Four systems own host state. Ansible only converges the first.
| Owner | Owns | Ansible's stance |
|---|---|---|
Ansible (host.yml) |
Debian packages, group membership, files, file modes/ownership, systemd units + timers, udev/module/apt drop-ins, prereq directories, the shared proxy docker network, daemon.json |
Full convergence, idempotent |
Compose (deploy.yml calls the Compose CLI) |
Containers, image pins, bind mounts, healthchecks, app networks | Ansible never manages container lifecycle; host.yml and deploy.yml are separately invocable |
OPNsense (router homelab_router_address) |
Addressing, DHCP, routing, DNS resolution, IPv6 advertisement | Ansible writes no static host networking; it only asserts runtime and fails loudly, never remediates |
| Manual prereq (documented, one-time) | Debian install, the homelab_user admin user, disk partitioning/formatting, /etc/fstab, snapraid.conf, Docker Engine install (creates the docker group), GPU driver, ansible-core (via apt), the three .env files |
Ansible verifies presence and refuses to run if absent; it never creates or destroys these |
Non-negotiable safety invariants (CI static-asserts these against host.yml):
- Routine convergence NEVER partitions, formats,
mkfs, orsnapraid inits any disk. CI static-asserts this across the entireansible/tree (playbooks/androles/*/tasks/*.yml, since implementation lives in roles, not justhost.yml), covering both module forms (community.general.parted/.filesystem,ansible.builtin.mountwith a format) and rawcommand:/shell:invocations ofmkfs,mkfs.*,wipefs,parted,sgdisk,dd of=/dev/, andsnapraid init. - No Ansible Vault. The three
.envfiles remain the sole server-side secret store. Roles verify presence + enforce0600, read withno_log: true, keep theEnvironmentFile=model, and never copy secret values into facts or inventory. - Docker restart is a gated disruptive op (see DOCKER-2).
How to read the matrix
Each concern maps to one role. Every row is one asserted behavior:
- Desired state: what must be true after converge.
- Source: where the behavior comes from today (file/line, unit, or live host).
- Why: the consequence of getting it wrong (the reason it is in the contract).
- Converge: what the role does to make it true (idempotent).
Every behavior carries two independent proofs, because the rehearsal environment cannot reproduce production:
- VM proof: provable on the disposable Debian 13 VM (
--tags, check-mode, double-converge idempotence, reboot persistence). This is the CI/rehearsal gate. - PROD proof: provable only against the real host (real mergerfs/SnapRAID pool, real SATA HBA, real backups, and real containers). This covers cutover and ongoing operational validation.
NEG marks a negative test: the behavior is proven by deliberately creating
the failure precondition (an absent mount stand-in, an invalid sshd candidate)
and asserting the role aborts. See Proof strategy for why the
VM cannot use the real pool and how stand-ins substitute for it.
Behavior matrix
PREFLIGHT - role: preflight
Runs first; a failure here aborts the whole play before any mutation.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| PRE-1 | /mnt/user is a fuse.mergerfs mount |
install.sh:9; live |
Writes below an unmounted union land on the OS disk (empty Plex library, downloads on /) |
assert FSTYPE == fuse.mergerfs, else fail |
| PRE-2 | /mnt/cache is mounted and its FSTYPE is btrfs |
install.sh:11 (mountpoint only); live: btrfs |
Appdata/secrets below an unmounted cache land on the OS disk; and an accidentally-ext4 cache passes a bare mountpoint check yet later breaks btrfs-scrub + the appdata-backup btrfs snapshot |
assert mountpoint and FSTYPE == btrfs, else fail (stricter than today's install.sh) |
| PRE-3 | The three .env files exist: /opt/homelab/.env, .../monitoring/.env, .../actual/.env |
install.sh:51, install-host.sh:10 |
Secrets are a manual prereq; roles must not fabricate them | stat, fail with a precise "create it, then re-run" message |
| PRE-4 | Running from /opt/homelab (repo is load-bearing) |
every ExecStart=/opt/homelab/... in units |
Systemd units hardcode /opt/homelab/scripts/...; a repo elsewhere silently breaks every timer |
assert playbook basedir == /opt/homelab |
| PRE-5 | homelab_user user and docker group both exist (manual prereqs) |
install.sh:96 (getent group docker, id homelab_user); ownership table |
HARD-3's group-membership converge is meaningless if either is absent; failing here beats silently skipping | assert both present, else fail with a "create the admin user / install Docker first" message |
| ID | VM proof | PROD proof |
|---|---|---|
| PRE-1 | NEG: bind-mount an empty dir at /mnt/user (wrong FSTYPE) → converge aborts; then a loop fuse stand-in typed as expected → passes |
findmnt /mnt/user == fuse.mergerfs on the real pool; full host.yml proceeds |
| PRE-2 | NEG: mount an ext4 loop stand-in at /mnt/cache → abort on FSTYPE; a btrfs loop stand-in → pass; unmounted → abort |
Live: /mnt/cache real btrfs present at cutover |
| PRE-3 | Touch/remove stub .env files → pass/abort accordingly |
Real escrowed .env files present |
| PRE-4 | Run play from /opt/homelab clone on VM → pass; from /tmp/x → abort |
Cutover runs cd /opt/homelab && git pull && cd ansible && ansible-playbook ... |
| PRE-5 | Create homelab_user + docker group on VM → pass; remove either → abort |
Live: both present (homelab_user in docker group) |
SECRETS - role: secrets
Enforces the secret-permission invariant from the ownership boundary and creates
private appdata directories required by fixed-UID containers before deployment.
PRE-3 only proves the secret files exist; this role proves they are not
world-readable. It never reads or copies secret values (no_log: true) and
never parses .env.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| SEC-1 | /opt/homelab/.env, .../monitoring/.env, .../actual/.env each 0600, owned by the deployment operator homelab_user (not root) |
install.sh:50-53 (chmod 600, ownership preserved); live: homelab_user:homelab_user 600 |
A pre-existing 0644 .env leaks credentials to every host user; but forcing root:root breaks deploys - homelab_user (in the docker group per HARD-3) runs docker compose non-root and the CLI reads .env/env_file as that user, so root-only makes Compose fail |
file mode 0600, owner homelab_user:homelab_user, no_log: true; tighten mode only, never widen, never template values |
| SEC-2 | /mnt/cache/appdata/anibridge private: dir 0700, tree owned 99:100 (PUID/PGID), no group/other bits, no executable regular files |
install.sh:55-67 at its 2026-07-14 retirement (install -d -m 0700, chown -Rh, chmod -Rh a-x,u=rwX,go=); added to the installer after this contract was derived, ported when the installer was removed |
AniBridge /config holds provider tokens, its database, and AniList backups; images before the compose UMASK=077 pin created world-readable files, and symlinks inside appdata must not be dereferenced during remediation |
file creates the dir 0700 99:100; a find detect task (read-only) registers violations; chown -Rh+chmod -Rh remediation runs only when a violation is found (changed=0 on a clean tree) |
| SEC-3 | Audiobookshelf config and metadata appdata directories exist as 0700, owned 99:100 |
Audiobookshelf container's fixed numeric user and writable bind mounts | Docker would otherwise create missing bind sources as root, preventing the non-root container from initializing; the private directories are included in the existing appdata snapshot and restic backups | file creates both pinned paths before Compose deployment; no .env parsing |
| ID | VM proof | PROD proof |
|---|---|---|
| SEC-1 | NEG: seed a stub .env as homelab_user at 0644 → converge tightens to 0600 (owner preserved as homelab_user); assert numeric mode/owner; a pre-0600 file reports changed: false (idempotent) |
Live: all three homelab_user:homelab_user 600; homelab_user can still docker compose config/up; secret values never appear in facts, output, or logs |
| SEC-2 | the secrets role executes inside the full VM rehearsal converge (dir created 0700 99:100, detect task runs); tests/test-anibridge.sh statically asserts the pinned path/UID/GID/0700, the symlink-safe -Rh remediation commands, and the detect-gated when: |
First post-retirement converge: SEC-2 tasks report changed=0 on the already-remediated live tree (detect task finds no violations) |
| SEC-3 | static runtime guards assert both pinned paths, numeric ownership, and 0700; full VM rehearsal exercises the role |
Pending owner-authorized host convergence before the first Audiobookshelf deployment |
STORAGE (assert-only) - role: storage_assert
Ansible never touches disks. It only re-asserts the mount topology that the
manual /etc/fstab prereq established, so a converge on a degraded pool fails
fast instead of writing into the void.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| STO-1 | All five branch mounts (/mnt/disk1../mnt/disk5) present, the mergerfs pool at /mnt/user is composed of exactly those branches, and parity (/mnt/parity) is outside the pool |
storage-layout.md; live |
A fuse.mergerfs type check alone (PRE-1) still passes on a degraded pool missing a branch, or a pool that wrongly included parity |
assert the five branch mountpoints exist and verify effective pool membership (mergerfs branch list) excludes /mnt/parity; no fstab writes |
| STO-2 | Cache is single-device btrfs, never mkfs |
storage-layout.md, host-facts.md |
The reused Unraid pool carries appdata on-device; reformat = total appdata loss | contract forbids any storage-formatting module in host.yml |
| ID | VM proof | PROD proof |
|---|---|---|
| STO-1 | Five loop/bind branch stand-ins + a stand-in union let the assert logic be exercised (all-present passes; drop one branch → fails); CI grep proves no destructive storage module/command anywhere under ansible/ |
Real mergerfs /mnt/user composed of the five XFS branches, parity confirmed outside the pool via the live branch list |
| STO-2 | CI grep (as above) proves absence of destructive modules | Real btrfs cache mounted as-is, untouched across converges |
BASE PACKAGES - role: base_packages
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| PKG-1 | hd-idle, jq, restic installed |
install.sh:16 |
Timers/scripts depend on them | apt present |
| PKG-2 | fail2ban installed |
install.sh:102 |
ssh brute-force protection | apt present |
| PKG-3 | smartmontools, btrfs-progs, curl installed |
scripts invoke smartctl (parity-smart-textfile.sh), btrfs (btrfs-scrub.sh), curl (watchdog/tautulli/healthchecks) - none installed by install.sh today |
A fresh-host timer installs fine, then fails at trigger time with a missing binary; installing smartmontools provides smartctl and is orthogonal to masking its service (JOB-5) |
apt present. snapraid/mergerfs stay manual storage prereqs (bound to the pool config in debian13-install.md), not owned here |
| PKG-4 | ansible-core present (manual apt prereq, documented) |
the behavior contract; live: Debian 13 trixie | Bootstrap chicken-egg; not self-installed | documented prereq + assert |
| ID | VM proof | PROD proof |
|---|---|---|
| PKG-1..3 | Converge on clean VM installs all of them; smartctl/btrfs/curl resolve on PATH after converge; second converge reports ok (idempotent); reboot → still present |
Same package set already present on host (converge is a no-op); smartmontools installed and its service masked (JOB-5) |
| PKG-4 | VM bootstrapped via apt install ansible-core per the documented prereq |
Host runs Debian 13 trixie; ansible-core from apt |
MAINTENANCE JOBS - role: maintenance_jobs
Six host systemd oneshot services + timers, their scripts, and the hd-idle
spindown config.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| JOB-1 | 6 units installed 0644 to /etc/systemd/system/: snapraid-sync, snapraid-scrub, homelab-watchdog, appdata-backup, btrfs-scrub, restic-deep-check (.service+.timer) |
install.sh:63-64; host/systemd/ |
Scheduled parity/backup/watchdog maintenance | copy mode 0644 + daemon_reload |
| JOB-2 | Every timer explicitly enabled --now |
install.sh:85-86; live: all enabled |
install.sh copies by glob but each new timer needs an explicit enable; a glob-copied-but-unenabled timer never fires |
systemd enabled+state=started per timer (never rely on a glob) |
| JOB-3 | scripts/*.sh executable |
install.sh:65 |
Oneshots exec them directly | file mode 0755 |
| JOB-4 | hd-idle.service active; /etc/default/hd-idle = parity-only spindown (-i 0 -a {{ homelab_parity_device }} -i 3600; the -a filter needs a full path or kernel name (a bare by-id name matches nothing, fixed 2026-07-14)) |
install.sh:19,85; host/systemd/hd-idle.default.j2; live |
Spinning down data disks adds read latency; only the read-idle parity disk should park (~5.6 W saved, zero read-latency cost) | copy default file 0644 + enable hd-idle |
| JOB-5 | smartmontools.service masked |
live (masked); CLAUDE.md |
Its DEVICESCAN default would wake the parked parity disk every poll, defeating JOB-4 |
systemd masked=true |
| JOB-6 | homelab-watchdog.timer runs every 5 min (OnBootSec=2min, OnCalendar=*:0/5, non-persistent) |
host/systemd/homelab-watchdog.timer |
mergerfs-mount / SAB-stuck / qbit-ionice watchdog | delivered by JOB-1/JOB-2 |
| ID | VM proof | PROD proof |
|---|---|---|
| JOB-1 | Files present at correct path + mode; systemctl cat matches repo |
Same on host |
| JOB-2 | After converge, systemctl is-enabled <t>.timer == enabled for all six; reboot → all still enabled; second converge idempotent |
Live audit: all six enabled (confirmed 2026-07-12) |
| JOB-3 | test -x scripts/*.sh on VM |
Same |
| JOB-4 | Default file content matches; hd-idle.service active. The parity by-id serial won't exist on the VM - assert the config file content, not spindown effect |
hd-idle active on host; parity disk actually parks (power draw ~43 W parked vs ~78 W all-spinning) |
| JOB-5 | systemctl is-enabled smartmontools == masked on VM (mask is safe/simulatable) |
Live: masked + inactive; parity stays parked (no SMART-poll wake) |
| JOB-6 | Timer schedule matches after converge | Watchdog firing every 5 min on host |
BACKUP - role: backup (restic)
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| BAK-1 | appdata-backup.service + restic-deep-check.service set Environment=RESTIC_CACHE_DIR=/var/cache/restic |
host/systemd/appdata-backup.service, restic-deep-check.service; live: dir exists root:root 0700 |
systemd oneshots have no $HOME; without an explicit cache dir restic re-pulls the whole repo index from B2 every run |
delivered via the unit files (JOB-1) + ensure /var/cache/restic dir |
| BAK-2 | Restic repos initialized (local /mnt/user/backups/appdata-restic + B2) only if .env provides RESTIC_B2_REPO+RESTIC_PASSWORD |
install.sh:26-33 |
First-run bootstrap; must be skippable without secrets | conditional; PROD-only (needs B2 creds); no_log: true |
| BAK-3 | Both units are Type=oneshot, Nice=15, IOSchedulingClass=idle, After=network-online.target |
host/systemd/*.service |
Backups must not starve foreground I/O | unit content (JOB-1) |
| ID | VM proof | PROD proof |
|---|---|---|
| BAK-1 | Unit files carry the Environment= line; /var/cache/restic exists 0700 root:root after converge |
Live: dir present; backups run cache-warm (no full B2 re-pull) |
| BAK-2 | With .env lacking restic keys → role skips init cleanly (no error) |
With real B2 creds → repos init/verify; escrowed .env |
| BAK-3 | Unit directives match repo | Nightly backup at 04:00 completes without I/O starvation |
MONITORING HOST - role: monitoring_host
Textfile collectors, their timers, Alertmanager secret files, and the numeric ownership of every monitored appdata dir. This is where UID/GID correctness matters most - a wrong owner means a silently empty dashboard.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| MON-1 | 5 collector units + timers installed and enabled --now: docker-healthcheck-textfile, docker-stats-textfile, parity-smart-textfile, reboot-required-textfile, tautulli-api-check |
install-host.sh:40-42; live: all enabled |
Feed node-exporter textfile metrics (cAdvisor doesn't work here) | copy 0644 + systemd enabled per timer |
| MON-2 | monitoring/scripts/*.sh executable; .prom outputs 0644 |
install-host.sh:39; collector scripts chmod 644 $TMP |
node-exporter runs as nobody; a non-644 .prom is unreadable → metric vanishes |
file +x; collectors self-chmod outputs |
| MON-3 | Numeric ownership of appdata data dirs (Docker creates missing binds root-owned → container can't write): prometheus + alertmanager/data = 65534:65534; grafana = 472:472; loki = 10001:10001 |
install-host.sh:34-36 |
Each container runs as a fixed non-root UID; wrong owner = container can't write its data dir | file/command: install -d -o.. -g.. idempotent |
| MON-4 | node-exporter/textfile dir exists (root:root, world-readable) |
install-host.sh:28; live 0755 root:root |
node-exporter mounts it to read .prom files |
ensure dir |
| MON-5 | Alertmanager secret files discord_webhook + healthchecks_url written 0600, owned 65534:65534, from monitoring/.env |
install-host.sh:19-25 |
Alertmanager (uid 65534) reads mounted secrets read-only; secret content from .env only |
copy/template with no_log: true, mode 0600, owner 65534 |
| MON-6 | parity-smart-textfile runs hourly and is spindown-safe (must not wake parked parity) |
monitoring/systemd/parity-smart-textfile.* |
A naive SMART poll would defeat JOB-4/JOB-5 | unit content (MON-1) |
| ID | VM proof | PROD proof |
|---|---|---|
| MON-1 | All 5 timers enabled after converge; reboot-persistent; idempotent |
Live: all 5 enabled (confirmed 2026-07-12) |
| MON-2 | Scripts +x; a converge-then-run produces 0644 .prom files (docker-stats collector needs a running container - use a throwaway container on the VM, or assert script+mode only) |
Live: all .prom files root:root 0644, node-exporter scraping them |
| MON-3 | After converge the four dirs have the exact numeric owners even when created fresh root-owned first; assert numeric UID/GID, not names (the VM has no grafana/nobody name mapping) |
Real Prometheus/Grafana/Loki/Alertmanager containers write their dirs without EACCES |
| MON-4 | Dir present, world-readable | node-exporter reads textfiles |
| MON-5 | PROD-adjacent: with a stub monitoring/.env on the VM the files render 0600/65534; real webhook/URL values are prod-only and no_log |
Real INFRA webhook + Healthchecks URL; Alertmanager delivers alerts |
| MON-6 | Timer schedule + collector logic match | parity stays parked; hourly SMART still reported via the spindown-safe path |
DOCKER HOST - role: docker_host
daemon.json, the boot-ordering drop-in, the WireGuard module, and the shared ingress network. Docker restart is the single most disruptive host op and is gated.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| DOCKER-1 | /etc/docker/daemon.json = json-file logging (max-size 50m, max-file 3) + live-restore: true, mode 0644 |
host/docker/daemon.json; live: live-restore enabled |
Log rotation caps disk; live-restore keeps containers up across engine upgrades | write canonicalized JSON; compare effective directives, not whitespace |
| DOCKER-2 | Docker restart happens only when daemon.json/drop-in effectively changed, AND is gated: allow_docker_restart=true required when Docker is active with running containers; validate candidate JSON before writing; the one live cutover passes the flag |
install.sh:36-70 (docker_changed guard); the behavior contract |
An unguarded restart mid-operation kills every container; whitespace-only diffs must not trigger it | parse+semantic-compare; conditional handler; fail-before-mutate if gated flag missing while containers run |
| DOCKER-3 | docker.service drop-in wait-mounts.conf (0644) with RequiresMountsFor=/mnt/user/data /mnt/cache/appdata |
host/systemd/docker-wait-mounts.conf; live: present |
Mounts are nofail; without this dockerd can start pre-mount and bind empty dirs into every container (fail-closed boot depends on it) |
copy drop-in + daemon_reload |
| DOCKER-4 | WireGuard kernel module loaded + /etc/modules-load.d/wireguard.conf (0644) |
install.sh:46-48; host/hardening/modules-wireguard.conf; live: loaded |
qBittorrent (binhex VPN) runs unprivileged and can't modprobe; the host must load it before the container starts |
modprobe + persist module-load file |
| DOCKER-5 | External docker network proxy exists |
install.sh:72-73 |
All three Compose projects declare it external; absent = compose up fails |
command/docker_network create-if-absent (ansible-core built-in preferred over community.docker) |
| DOCKER-6 | 51unattended-upgrades-docker apt drop-in (0644) scoping unattended-upgrades to the Docker CE origin |
install.sh:44; host/hardening/51unattended-upgrades-docker |
download.docker.com isn't a Debian security origin; without it the engine never auto-patches | copy drop-in |
| ID | VM proof | PROD proof |
|---|---|---|
| DOCKER-1 | File content + mode match; a re-ordered-but-equivalent daemon.json produces changed: false |
Live: Live Restore Enabled: true; log rotation active |
| DOCKER-2 | NEG: on a VM with Docker running a stub container, a real diff without allow_docker_restart fails before mutating; with the flag it restarts. Fresh VM (no daemon.json, no containers) converges with no flag needed. Whitespace-only edit → no restart |
The single cutover passes the flag with containers running and restarts once, cleanly |
| DOCKER-3 | Drop-in present; systemctl cat docker shows RequiresMountsFor. NEG (fail-closed): the VM fixture must define real .mount/fstab units for /mnt/user/data and /mnt/cache/appdata (otherwise systemd treats the already-mounted root as satisfying the dependency and Docker starts anyway). Make one mount unit fail → docker.service refuses to start; make both succeed → Docker starts. Test this separately from preflight, since PRE-1/PRE-2 would abort the play before Docker is reached |
Live: drop-in active; a genuinely dead data disk keeps docker down deliberately (fail-closed) |
| DOCKER-4 | lsmod shows wireguard after converge; module-load file present; reboot-persistent |
Live: module loaded; qbit VPN comes up |
| DOCKER-5 | docker network inspect proxy succeeds post-converge; idempotent |
proxy net present; all 3 projects attach |
| DOCKER-6 | apt drop-in present/parseable | Engine auto-patches from Docker CE origin |
HARDENING - role: hardening
Host-level hardening drop-ins not covered elsewhere.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| HARD-1 | /etc/udev/rules.d/50-sata-link-power.rules (0644) + all scsi_host*/link_power_management_policy = max_performance |
install.sh:55-60; host/hardening/50-sata-link-power.rules; live: all 8 hosts max_performance |
Debian's med_power_with_dipm default caused Exos hard resetting link I/O errors (not drives/cables) |
copy udev rule + udevadm control --reload + set each sysfs policy |
| HARD-2 | fail2ban sshd jail on the configured SSH port (/etc/fail2ban/jail.d/homelab.local 0644); fail2ban enabled + active |
install.sh:101-105; host/hardening/fail2ban-jail.local |
ssh brute-force protection on the non-default port | copy jail + enable/restart fail2ban |
| HARD-3 | homelab_user in the docker group |
install.sh:95-99 |
Non-root docker access for the admin | user groups append - unconditional; the user+group are guaranteed present by PRE-5, so this never silently skips |
| ID | VM proof | PROD proof |
|---|---|---|
| HARD-1 | udev rule file present; policy-set task is idempotent. VM has no real SATA scsi_hosts → the sysfs loop is a no-op assert (skip cleanly when no matching hosts), file content is the VM gate | Live: all 8 host*/link_power_management_policy == max_performance; journalctl -b \| grep -c 'hard resetting link' stays flat |
| HARD-2 | Jail file present; fail2ban active after converge; reboot-persistent |
fail2ban banning on the real internet-facing port |
| HARD-3 | With PRE-5 satisfied on the VM, membership converges; second converge idempotent (no skip path - absence is caught earlier by PRE-5) | Live: homelab_user in docker group |
SSH HARDENING - role: ssh_hardening (runs LAST)
Highest-risk role - a bad sshd config can lock everyone out. Ordered last so a failure leaves the rest of the converge intact.
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| SSH-1 | /etc/ssh/sshd_config.d/99-hardening.conf (0644): configured port, PermitRootLogin no, PasswordAuthentication no, KbdInteractiveAuthentication no, AuthenticationMethods publickey, X11Forwarding no |
install.sh:107-126; host/ssh/99-hardening.conf |
Key-only, no-root, non-default port | write drop-in |
| SSH-2 | Validate before reload: sshd -t on the candidate; on failure restore last-known-good drop-in (never leave no drop-in → Debian defaults port 22 + password auth); reload only on success |
install.sh:110-126 |
A deleted-on-failure drop-in reverts sshd to insecure defaults on next restart | validate candidate → reload-on-success handler; keep .lkg fallback |
| ID | VM proof | PROD proof |
|---|---|---|
| SSH-1 | Drop-in present with exact directives; sshd -T reflects them |
Real host still reachable on the configured port key-only after converge |
| SSH-2 | NEG: inject a syntactically invalid candidate → role runs sshd -t, refuses to reload, restores the prior valid drop-in, and the VM's sshd stays on the good config (verify via sshd -T) |
Cutover reloads sshd once; existing session survives; a fresh key-only login on the configured port confirms before closing the old session |
NETWORK ASSERT - role: network_assert (assert-only, no remediation)
OPNsense owns addressing/DNS/routing/IPv6. Ansible only asserts the runtime reality with precise failures. Re-verified live before encoding (both flagged as "do not trust the notes"):
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| NET-1 | No IPv6 default route present (no v6 egress) | live 2026-07-12: no ::/0 route (one residual global inet6 addr, but no default route) |
A blackholed v6 default route caused ~200 ms happy-eyeballs stalls (Seerr thumbnails); the fix removed v6 egress, OPNsense-side | assert ip -6 route show default empty; fail if a v6 default reappears |
| NET-2 | Host DNS resolver = homelab_router_address (router); systemd-resolved not in the path |
live 2026-07-12: resolv.conf nameserver homelab_router_address; resolvectl empty |
The Cloudflare DNS pin was reverted 2026-07-10 back to the router; encoding a stale resolver would misdirect all lookups | assert resolv.conf nameserver == homelab_router_address, fail otherwise; no rewrite |
| ID | VM proof | PROD proof |
|---|---|---|
| NET-1 | Assert-logic unit-testable: seed a fake ip -6 route with/without a default → role passes/fails. VM's own v6 topology is irrelevant (this is a prod assertion) |
Live: no v6 default route on the host (confirmed 2026-07-12) |
| NET-2 | Assert-logic testable against a stubbed resolver value | Live: nameserver homelab_router_address, no systemd-resolved (confirmed 2026-07-12) |
These two rows are the reason
network_assertexists as a role: prior notes on the IPv6 and DNS state were re-verified against the running host rather than trusted, per the behavior contract. Ansible asserts this runtime and never rewrites it: OPNsense remains the owner.
COMPOSE DEPLOY - playbooks/deploy.yml + role: compose_deploy
Not part of host convergence (host.yml): deploy.yml is the separately
invocable workload entrypoint from the behavior contract ownership split. Compose stays
authoritative for containers, image pins, mounts, healthchecks, and app
networks; the playbook is a thin, gated wrapper around the Compose CLI and
never manages container lifecycle through Ansible modules (no
community.docker). It reuses role: preflight unchanged, so PRE-1..PRE-5
gate a deploy exactly as they gate a converge.
Routine invocation (note: no sudo - DEP-1):
cd /opt/homelab && git pull
cd ansible && ansible-playbook -c local playbooks/deploy.yml
| ID | Desired state | Source | Why | Converge |
|---|---|---|---|---|
| DEP-1 | The three Compose projects converge via the Compose CLI, executed from each project directory in order: repo root (media stack), monitoring/, actual/; run non-root as the deploy operator homelab_user |
today's operator flow (docker compose pull && docker compose up -d per project); the behavior contract |
Running from the project dir preserves today's project names, relative bind mounts, and automatic .env resolution; SEC-1 keys .env to homelab_user precisely because the operator runs Compose non-root (HARD-3), so a root-run deploy would silently invert that ownership model |
command: docker compose up --detach with chdir: per project; play runs unprivileged |
| DEP-2 | Every project's Compose file must validate before any project mutates; one invalid project aborts the whole deploy with zero container changes | gate-before-mutation pattern (DOCKER-2, SSH-2) | A deploy that brings up project 1 and then dies on project 3's syntax error leaves the fleet half-converged with no signal which half | docker compose config --quiet across all projects first, then a single assert over all results, only then any up |
| DEP-3 | Preconditions asserted, never remediated: Docker daemon reachable, external proxy network present |
DOCKER-5 (host.yml owns creation); all three projects declare proxy external |
deploy.yml must not grow host-convergence side effects; a missing proxy network means host.yml never ran, and creating it here would let the two entrypoints co-own a resource |
assert docker version and docker network inspect proxy succeed, with a fail message pointing at host.yml (docker version, not docker info: docker info --format exits 0 with an unreachable daemon, so it cannot gate) |
| DEP-4 | Re-running deploy.yml on a converged fleet reports changed=0; no pending-work marker needed |
universal idempotence gate | docker compose up -d is itself the convergence: there are no handlers whose loss needs /run recovery (unlike DOCKER-2); a re-run is the recovery |
changed_when parses the Compose action lines (Created/Started/Recreated/Removed), so a no-op up is changed=0 |
| DEP-5 | Explicit image refresh only on request: -e compose_deploy_pull=true runs docker compose pull per project before up |
today's update flow (Renovate merge → pull → up) | Images are semver-pinned, so routine deploys need no pull; the flag mirrors the manual update flow without making every deploy hit registries | docker compose pull --quiet --ignore-buildable gated on the flag (--ignore-buildable: monitoring builds intel-gpu-exporter locally, and its local-only tag must not be pulled); changed when the local image set differs before/after |
| DEP-6 | A content change to a startup-only bind-mounted config file recreates its service on the next deploy | docker compose up -d ignores mounted-file content changes (only image/env/mount-definition changes trigger a recreate), so an edited config is read only on the next unrelated recreate; this masked the authelia idle-timeout fix for 3 days (committed 2026-07-16, live only after an unrelated 2026-07-19 recreate) |
stat (sha256) each file in compose_deploy_config_hashes, inject the digest as an env var (AUTHELIA_CONFIG_SHA), which the project's compose file references in a homelab.config-sha service label; a changed digest flips the label and Compose recreates just that service. Runs before validate/pull/up so one environment governs all three |
| ID | VM proof | PROD proof |
|---|---|---|
| DEP-1 | Stand-in Compose projects (pinned BusyBox, one per real project shape) converge via compose_deploy_projects override; containers run as compose-labeled projects; play runs unprivileged as homelab_user |
The three real projects converge with today's project names; container fleet state matches pre-cutover docker compose ps |
| DEP-2 | NEG: corrupt one stand-in project's Compose file → deploy aborts on the validation assert and no stand-in container from any project was created/recreated | Same gate; never expected to fire on a healthy repo (CI already docker compose configs all three projects) |
| DEP-3 | NEG: remove the proxy network → deploy aborts pointing at host.yml; stop Docker → deploy aborts on the daemon probe |
Live: proxy exists (DOCKER-5), daemon active |
| DEP-4 | Second deploy immediately after the first reports changed=0 |
Same on the real fleet |
| DEP-5 | Deploy with -e compose_deploy_pull=true succeeds and stays green with pinned stand-in images already present |
Post-Renovate deploy pulls exactly the bumped images |
| DEP-6 | Empty compose_deploy_config_hashes (VM has no real config file) leaves the overlay a no-op; deploy stays green |
Editing identity/authelia/configuration.yml then deploying recreates only authelia (changed=1); a second deploy is changed=0 |
deploy.yml also inherits the PRE-3 negative proof: with a secret .env
removed, the deploy aborts in preflight before touching Docker.
Proof strategy
Why two proof columns
The rehearsal target is a disposable Debian 13 VM. It faithfully reproduces
the convergence surface (packages, files, modes, systemd units, timers, module
loads, apt/udev drop-ins, docker daemon.json, the proxy network, sshd
validation). It cannot reproduce:
- the real mergerfs + SnapRAID pool over 6 physical XFS/btrfs disks,
- the real SATA HBA with 8
scsi_hostlinks (ALPM policy), - the parity disk's real by-id serial (hd-idle target),
- Backblaze B2 + the escrowed secret values,
- the real container fleet that reads the numeric-owned appdata dirs.
So every behavior is proven twice: the VM proof is the automatable gate that runs during rehearsal (and, for syntax/lint, in CI); the PROD proof is what the single scheduled cutover and ongoing operation confirm on the real host.
Negative tests (the fail-closed contract)
The most important safety behaviors are refusals, and a refusal can only be proven by creating the precondition it refuses. The VM proves these with stand-ins rather than the real pool:
- Mount fail-closed (PRE-1/PRE-2, DOCKER-3): a present-vs-absent loop/bind
mount stand-in at
/mnt/userand/mnt/cache. Absent or wrong-FSTYPE → preflight anddocker.servicemust fail. Present, correctly typed → converge proceeds. This substitutes for the real fail-closed pool the VM lacks. - Docker restart gate (DOCKER-2): a throwaway container makes Docker
"active with running containers"; a real daemon.json diff without
allow_docker_restartmust abort before mutating. - sshd validation (SSH-2): an intentionally invalid candidate must be
rejected by
sshd -tand the prior good drop-in restored.
CI runs --syntax-check + ansible-lint + the static safety-invariant greps
(no destructive storage modules or raw mkfs/wipefs/parted/sgdisk/dd
of=/dev//snapraid init command:/shell: forms anywhere under ansible/,
playbooks/ and roles/ alike). Full check-mode, double-converge
idempotence, reboot-persistence, and the negative tests run on the VM, not the
GitHub runner.
Idempotence & persistence (applies to every converging row)
Two universal proofs not repeated per-row:
- Idempotence: a second converge immediately after the first reports zero
changedfor that role. - Reboot persistence: after a VM reboot, all enabled timers/services/modules are still enabled/loaded and all files/modes intact.
Live audit snapshot
Read-only audit of the production host, 2026-07-12, used to reconcile this contract against reality. No mutations were made.
| Fact | Observed |
|---|---|
| OS | Debian GNU/Linux 13 (trixie) |
/mnt/user |
fuse.mergerfs ✓ |
/mnt/cache |
btrfs ✓ |
| IPv6 default route | none (no v6 egress); one residual global inet6 addr |
| DNS | nameserver homelab_router_address; systemd-resolved not in path |
| smartmontools.service | masked + inactive ✓ |
| hd-idle.service | active; parity-only opts as configured ✓ |
| SATA ALPM | all 8 scsi_host* = max_performance ✓ |
| wireguard module | loaded ✓ |
| docker wait-mounts drop-in | active (RequiresMountsFor=/mnt/user/data /mnt/cache/appdata) ✓ |
| docker live-restore | enabled ✓ |
/var/cache/restic |
present, root:root 0700 ✓ |
| Root timers enabled | all 6 (snapraid-sync/scrub, homelab-watchdog, appdata-backup, btrfs-scrub, restic-deep-check) ✓ |
| Monitoring timers enabled | all 5 (docker-healthcheck/stats-textfile, parity-smart-textfile, reboot-required-textfile, tautulli-api-check) ✓ |
Textfile .prom files |
all root:root 0644 under /mnt/cache/appdata/node-exporter/textfile ✓ |
Everything the notes claimed was confirmed against the live host; the two network rows (NET-1/NET-2) were the ones flagged for re-verification and both matched current reality.