Acceptance record: Ansible slice 2 (secrets, storage_assert, network_assert, backup, monitoring_host, homepage_config)

Status update (2026-07-13): the homepage_config role and HP-1 contract were retired after this rehearsal when Grafana replaced Homepage as the service launcher. The VM evidence below is retained as a historical record; HP-1 is no longer part of host.yml or the production cutover checklist.

This records the proofs for the second vertical slice of the Ansible host-convergence re-platform (the behavior contract). It implements six more of the twelve roles in the behavior contract: secrets (SEC-1), storage_assert (STO-1..2), network_assert (NET-1..2), backup (BAK-1..3), monitoring_host (MON-1..6), and homepage_config (HP-1). The remaining three roles (docker_host, hardening, ssh_hardening) stay skeleton stubs that fail "not implemented" and are delivered last by design.

Per the contract's proof strategy, every behavior carries two independent proofs. This file keeps them strictly separated:

• VM-simulation proofs - automatable on the disposable Debian 13 VM; green now (evidence below).
• PROD-validation proofs - provable only against the real host at a single scheduled cutover (real mergerfs/SnapRAID pool, real B2 credentials, real containers, real router). These are deferred and listed for the cutover checklist.

The VM cannot reproduce the real pool, real secrets, or real container UIDs, so the assert-only and secret-touching behaviors are proven against seeded stand-ins (five bind-mounted branches under a real mergerfs union, stub .env files, stub network probe inputs), never real disks or credentials. install.sh stayed frozen as the rollback and was not edited (retired 2026-07-14 once every proof closed).

Ordering fix (closes the slice-1 follow-up)

Slice 1 flagged that JOB-2 enables the Persistent backup timers with --now, which can catch-up-fire appdata-backup/restic-deep-check before the restic repos exist. host.yml now runs backup before maintenance_jobs, so the /var/cache/restic cache dir and the repos are initialised (or the init knowingly skipped when .env has no credentials) before any timer can fire. JOB-2 still owns all six timers; ownership is not split. This matches install.sh's original init-then-enable ordering. See the resolved note in acceptance-slice1.md.

Rehearsal harness extensions

ansible/tests/vm/ (ansible-native, no Molecule/galaxy per the behavior contract) grew:

• fixtures.sh: five bind-mounted branches at /mnt/disk1..5 composed into a real fuse.mergerfs union at /mnt/user (so storage_assert reads a real branch list from the mergerfs control-file xattr); branch-absent/ branch-restore for the STO-1 degraded-pool negative; env-loose (a 0644 stub .env) for the SEC-1 tighten proof; net-good/net-bad-v6/ net-bad-dns stubs for the network negatives; and MON-5 variables (INFRA_WEBHOOK/HEALTHCHECKS_URL) appended to the stub monitoring/.env. mkfs/mergerfs/mount live only here, never under the ansible/ role tree, so the destructive-storage guard stays green.
• vm-overrides.yml: points the network_assert probe inputs at the fixture stubs (the VM's own IPv6/DNS state cannot stand in for prod); the expected values stay contract-hardcoded. Never used on the real host.
• rehearse.sh: runs all nine implemented roles' tags in play order, threads the overrides through every play, and adds the slice-2 negatives, state checks, and the five monitoring timers to the timer/reboot checks.

Run it from ansible/tests/vm with ./rehearse.sh all (override the target with VM=); it refuses to run against the production host.

VM-simulation proofs - ALL PASSING

Full rehearse.sh all on a fresh Debian 13 (trixie) VM, systemd 257, ansible-core 2.19.4 (apt), local connection: 48 checks passed, 0 failed.

Negative tests (fail-closed contract) - all abort as required

Contract	Precondition seeded	Result
STO-1	one pool branch (/mnt/disk3) unmounted	converge aborts ("is not mounted") ✓
NET-1	a stubbed IPv6 default route present	converge aborts ("IPv6 default route") ✓
NET-2	stubbed resolver not homelab_router_address	converge aborts ("expected exactly ['homelab_router_address']") ✓

(The seven slice-1 PRE-1..5 negatives also still abort - full list in the run log.)

Positive convergence + idempotence

Proof	Evidence
First converge (all stand-ins good, all nine roles) succeeds	play returns rc 0
Idempotence - second converge	ok=41 changed=0 failed=0 skipped=2
Check-mode gate - --check on healthy host	play returns rc 0 (read-only probes carry check_mode: false)
--tags maintenance_jobs selects only that role	only JOB-* tasks run

Per-role state after converge

Contract	Evidence on VM
SEC-1	a 0644 stub .env is tightened to 0600, owner preserved as homelab_user (stat == 600:homelab_user); a pre-0600 file re-converges changed=0
STO-1	five branch mounts present; effective mergerfs branch list == the five branches, parity outside (proven via a real fuse.mergerfs union over the bind stand-ins)
STO-2	CI grep guard proves no destructive storage module/command anywhere under ansible/
NET-1/NET-2	assert logic driven against seeded stub inputs (empty/non-empty v6 route, homelab_router_address/wrong resolver) - passes and fails deterministically
BAK-1	/var/cache/restic present 0700 root:root
BAK-2	with a stub .env lacking restic keys the init skips cleanly ("skipping restic repo init"); no error
MON-3	the four data dirs own by exact numeric UID/GID (prometheus/alertmanager/data 65534:65534, grafana 472:472, loki 10001:10001) even when created fresh
MON-4	node-exporter/textfile present 0755 root:root
MON-5	both Alertmanager secret files render 0600 65534:65534 with the stub content; secret values never appear in facts/output (no_log)
MON-1	all five collector timers is-enabled == enabled after converge
HP-1	homepage/*.yaml copied 0644, dirs owned 99:100 (numeric)

Reboot persistence (no re-converge)

After systemctl reboot (verified by a changed /proc/sys/kernel/random/boot_id so a rejected reboot cannot yield a false pass), all six maintenance timers and all five monitoring timers are still enabled, and smartmontools.service is still masked. ✓

PROD-validation proofs - 2026-07-13 production cutover and 2026-07-14 next-day checks

Contract	PROD proof	Result (2026-07-13 cutover, 2026-07-14 next-day)
SEC-1	all three real escrowed .env files homelab_user:homelab_user 0600; homelab_user can still docker compose config/up (non-root Compose unbroken)	✅ perms confirmed; docker compose config -q clean for all three projects as homelab_user; deploy.yml ran without sudo
STO-1	real /mnt/user mergerfs composed of the five XFS branches; parity confirmed outside the pool via the live branch list	✅ /mnt/disk1..5 XFS branches live; /mnt/parity mounted separately outside the pool
STO-2	real btrfs cache mounted as-is, untouched across converges	✅ btrfs at /mnt/cache, unchanged through both converges
NET-1/NET-2	live host has no IPv6 default route and resolver homelab_router_address (re-verified 2026-07-12)	✅ zero IPv6 default routes; nameserver homelab_router_address
BAK-2	with real B2 creds the local + B2 repos init/verify; escrowed .env	✅ nightly appdata-backup.service finished successfully at 04:01 on cutover day against both repos
BAK-1/BAK-3	nightly backup runs cache-warm (no full B2 re-pull) and does not starve foreground I/O	✅ 2026-07-14 04:00 run: local snapshot 36 s, restic check clean, B2 pass 37 s, finished 04:01:46, exit 0, 3 min 7 s CPU - matches the warm-cache baseline, no B2 re-pull (BAK-1). Starvation (BAK-3): the whole run took 92 s wall-clock while the concurrent SnapRAID sync completed normally, and no Prometheus alert (beyond the Watchdog heartbeat) fired in the window
MON-3	real Prometheus/Grafana/Loki/Alertmanager containers write their data dirs without EACCES	✅ all monitoring containers up, Prometheus reports 35 targets up==1, metric freshness ~13 s
MON-5	real INFRA webhook + Healthchecks URL; Alertmanager delivers alerts	✅ secrets in place (secrets/ dir root-owned per role intent, files 0600 uid 65534); Alertmanager healthy; delivery path unchanged from pre-cutover, no test alert fired
MON-6	parity stays parked; hourly SMART still reported via the spindown-safe path	✅ 2026-07-14: with parity in standby the hourly collector's smartctl -n standby skipped without waking the disk (rc 2, status 0 emitted); see slice 1 JOB-4 for the hd-idle config fix; the unattended park was observed later the same morning (daemon parked the disk within its 3600 s idle timeout, collector emitted status 0 without waking it)

Out of scope for this slice

The three highest-risk roles (docker_host, hardening, ssh_hardening) remain stubs and are delivered in the final slice, ssh_hardening last so a bad sshd config cannot block the rest of the converge (the behavior contract).