Name-Based Rotation Misses Variables Hidden in Layers

The architecture lesson here is about what happens when override semantics run into rotation. When a variable is defined in a group file and then shadowed in a host file, the host value wins. So if you rotate the group file, the host's copy stays old, and nothing tells you. No error, no warning, just a stale value sitting there.

Fingerprint, don't trust names

The sweep compared same-named vars across files by sha256. That is how "the leaked staging token is the live prod admin token" got discovered (value-identical across all five vaults), and how deliberate prod overrides were distinguished from accidental shadows (same name, deliberately different value). Names tell you where a variable lives; fingerprints tell you what it is.

Recreated secrets don't propagate

The sneakiest bug: when you `rm` and recreate a container secret, the quadlet unit itself doesn't change, so no restart handler fires. The running containers just keep using the old mounted secret. A metrics scrape failed on the stale token until someone manually restarted things. The fix connects secret tasks to `notify` the consumer's restart handler, with `skip_existing: true` normalized so notifications only fire when a secret is actually recreated. That keeps it idempotent and matches the rm-then-redeploy contract. Mount-point freshness is the consumer's lifecycle problem, not the secret store's.

Exercise masked fields through their test endpoint

An admin API masks stored passwords as `**********` on read, so you can't compare them directly. The workaround is clever: POST the mask itself to the `testSMTPConnection` endpoint. The server resolves the mask to the stored value, which means the test exercises the actual stored credential. When the read path masks, find the path that exercises the stored value rather than reveals it.

The residue layer

Scrubbing transcripts isn't a one-pass job. The write-ahead log re-embeds values into the session database as sessions close. Pass one removed 570, and 4 more turned up afterward. The WAL is a resting place most sweeps never think to check.

flowchart TB
    subgraph LAYERS[override layers]
        GV[group_vars] --> HV[host_vars shadow
wins silently]
    end
    subgraph STALE[staleness vectors]
        SECRET[secret recreated] -->|no notify| RUN[running container
keeps old mount]
        WAL[WAL re-embeds] --> LOGS[transcript residuals]
    end
    subgraph VERIFY[masked verification]
        MASK[POST mask → test endpoint
resolves stored value]
    end