Verify Masked Fields and Scrub Shadows in Your Rotation Sweep

Field notes from the sweep phase of a rotation programme. The techniques:

1. Fingerprint the shadows

# list var NAMES from every vault file
# (names only — never values), then compare
# sha256 of same-named vars across files:
# same name + different hash = shadowed,
#   rotation must update each file
# same name + same hash    = shared value,
#   revocation splits by env

2. Verify masked fields by exercising them

# admin GET masks stored password as
# ********** — can't diff. but:
# POST testSMTPConnection {password: mask}
# → server resolves mask → stored value
# → the test exercises the REAL credential

3. Secret recreation needs a notify

- name: container secret
  podman_secret:
    name: "{{ svc }}_key"
    skip_existing: true      # notify fires only
                            # on real recreation
  notify: restart {{ svc }}  # stale mounts die
# state: started + handler = the silent gap
# unconditional restarted = already safe

4. Scrub the WAL too

# transcript residuals hide in the session
# DB's write-ahead log — closing sessions
# re-embed values. scrub once, then AGAIN
# after sessions close: pass1 ~570, pass2 4

5. Dead keys: fingerprint before rm

# record hash, check live refs, then rm —
# the 'dead' staging token was value-
# identical to live prod admin across all
# 5 vaults. names lie; fingerprints don't.

flowchart LR
    SWEEP[the sweep] --> SHADOW[group vs host
hash compare]
    SWEEP --> MOUNT[recreate + notify
restart handler]
    SWEEP --> DEAD[dead keys:
hash → refs → rm]
    SWEEP --> WAL[session DB
residuals]
    SWEEP --> MASK[masked field →
test endpoint]