Your Rotated Secret Still Lives Somewhere
The new secret was written and verified. But the rotation wasn't done, because the old value still had places to hide: shadowed copies in per-host files, a dead-looking key that was actually live in production, a running container still holding the stale mount, and four residuals in a session database's write-ahead log.
"Rotated" doesn't mean the new value exists; it means the old one has nowhere left to be.The write is the easy part
Writing a new credential is one operation. Completing a rotation means auditing everywhere the old one still lives: same-named vars in per-host files that override the group (rotation updated the group, the host kept the old secret, silently, with no error); dead keys that turn out to be live somewhere else; running processes still holding the old mount.
The sweep finds what the write leaves behind.Dead-looking names carry live weight
A datastore credential marked for removal turned out to be value-identical across all five environments. The "dead staging" token was the live production admin token. Names lie about liveness; only fingerprint comparison told the truth. You can't retire what you haven't fingerprinted.
flowchart TB
NEW[new value written] --> SWEEP[the sweep]
SWEEP --> SHADOW[host-var shadows:
different values]
SWEEP --> DEAD[dead keys:
live elsewhere?]
SWEEP --> MOUNT[running containers:
stale mounts]
SWEEP --> LOGS[transcript store:
WAL residuals]
SWEEP --> UNROT[never-rotated:
predates the incident]Exposure predates the incident
The biggest surprise was this: old session summaries still held values from before the leak that triggered the programme. The incident was the trigger, not the boundary. The exposure window reached backwards.
An incident discovers a leak; it doesn't define where that leak begins or ends.
Your shadowed secret.
The rotation updated the group file while a host file kept the old value, and this happened after the sweep that found it.