Freshness budget
Maximum accepted age or minimum remaining lifetime.
Chapter contract
Prerequisites:
Shared local lab running for the outage drill and Chapters 8-10 understood.
User-visible outcome:
orders-api follows a bounded fresh, stale, expired, and recovered lifecycle during a Vault outage.
Learning objectives
- Set freshness, retry, renewal, and stop-using thresholds.
- Reload atomically without dropping healthy traffic prematurely.
- Test Vault outage and recovery without weakening verification.
Consumer flow
Ownership boundary
| Owner | Consumer-facing responsibility |
|---|---|
| You own | States the consumer contract and proves reload and degraded behavior. |
| Platform team owns | Owns runtime identity, integration components, rollout, and operational health. |
| Vault owns | Owns server-side trust, policy, engines, audit, and Vault service health. |
Core concepts
Retry budget
Bounded attempts with backoff and jitter.
Stale grace
Deliberate continued use of still-valid material.
Fail closed
Stop sensitive work before using invalid material.
Commands and configuration
cd learn-vault/labs/failure-drill
python3 state_machine.py --self-test
./run-drill.sh
jq '{state,lease_remaining_seconds,render_age_seconds,reload_generation}' ../../.runtime/drill-status.jsonSecret-safety stop
Stop: Never make TLS verification optional as an outage workaround. Availability controls must not downgrade server identity validation.
Failure drill
| Symptom | Vault is unavailable during renewal. |
|---|---|
| Likely causes | Network, DNS, trust, Vault leader or storage event, rate limit, or client retry storm. |
| Inspect safely | Track last success, TTL remaining, attempt count, error class, circuit state, and request ID. |
| Do not print | Cached data, tokens, renewal payloads, renders, or exceptions containing headers. |
Break it / fix it
Break it: Pause Vault during the simulated renewal window.
Show diagnosis and fix
Fix it: Use bounded retry and still-valid material only until hard expiry; validate replacement before recovery.
Escalate with time, endpoint, auth path, role, mount, namespace when relevant, status code, request ID, and sanitized error class. Never attach a credential or response body.
Try it
Bootstrap boundary
labs/bootstrap owns the disposable server, auth mounts, policies, roles, and engines; learner actions begin after setup.
Cost:
$0; local containers only
Starting state:
Run labs/bootstrap/setup.sh and source .runtime/learner.env unless the chapter lab says AppRole alone is sufficient.
User actions:
Run the deterministic state-machine test, pause Vault, observe bounded stale behavior, resume it, and verify recovery.
cd learn-vault/labs/failure-drill
python3 state_machine.py --self-test
./run-drill.sh
jq -e '.transitions|index("STALE") and index("RECOVERED")' ../../.runtime/drill-result.json
./cleanup.shSuccess criteria:
Complete every item in the Verify section using metadata-only evidence; no secret value appears in terminal output or tracked files.
Troubleshooting:
Use the Failure drill and Break it / fix it evidence fields. Stop before broadening policy, weakening identity or TLS checks, or copying secret-bearing diagnostics.
Verify
- The model never uses material past hard expiry.
- Retries are bounded and jitter-ready.
- Recovery validates replacement material before healthy.
Evidence rule: retain only status, policy names, TTLs, versions, timestamps, fingerprints, and error classes. Never retain secret values.
Cleanup
The drill resumes Vault through a trap. Cleanup removes results and confirms shared service status.
Project Lockbox increment
Readiness requires auth validity, DB lease margin, cert margin, render freshness, and accepted reload generation.
Quick check
Stop on every read failure?
Not always; still-valid material may have bounded grace based on risk.
What never passes hard expiry?
Use of expired or invalid material.
Why jitter?
It prevents synchronized retry and renewal storms.
What makes recovery safe?
Acquire, validate, install, then declare healthy.
Gotchas
- Readiness flapping can amplify dependency outages.
- Caching without max age converts dynamic data into static data.
- A reload signal is not proof of acceptance.
Recap and next
Resilience is a timed state machine with a hard security boundary, not an infinite retry loop.