Chapter 16 of 16 · Application/workload user · orders-api

Project Lockbox Readiness Drills

Assemble the design and prove it through rotation, revocation, outage, denial, cleanup, and recovery evidence.

Chapter contract

Persona: Application/workload user · orders-apiPhase: AWS and EKS workloadsEstimated time: 60 minutes

Prerequisites:

All previous chapters; Docker is optional for the runtime smoke but required for live lab evidence.

User-visible outcome:

The team can run Lockbox checks, record non-secret evidence, identify gaps, and clean the lab completely.

Learning objectives

Consumer flow

Identity proofstep 1Three lifecyclesstep 2Failure drillsstep 3Readiness decisionstep 4

Ownership boundary

OwnerConsumer-facing responsibility
You ownStates the consumer contract and proves reload and degraded behavior.
Platform team ownsOwns runtime identity, integration components, rollout, and operational health.
Vault ownsOwns server-side trust, policy, engines, audit, and Vault service health.

Core concepts

Consumer SLO

Application-visible outcome, not server uptime alone.

Evidence

A repeatable command and non-secret result.

Failure drill

A bounded experiment with abort, recovery, and cleanup.

Readiness gate

A decision requiring named owners and verified evidence.

Commands and configuration

cd learn-vault
python3 ../tools/check_course.py . ../index.html
./labs/run_all_checks.sh
git status --short

Secret-safety stop

Stop: A green happy-path demo is not readiness. Every secret type needs expiry, rotation, outage, revocation, telemetry, recovery, and ownership.

Failure drill

SymptomThe capstone check fails or leaves generated files.
Likely causesInterrupted lab, Docker unavailable, contract drift, cleanup gap, or risky content.
Inspect safelyUse named check, service status, file names and modes, state metadata, and git status.
Do not printAny .runtime content, service environments, credentials, tokens, or container inspection.

Break it / fix it

Break it: Run the capstone after interrupting a prior lab.

Show diagnosis and fix

Fix it: Use the named failing check, restore the expected service state, and run bounded cleanup before retry.

Escalate with time, endpoint, auth path, role, mount, namespace when relevant, status code, request ID, and sanitized error class. Never attach a credential or response body.

Try it

Bootstrap boundary

labs/bootstrap owns the disposable server, auth mounts, policies, roles, and engines; learner actions begin after setup.

Cost:

$0; local containers only

Starting state:

Run labs/bootstrap/setup.sh and source .runtime/learner.env unless the chapter lab says AppRole alone is sufficient.

User actions:

Run offline checks and, when Docker is available, the runtime smoke. Review each readiness requirement by owner.

cd learn-vault
./labs/run_all_checks.sh
python3 ../tools/check_course.py . ../index.html
test -z "$(git status --short -- . ':!.runtime')"
cd labs/bootstrap && ./cleanup.sh

Success criteria:

Complete every item in the Verify section using metadata-only evidence; no secret value appears in terminal output or tracked files.

Troubleshooting:

Use the Failure drill and Break it / fix it evidence fields. Stop before broadening policy, weakening identity or TLS checks, or copying secret-bearing diagnostics.

Verify

  1. Content, links, safety, contracts, and offline tests pass.
  2. Runtime smoke passes or explicitly reports Docker unavailable.
  3. Cleanup removes Compose state and runtime material without tracked changes.
Evidence rule: retain only status, policy names, TTLs, versions, timestamps, fingerprints, and error classes. Never retain secret values.

Cleanup

Run labs/bootstrap/cleanup.sh and confirm .runtime is absent. Never use a broad recursive cleanup target.

Project Lockbox increment

orders-api authenticates from runtime identity, consumes Agent-rendered KV, rotates DB pools and PKI, and exposes lifecycle metadata only.

Quick check

What is the readiness unit?

Identity, policy, lifecycle, delivery, reload, telemetry, failure, and ownership.

Can Vault uptime prove app health?

No. Leases, renders, reload, or policy can still fail.

What evidence is safe?

Status, times, versions, TTL margins, fingerprints, generations, error classes, and IDs.

When is the lab clean?

Compose resources and .runtime are gone and tracked files are unchanged.

Gotchas

Recap and next

Lockbox is ready when every credential has trusted identity, least privilege, bounded lifetime, verified reload, safe telemetry, recovery, and an owner.

Primary references

Continue →