Learning objectives
- Place EBS CSI IAM and add-ons in Terraform/OpenTofu.
- Use encrypted gp3 and the existing workload KMS key.
- Explain Availability Zone binding and restore scheduling.
- Separate snapshots, backups, and high availability.
Schedule first, provision second
WaitForFirstConsumer delays provisioning until scheduling provides topology. Immediate binding can create a volume in an Availability Zone where the pod cannot run.
Storage is durable, not magically available
| Concern | Course choice | Why it matters |
|---|---|---|
| Volume type | gp3 | Capacity and baseline performance are not tied as tightly as gp2; tune IOPS/throughput only from measurements. |
| Encryption | encrypted: "true" plus workload KMS alias | Prevents accidental use of an account default key outside this workload boundary. |
| Binding | WaitForFirstConsumer | Scheduler and CSI agree on the Availability Zone. |
| Reclaim | Retain | Deleting a PVC does not immediately delete production data; cleanup becomes explicit. |
| Snapshot deletion | Retain | Deleting a Kubernetes snapshot object does not silently delete the AWS recovery point. |
- The EBS CSI node component is a DaemonSet on EC2. EBS volumes cannot mount to Fargate pods.
- The managed snapshot-controller add-on supplies the controller and CRDs; EBS CSI supplies snapshot capability.
- KMS grant permissions are scoped to the foundation workload key.
- A single EBS volume is Availability Zone bound. Replicas across zones, application replication, and tested failover provide HA.
Lab: inspect storage without applying it
# Terraform mocked plan
terraform -chdir=learn-terraform/capstone/infra/modules/eks test \
-filter=tests/eks.tftest.hcl
# OpenTofu mocked plan
tofu -chdir=learn-terraform/capstone/infra/modules/eks test \
-filter=tests/eks.tftest.hcl
# Static manifest checks
yamllint learn-terraform/capstone/gitops/platform/storage
rg 'gp3|encrypted|kmsKeyId|WaitForFirstConsumer|Retain' \
learn-terraform/capstone/gitops/platform/storageThese commands do not create a PVC, EBS volume, or snapshot. A real restore belongs at a separately approved runtime checkpoint.
Clean-target restore sequence
- Quiesce writes or use the database's application-aware backup procedure before taking the snapshot.
- Create a new
VolumeSnapshotreference and wait untilreadyToUse=true. - Create a new PVC whose
dataSourceis that snapshot. Never overwrite or mutate the live PVC. - Start a separate restore workload on the clean target and validate data before switching traffic.
- Keep the source volume and snapshot until acceptance checks and rollback time have passed.
Terraform replacement and state review
Before approving an EKS plan, inspect both resource actions and state addresses:
# Pick one CLI and its state
terraform -chdir=learn-terraform/capstone/infra/stacks/eks plan -out=eks.tfplan
terraform -chdir=learn-terraform/capstone/infra/stacks/eks show eks.tfplan
terraform -chdir=learn-terraform/capstone/infra/stacks/eks state list
tofu -chdir=learn-terraform/capstone/infra/stacks/eks plan -out=eks.tfplan
tofu -chdir=learn-terraform/capstone/infra/stacks/eks show eks.tfplan
tofu -chdir=learn-terraform/capstone/infra/stacks/eks state list| Plan signal | Review |
|---|---|
| EBS CSI role replacement | Existing controller credentials can stop refreshing. Confirm association ordering and rollback. |
| Add-on version update | Check EKS compatibility, CRDs, release notes, and node rollout impact. |
| KMS ARN change | New volumes use the new key; existing volumes are not transparently re-encrypted. |
| StorageClass edit | Not in Terraform state. Review GitOps diff; many StorageClass fields are immutable after creation. |
A Terraform or OpenTofu output can expose the KMS ARN, but marking it sensitive would only redact CLI display. Sensitive values still exist in state; key ARNs are identifiers, not key material.
Failure drill
nodeName; it bypasses scheduler logic required by WaitForFirstConsumer.Verify boundaries
- EBS CSI and snapshot-controller are EKS managed add-ons.
- The controller uses Pod Identity only on EC2 and the workload KMS key only.
- The default class is encrypted gp3 with expansion, Retain, and
WaitForFirstConsumer. - VolumeSnapshotClass uses
ebs.csi.aws.comand Retain. - The restore procedure always creates a clean target before traffic changes.
Cleanup
The chapter's tests leave no AWS resources. In an applied lab, list retained PVs, EBS volumes, and snapshots before deleting the cluster. Retain prevents accidental deletion, so it also requires a deliberate inventory and separately approved cleanup.
Quick check
Why is an EBS snapshot not a replica?
It is a recovery point, not a serving copy that automatically receives new writes or handles traffic.
Why restore to a new PVC?
It preserves the source and gives validation and rollback a stable boundary before traffic switches.
Can Cassandra use this StorageClass?
Later, yes, on its dedicated EC2 nodes. Cassandra replication and repair provide service-level resilience; EBS snapshots remain one recovery mechanism.
Recap and next
CSI add-ons and IAM live in Terraform/OpenTofu; topology and storage policy live in Argo. Encrypted volumes, retained snapshots, and a clean-target workflow make recovery reviewable.
Next: expose Envoy Gateway through one controller-owned private NLB without adding a second L7 proxy.