Learning objectives
- Distinguish requests and limits, HPA/VPA, and node capacity.
- Use Karpenter only for application EC2 capacity.
- Trace interruption events and Terraform/OpenTofu graph edges.
- Diagnose CNI IP pressure before blaming CPU.
Three loops, three signals
HPA scales replicas; Karpenter observes unschedulable requests and supplies nodes. Neither can compensate for missing requests and limits.
| Mechanism | Changes | Course boundary |
|---|---|---|
| HPA | Replica count | Workload GitOps, later chapter. |
| VPA | Recommended or applied requests | Use recommendation mode first; avoid fighting HPA on the same CPU or memory signal. |
| Karpenter | Application EC2 nodes | Argo owns NodePool/EC2NodeClass; Terraform owns AWS identity and events. |
| Managed node groups | Fixed platform pools | The managed system and Cassandra node groups stay outside Karpenter. |
| Fargate | One matching pod's compute | Karpenter does not scale Fargate. |
AWS capacity details
- The Karpenter controller uses EKS Pod Identity on Linux EC2 system nodes. Pod Identity does not work on Fargate.
- A dedicated Terraform-owned instance profile and
EC2_LINUXaccess entry authorize only Karpenter-created nodes. - The default application NodePool is On-Demand-only. Spot remains an explicit Spot opt-in through a separate tainted capacity boundary so ordinary application pods cannot bypass the required toleration.
- Five EventBridge rules cover five event types: Spot warnings, rebalance recommendations, instance state changes, AWS Health events, and capacity-reservation interruptions. Those rules use two source values,
aws.ec2andaws.health. - The queue policy accepts messages only from this account's named rules and denies insecure transport.
CNI IP pressure is a separate capacity ceiling
Each EC2 node and Fargate pod needs VPC addresses. A subnet with no usable IPs can leave pods Pending while CPU and memory look empty. Inspect subnet free addresses, ENI limits, pod density, and VPC CNI prefix delegation before increasing node limits.
Lab: test the ownership graph
Run mocked plans only. Provider installation may require registry access; these commands never apply AWS resources.
# Terraform
terraform -chdir=learn-terraform/capstone/infra/modules/eks test \
-filter=tests/eks.tftest.hcl
terraform -chdir=learn-terraform/capstone/infra/modules/eks graph
# OpenTofu
tofu -chdir=learn-terraform/capstone/infra/modules/eks test \
-filter=tests/eks.tftest.hcl
tofu -chdir=learn-terraform/capstone/infra/modules/eks graphThe Terraform graph and OpenTofu graph infer edges from references: the queue policy depends on rule ARNs and the Karpenter role policy depends on the node role, instance profile, cluster, and queue. Use explicit depends_on only for behavior that references cannot express.
Review the plan as a capacity contract
| Plan object | Question | Reject when |
|---|---|---|
| Karpenter role | Are launch and termination actions tag scoped? | It can terminate arbitrary EC2 instances. |
| Interruption queue | Are encryption, retention, SourceArn, SourceAccount, and TLS denial visible? | The queue accepts arbitrary senders. |
| Node access entry | Is it EC2_LINUX for the dedicated role? | It grants administrator access. |
| Outputs | Do they expose names/ARNs needed by GitOps? | They export credentials or duplicate resources. |
An output adds a consumer contract, not an AWS object. Both tools derive output dependencies from its value expression; an explicit output dependency is a last resort.
Failure drill
Verify boundaries
- Karpenter NodePool labels only
workload=application. - System and Cassandra managed node groups remain fixed and On-Demand.
- Controller pods select and tolerate only EC2 system nodes.
- All five interruption rules target one encrypted queue.
- No Kubernetes or Helm provider exists in Terraform/OpenTofu.
Cleanup
Mocked tests and graph commands create no AWS resources. In a future applied environment, delete the Argo-owned NodePool first and wait for its NodeClaims to drain before removing the Terraform-owned Karpenter IAM, queue, or access entry.
Quick check
Why can Karpenter not fix a pod with no request?
Scheduling demand is based on requests. Without them, packing and scaling signals do not represent the pod's real need.
Should VPA and HPA both change CPU for the same workload?
Usually not without careful policy. VPA changing CPU requests changes HPA utilization math and can create feedback loops.
What survives if Karpenter is unavailable?
The fixed system and Cassandra node groups, existing application nodes, and the Fargate profile remain. New application EC2 capacity may not appear.
Recap and next
Requests describe demand, HPA changes pods, and Karpenter changes application EC2 supply. Terraform/OpenTofu provide the AWS identity and interruption graph; Argo owns cluster desired state.
Next: connect EBS CSI, encrypted gp3 storage, Availability Zone scheduling, snapshots, and clean-target restore.