Kubernetes on AWS (EKS)

The cheat table: AKS → EKS

EKS control plane

EKS is a managed Kubernetes control plane - same flavor as upstream Kubernetes, plus a few AWS-specific add-ons (the VPC CNI, kube-proxy, CoreDNS, EBS CSI). AWS runs the API server, scheduler, and etcd in their own account. You bring the worker nodes (or use Fargate). The boundary is the same as AKS conceptually; the bill is not.

What you pay for, and what you don't

Provision a cluster in Terraform

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "compass-aks"
  location            = "eastus"
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "compass"
  kubernetes_version  = "1.30"

  default_node_pool {
    name       = "system"
    node_count = 2
    vm_size    = "Standard_D2s_v5"
  }

  identity { type = "SystemAssigned" }

  network_profile {
    network_plugin = "azure"   # Azure CNI
  }
}

# Real-world: use the terraform-aws-modules/eks module.
# Bare resource shown for clarity.
resource "aws_eks_cluster" "eks" {
  name     = "compass-eks"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.30"

  vpc_config {
    subnet_ids = module.vpc.private_subnet_ids
    endpoint_private_access = true
    endpoint_public_access  = true
  }

  enabled_cluster_log_types = [
    "api", "audit", "authenticator",
  ]
}

resource "aws_eks_node_group" "ng" {
  cluster_name    = aws_eks_cluster.eks.name
  node_group_name = "default"
  node_role_arn   = aws_iam_role.nodes.arn
  subnet_ids      = module.vpc.private_subnet_ids
  instance_types  = ["m6i.large"]
  scaling_config { desired_size = 2  min_size = 2  max_size = 6 }
}

Node groups: managed, self-managed, Fargate

AKS has node pools. EKS has three patterns for "where do my pods run?" - and the trade-offs are unique enough that picking the wrong one bites for the lifetime of the cluster.

Picking between them

VPC CNI: pods get real IPs

The VPC CNI is the most-AWS-specific piece of EKS. AKS' Azure CNI does the same trick (real VNet IPs), so you may already be used to this style - but AWS' per-instance pod limit is much stricter, and the workarounds are unique enough to be worth understanding.

The pod-density ceiling

Every EC2 instance type has a fixed max ENIs and max IPs-per-ENI. Pod count per node = (max ENIs × max IPs-per-ENI) - 1 (the primary IP). Some real numbers:

Subnet sizing trap

Because every pod consumes a real VPC IP, the cluster's pod-network subnets need to be big. A common pattern from VNet thinking is /24 subnets (~250 usable IPs). Three /24 subnets across AZs gives you maybe 750 IPs - which is fine for a handful of nodes, but means your cluster maxes out at ~700 pods total before you're out of IPs.

Configure prefix delegation in Terraform

resource "aws_eks_addon" "vpc_cni" {
  cluster_name = aws_eks_cluster.eks.name
  addon_name   = "vpc-cni"
  addon_version = "v1.18.1-eksbuild.3"

  configuration_values = jsonencode({
    env = {
      ENABLE_PREFIX_DELEGATION = "true"
      WARM_PREFIX_TARGET       = "1"
    }
  })
}

IRSA and EKS Pod Identity

There are now two ways to give pods AWS credentials. The older path (IRSA) ships through OIDC and STS. The newer path (EKS Pod Identity, GA late 2023) uses a dedicated agent on the node. They achieve the same end state. For new EKS clusters, EKS Pod Identity is the preferred approach - it's simpler, doesn't need OIDC provider registration, and supports role chaining. IRSA is still the right tool for self-managed Kubernetes or any non-EKS cluster with OIDC.

The three pieces of IRSA

To get a pod talking to (say) DynamoDB via IRSA, you need:

1. An OIDC provider registered in IAM, pointed at the cluster's OIDC issuer URL.
2. An IAM role with a trust policy that allows sts:AssumeRoleWithWebIdentity for the specific ServiceAccount.
3. A Kubernetes ServiceAccount annotated with the role ARN.

IRSA in Terraform

# 1. Tell IAM about the cluster's OIDC issuer (once per cluster)
data "tls_certificate" "eks_oidc" {
  url = aws_eks_cluster.eks.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks_oidc.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.eks.identity[0].oidc[0].issuer
}

# 2. Role with trust policy keyed to ns/sa
resource "aws_iam_role" "gfn_reports" {
  name = "gfn-reports-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub" =
            "system:serviceaccount:gfn:gfn-reports"
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:aud" =
            "sts.amazonaws.com"
        }
      }
    }]
  })
}

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gfn-reports
  namespace: gfn
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/gfn-reports-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gfn-reports
  namespace: gfn
spec:
  replicas: 2
  selector: { matchLabels: { app: gfn-reports } }
  template:
    metadata: { labels: { app: gfn-reports } }
    spec:
      serviceAccountName: gfn-reports   # wired via annotation above
      containers:
        - name: app
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/gfn-reports:0.1
          env:
            - { name: AWS_REGION, value: us-east-1 }

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/ABCD1234"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.us-east-1.amazonaws.com/id/ABCD1234:sub": "system:serviceaccount:default:my-app"
      }
    }
  }]
}

EKS Pod Identity (the newer, simpler path)

In late 2023 AWS shipped EKS Pod Identity, which is functionally equivalent to IRSA but trades the OIDC dance for a per-cluster agent. You install the eks-pod-identity-agent add-on, then bind a ServiceAccount to a role via an aws_eks_pod_identity_association resource. No OIDC provider registration, no thumbprint, no trust-policy mini-DSL. Trust policy is just "Service": "pods.eks.amazonaws.com".

# Register OIDC, write trust policy
# with namespace+SA condition,
# annotate the ServiceAccount,
# deploy.

# Works on any k8s with OIDC.
# Cross-cluster reusable.
# Slightly more wiring.

resource "aws_eks_pod_identity_association" "gfn" {
  cluster_name    = aws_eks_cluster.eks.name
  namespace       = "gfn"
  service_account = "gfn-reports"
  role_arn        = aws_iam_role.gfn_reports.arn
}
# Trust policy is just:
#   Principal: { Service: "pods.eks.amazonaws.com" }
#   Action: sts:AssumeRole + sts:TagSession

AWS Load Balancer Controller

The AWS LB Controller is an add-on pod (not bundled with EKS by default) that turns Kubernetes Ingress and Service resources into ALBs (Application Load Balancers) or NLBs (Network Load Balancers). Annotations on the Ingress/Service drive almost every decision the controller makes - SSL cert, target type (instance vs IP), scheme (internal vs internet-facing), health-check paths.

Service vs Ingress, ALB vs NLB

An ALB Ingress with TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gfn-reports
  namespace: gfn
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip   # required for IRSA/Pod Identity pods
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123:certificate/abc-...
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  rules:
    - host: reports.compass.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gfn-reports
                port: { number: 8080 }

Installing the controller

The controller itself is a Helm chart. It needs an IAM role (IRSA or Pod Identity) with permissions to create/delete ALBs, target groups, listeners, and to read EC2 subnets/security-groups. AWS publishes the policy JSON; you attach it to a role that the controller's ServiceAccount can assume.

# Add the EKS chart repo
helm repo add eks https://aws.github.io/eks-charts
helm repo update

# Install (assumes you've already created the SA + IAM role)
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=compass-eks \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

EBS CSI, EFS CSI, and Karpenter

Two final EKS-specific bits an Azure dev will hit on day one: persistent volumes and dynamic node provisioning.

EBS CSI driver - block storage

The EBS CSI driver lets PersistentVolumeClaims (PVCs) provision Elastic Block Store volumes. It's the same shape as Azure CSI for Managed Disks. EKS ships it as a managed add-on - one Terraform line installs it.

resource "aws_eks_addon" "ebs_csi" {
  cluster_name             = aws_eks_cluster.eks.name
  addon_name               = "aws-ebs-csi-driver"
  service_account_role_arn = aws_iam_role.ebs_csi.arn   # IRSA
}

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  iops: "3000"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

EFS CSI - shared filesystems

EFS is AWS' managed NFS - the equivalent of Azure Files. Useful when you need a single volume mounted RWX (ReadWriteMany) across many pods. The EFS CSI driver is also a managed add-on. The PVC sizing field is ignored (EFS is elastic and pay-per-GB-stored, no preallocation).

Karpenter - dynamic node provisioning

Karpenter was open-sourced by AWS in 2021 and has largely replaced Cluster Autoscaler as the recommended scaler for EKS. The key win is consolidation: not just scaling up, but proactively scaling down by repacking. Cluster Autoscaler can only scale down a node group when its instance type matches; Karpenter rebalances across instance families.

# Per node pool: min/max + a chosen instance type
# Scale-up: pod pending → CA asks ASG for +1 node
#           (of the pool's instance type)
# Scale-down: node empty for 10m → CA drains it

# Pros: simple, predictable
# Cons: can't switch instance type;
#       can't consolidate across pools

# Per "NodePool" CRD: instance-type filters, zones,
#                     spot/on-demand mix, taints
# Scale-up: pod pending → Karpenter picks the
#           cheapest fit, calls RunInstances directly
# Scale-down: continuous consolidation - repacks
#             pods onto fewer/smaller nodes, terminates excess

# Pros: cheaper, faster, mixes instance types
# Cons: more moving parts, eats node IAM roles for breakfast

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "c"]            # general-purpose + compute-optimized
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]   # mix - Karpenter prefers Spot when safe
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s

Try it: read-only cluster inspection

aws eks list-clusters --region us-east-1
# {
#   "clusters": [ "compass-dev", "platform-prod" ]
# }

aws eks describe-cluster --name compass-dev --region us-east-1
# Look for:
#   .cluster.version          -- the k8s minor version
#   .cluster.identity.oidc.issuer -- the IRSA OIDC issuer URL
#   .cluster.endpoint         -- the API server URL (private/public)
#   .cluster.resourcesVpcConfig.endpointPublicAccess
#   .cluster.logging.clusterLogging -- which log types are enabled

aws eks update-kubeconfig \
  --name compass-dev \
  --region us-east-1 \
  --profile compass

# Verify the context
kubectl config current-context
# arn:aws:eks:us-east-1:123456789012:cluster/compass-dev

kubectl get nodes -o wide
# Note the INTERNAL-IP column - those are real VPC IPs

kubectl get pods -A -o wide | head -20
# Pod IPs come from the same VPC range as the nodes (VPC CNI)

kubectl get daemonset aws-node -n kube-system
# aws-node is the VPC CNI's per-node agent

kubectl describe configmap amazon-vpc-cni -n kube-system | grep -i prefix
# Tells you whether prefix delegation is on for this cluster

kubectl get sa -A -o json | jq -r '.items[]
  | select(.metadata.annotations["eks.amazonaws.com/role-arn"])
  | "\(.metadata.namespace)/\(.metadata.name) -> \(.metadata.annotations["eks.amazonaws.com/role-arn"])"'

Quick check

Gotchas for Azure devs

Project Compass: migrate the gfn-reports worker to EKS

Through chapters 4-9 the gfn-reports worker ran as a Lambda. The product team wants to enable longer-running aggregation jobs (over the 15-minute Lambda ceiling), keep stateful in-memory caches between invocations, and consolidate with three other backend services. EKS is the right tool. This slice provisions the cluster, sets up workload identity, and proves a pod can read DynamoDB via the same gfn-reports-role we built up over the last nine chapters.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "compass-eks"
  cluster_version = "1.30"

  cluster_endpoint_public_access = true

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnet_ids
  control_plane_subnet_ids = module.vpc.private_subnet_ids

  enable_irsa = true      # registers the OIDC provider for us

  cluster_addons = {
    coredns      = { most_recent = true }
    kube-proxy   = { most_recent = true }
    vpc-cni      = {
      most_recent = true
      configuration_values = jsonencode({
        env = { ENABLE_PREFIX_DELEGATION = "true" }
      })
    }
    eks-pod-identity-agent = { most_recent = true }
  }

  eks_managed_node_groups = {
    default = {
      desired_size   = 2
      min_size       = 2
      max_size       = 4
      instance_types = ["m6i.large"]
      # Enforce IMDSv2 with hop-limit 1 - blocks pod-from-IMDS attack
      metadata_options = {
        http_tokens                 = "required"
        http_put_response_hop_limit = 1
      }
    }
  }

  tags = { Project = "Compass", ManagedBy = "learn-aws" }
}

# gfn-reports-role already exists (chapter 2 onward).
# Through chapter 9 its trust policy allowed lambda.amazonaws.com.
# Now we add an IRSA trust statement so EKS pods can assume it too.

data "aws_iam_policy_document" "gfn_reports_trust" {
  statement {
    # Keep the Lambda principal during migration (remove later)
    actions    = ["sts:AssumeRole"]
    principals { type = "Service" identifiers = ["lambda.amazonaws.com"] }
  }

  statement {
    # Allow the EKS ServiceAccount via OIDC (IRSA)
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [module.eks.oidc_provider_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${module.eks.oidc_provider}:sub"
      values   = ["system:serviceaccount:gfn:gfn-reports"]
    }
    condition {
      test     = "StringEquals"
      variable = "${module.eks.oidc_provider}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "gfn_reports" {
  name               = "gfn-reports-role"
  assume_role_policy = data.aws_iam_policy_document.gfn_reports_trust.json
}

apiVersion: v1
kind: Namespace
metadata: { name: gfn }
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gfn-reports
  namespace: gfn
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/gfn-reports-role
---
apiVersion: batch/v1
kind: Job
metadata:
  name: gfn-reports-irsa-check
  namespace: gfn
spec:
  template:
    spec:
      serviceAccountName: gfn-reports
      restartPolicy: Never
      containers:
        - name: probe
          image: amazon/aws-cli:2.15
          env:
            - { name: AWS_REGION, value: us-east-1 }
          command: ["sh", "-c"]
          args:
            - |
              echo "==> who am I?"
              aws sts get-caller-identity
              echo "==> can I list the table?"
              aws dynamodb describe-table --table-name gfn-reports-aggregates

Recap & next

What stuck?

• EKS control plane: $0.10/hr per cluster, every cluster, forever. The single biggest cost surprise. Consolidate environments, don't spin up clusters casually.
• Node groups: managed for default, self-managed + Karpenter for cost optimization, Fargate only for bursty/untrusted workloads.
• VPC CNI: pods get real VPC IPs. Pod density per node is capped by ENI math. Turn on prefix delegation to lift the cap.
• IRSA / EKS Pod Identity: the two paths to give pods IAM permissions. EKS Pod Identity is the new default for in-EKS workloads; IRSA stays relevant for OIDC-based federation outside EKS.
• AWS LB Controller: turns Ingress objects into ALBs via annotations. certificate-arn drives HTTPS, target-type: ip is the standard.
• EBS CSI + Karpenter: persistent block storage via PVCs (use WaitForFirstConsumer); Karpenter beats Cluster Autoscaler on cost via continuous consolidation.

The mental model in one sentence

Common pitfalls so far