Chapter 18: Envoy Gateway at the Edge

01Learning objectives

Trace private NLB to Envoy without an ALB.
Assign Gateway API resources to platform and application owners.
Keep Envoy control and data plane pods on EC2 system nodes.
Explain why CloudFront and WAF wait for later edge state.

02One L4 handoff, one L7 owner

The NLB forwards TCP to Envoy pod IPs. Envoy owns host, path, TLS, traffic splitting, retries, and rate limits. An ALB in front would duplicate Layer-7 policy and failure modes.

03Who owns each object?

Owner	Objects	Lifecycle signal
Terraform/OpenTofu	LBC Pod Identity IAM; internal NLB frontend security group; cluster/VPC outputs.	AWS plan and state.
Argo CD	LBC and Envoy Helm values; EnvoyProxy; GatewayClass; Gateway.	Git commit and Kubernetes status.
AWS Load Balancer Controller	NLB, listeners, target groups, backend security-group rules.	`Service` reconciliation and AWS tags.
Envoy Gateway	Managed Envoy Deployment, Service, and routing configuration.	Gateway API conditions.
Application teams, later	HTTPRoute and backend Service references in permitted namespaces.	Route Accepted/ResolvedRefs conditions.

loadBalancerClass: service.k8s.aws/nlb selects AWS Load Balancer Controller explicitly.
scheme: internal and IP targets make the private exposure deliberate.
A deterministic NLB name, tags, and Terraform-owned security-group name give the later edge state a stable discovery contract.
Both the Envoy Gateway control plane and managed Envoy Proxy select and tolerate only workload=system EC2 nodes.
Pod Identity is used for LBC because it runs on Linux EC2, not Fargate.

04Lab: render, never apply

Run from learn-terraform/capstone/gitops/platform. These commands render pinned charts to stdout and do not connect to the cluster:

helm template karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.14.1 --namespace karpenter \
  --values karpenter/values.yaml

helm template aws-load-balancer-controller eks/aws-load-balancer-controller \
  --version 1.15.0 --namespace kube-system \
  --values aws-load-balancer-controller/values.yaml

helm template envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
  --version v1.9.1 --namespace envoy-gateway-system \
  --values envoy/values.yaml

yamllint envoy/gateway.yaml karpenter storage \
  aws-load-balancer-controller/values.yaml

OCI and Helm repository downloads need network access. A failed download is not permission to use kubectl apply. Validate the pinned Gateway API 1.6.1 and Envoy Gateway 1.9.1 CRDs before any future Argo sync.

Gateway API policy placement

Need	API owner
TLS certificate reference	Gateway listener; secret lifecycle belongs to the certificate/GitOps owner.
Host/path and traffic splitting	HTTPRoute backendRefs with explicit weights.
Retries and timeouts	HTTPRoute or Envoy `BackendTrafficPolicy`, depending on supported fields.
Rate limits	Envoy `BackendTrafficPolicy`, reviewed separately from AWS WAF.

05Plan review: absence is part of the contract

# Terraform
terraform -chdir=learn-terraform/capstone/infra/stacks/eks plan -out=eks.tfplan
terraform -chdir=learn-terraform/capstone/infra/stacks/eks show eks.tfplan

# OpenTofu
tofu -chdir=learn-terraform/capstone/infra/stacks/eks plan -out=eks.tfplan
tofu -chdir=learn-terraform/capstone/infra/stacks/eks show eks.tfplan

The plan should contain the LBC IAM role/Pod Identity association and Envoy NLB security group. It must not contain aws_lb, an ALB, Kubernetes resources, Helm releases, CloudFront, or WAF.

Why CloudFront and WAF wait: the private NLB does not exist until Argo and both controllers are healthy. A later edge Terraform state discovers the controller-owned NLB by deterministic name/tags, then builds the VPC origin and WAF-protected distribution. This forward dependency avoids a circular state read.

!Failure drill

The Gateway has no address. Read Gateway and Service conditions, then LBC events. Verify loadBalancerClass, private subnet tags, Pod Identity, exact VPC ID injection, frontend security-group name, and backend rule management. Do not add an ALB or switch the NLB to internet-facing just to obtain an address.

06Verify boundaries

Exactly one GatewayClass names the Envoy controller.
Exactly one EnvoyProxy config creates one internal NLB Service.
Envoy pods run only on dedicated EC2 system nodes.
The NLB uses IP targets, the deterministic security group, and edge-discovery tags.
There is no Ingress object, ALB annotation, aws_lb, CloudFront, or WAF resource.
HTTPRoute, TLS, retries, traffic splitting, and rate limits remain GitOps-owned follow-on objects.

07Cleanup

Rendering and linting create no cluster or AWS resources. In a future applied environment, remove the Argo-owned Gateway and wait for the controller to delete the NLB and targets before removing the Terraform-owned LBC role or frontend security group. Confirm tagged AWS resources are gone.

08Quick check

Why not put an ALB before Envoy?

Both would make HTTP routing decisions. One single L7 owner keeps TLS, route, retry, and policy behavior observable in one place.

Why does Terraform not create the NLB?

The LoadBalancer Service is Argo-owned and AWS LBC reconciles its AWS resources. Terraform owns only prerequisites and later discovers the stable result.

Where should an application traffic split live?

In its GitOps-owned HTTPRoute backendRefs, with Envoy Gateway reconciling it. It does not belong in Terraform state.

09Recap

Terraform/OpenTofu build AWS prerequisites. Argo supplies Gateway desired state. AWS LBC owns one private NLB, and Envoy remains the single L7 owner. CloudFront and WAF wait for a later state with a real NLB to discover.

Phase 4 is complete. Chapter 19 will bootstrap Argo CD and formalize the GitOps application boundary.

01Learning objectives

02One L4 handoff, one L7 owner

03Who owns each object?

04Lab: render, never apply

Gateway API policy placement

05Plan review: absence is part of the contract

!Failure drill

06Verify boundaries

07Cleanup

08Quick check

09Recap

10Source notes