Amazon ANS-C01: Container and Serverless Networking — Study Guide
Part of the AWS Advanced Networking Specialty ANS-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.
EKS networking (CNI, pod networking)
Pod networking in Amazon EKS is dominated by the Amazon VPC CNI plugin (amazon-vpc-cni-k8s) which gives each pod an IP address from the VPC and places pod traffic directly on the VPC network. That design delivers predictable VPC-level security control (security groups, NACLs) and low-latency routing but requires careful IP address and ENI capacity planning because the number of secondary IPv4 addresses per ENI and the number of ENIs per instance type are hardware-limited. The aws-node daemonset drives IP allocation and attach/detach operations; its ConfigMap is edited with kubectl to tune behavior (for example, set WARM_IP_TARGET, WARM_ENI_TARGET, or ENABLE_PREFIX_DELEGATION). Prefix delegation and pod ENI modes reduce per-node IP exhaustion by letting a node allocate whole /28 prefixes to an ENI (ENABLE_PREFIX_DELEGATION=true) or by assigning a dedicated ENI per pod (useful for high-security isolation).
Alternatives to the Amazon VPC CNI such as Cilium (eBPF) or Calico can provide different tradeoffs. Cilium can replace kube-proxy and implement high-performance L3/L4 forwarding using eBPF, enable transparent encryption between nodes (WireGuard or IPsec), and reduce node-level IP pressure by using overlay or masquerading approaches. With Cilium you still integrate with VPC routing for egress and ingress, but you avoid frequent ENI attach/detach operations; this matters at high pod churn rates. For very high connection counts and strict L7 behavior, you also consider kube-proxy mode (IPVS) tuning and node kernel settings: adjust conntrack_max, tcp_tw_recycle/tcp_tw_reuse, and ip_local_port_range, and expose these via kubelet or daemonset init scripts to avoid ephemeral port exhaustion under thousands of concurrent long-lived gRPC connections.
ECS networking modes and Lambda VPC integration
ECS task networking has three primary modes: bridge, host, and awsvpc. The awsvpc mode is the most comparable to Kubernetes pod networking because it attaches an ENI to each task (or task group) and assigns a private IP and security groups directly to the task. Configure awsvpc by specifying awsvpcConfiguration with subnets and securityGroups in RunTask or CreateService API calls. Fargate enforces awsvpc and therefore provides task-level network isolation and integrates with AWS Cloud Map for service discovery. Use awsvpc when you need security group-based traffic filtering per-task, or when you need to expose standard VPC routing and metrics.
Lambda functions that need VPC access are attached to the VPC via ENIs in the function’s configured subnets and security groups. These ENIs are created and managed by the Lambda control plane, but ENI provisioning can add cold-start latency and has historically limited rapid scale-up unless mitigated with provisioned concurrency or by using VPC endpoints (AWS PrivateLink) and carefully designed subnet architecture. When placing many Lambda functions in a VPC, ensure subnets have available IPs, use NAT Gateway or NAT instances for egress as required, and prefer VPC endpoints (com.amazonaws.* endpoints via AWS::EC2::VPCEndpoint) to avoid routing egress through the internet where possible. Monitor ENI attach/detach using CloudWatch Logs and VPC Flow Logs to observe scaling behavior and troubleshoot concurrency-related throttling.
App Mesh and service discovery
AWS App Mesh uses Envoy sidecars as the data plane and provides L3–L7 observability, traffic shaping, retries, and TLS origination/termination controls. Define meshes, virtual nodes, and virtual services via the App Mesh API (CreateMesh, CreateVirtualNode, CreateVirtualService) or the App Mesh controller for Kubernetes. App Mesh supports mTLS by configuring a VirtualNode listener’s TLS block with clientPolicy and a certificate authority, and you can integrate with AWS Certificate Manager (ACM) or SDS for certificate distribution. However, note that App Mesh sidecars terminate and re-encrypt traffic by design; if a requirement mandates that application traffic remain encrypted end-to-end between the client and the application pod (no decryption in the network proxy), you must ensure TLS is terminated only at the pod and avoid terminating it at the mesh ingress or load balancer.
Service discovery is commonly done with Kubernetes Services and CoreDNS for EKS, with Cloud Map (CreateService, RegisterInstance) for cross-platform, and with Route 53 private hosted zones for DNS-based lookup. AWS Cloud Map integrates directly with ECS and App Mesh, enabling SRV or A records and API-driven health checks. For dynamic environments where instances scale rapidly, combine short DNS TTLs and Cloud Map health checks to avoid stale resolution; if you need immediate consistency, use a service mesh control-plane API to fetch endpoints rather than relying on DNS caching.
Design patterns and trade-offs
When designing for large-scale gRPC over TLS with mTLS, you must decide where TLS terminates. Terminating at the load balancer (ALB) allows offloading certificates to ACM and simplifies certificate rotation, but breaks end-to-end encryption and cannot provide mutual TLS to backend pods unless the backend re-establishes TLS with the client certificate information forwarded. For true end-to-end mTLS where application endpoints authenticate clients directly, use an L4 passthrough gateway such as a Network Load Balancer and let the pod/application handle TLS/mTLS. Combine NLB target-type ip with the AWS Load Balancer Controller annotation service.beta.kubernetes.io/aws-load-balancer-target-type: “ip” to register pod IPs directly; this pattern scales well because the NLB is designed for millions of connections and supports long-lived TCP/gRPC sessions without terminating TLS.
For ingress and path-based routing with HTTPS termination, Application Load Balancer is more appropriate because it supports host/path rules, redirects, and integration with WAF. To preserve client IPs when ALB terminates TLS, rely on X-Forwarded-For headers; backend web servers must consume and log X-Forwarded-For, and you should enable the ALB access logs for verification. If you need the true client socket address at the server layer (for legacy software), use NLB with proxy protocol v2 and ensure backend services support proxy protocol.
Service connectivity across multiple AWS accounts and VPCs scales differently depending on the pattern. VPC peering is simple but N^2 in management; Transit Gateway centralizes routing and scales better for many VPCs with route table segregation; AWS PrivateLink (Interface VPC Endpoints) provides the most granular per-service, identity-aware access model because you expose an endpoint service via NLB and consumers create interface endpoints in their VPCs. For multi-account shared services with strict access control and scalable onboarding, prefer PrivateLink because it isolates routing (no route table changes in consumer VPCs) and uses security groups for fine-grained controls.
Common pitfalls and decision criteria
A recurring pitfall is assuming the same networking model fits all workloads. Stateful or long-lived connection workloads (gRPC, databases) favor L4 passthrough (NLB) with pod-level TLS termination or hostPort/hostNetwork patterns to avoid proxy-induced latency; HTTP microservices that need path-based routing, WAF, or WebSocket termination benefit from ALB and App Mesh features. Another mistake is not accounting for ENI/IP limits when scaling EKS nodes or ECS tasks in awsvpc mode: always consult the EC2 instance type ENI and IP-per-ENI table and use prefix delegation or Cilium overlays when you need high pod density.
Monitoring and debugging require multiple sources: VPC Flow Logs and ENI metrics to see traffic egress/ingress, CloudWatch metrics for AWS Load Balancers (ActiveFlowCount, ProcessedBytes), and application-level telemetry from Envoy/App Mesh or the AWS X-Ray agent. For Lambda and Fargate, remember that cold starts tied to ENI operations can be mitigated by provisioned concurrency or by re-architecting access patterns to use VPC endpoints and PrivateLink so functions do not need broad egress.
Practical Problem: Use-Case Scenario
Named company: Acme Payments Inc. Challenge: Acme Payments runs a gRPC service on Amazon EKS that must support thousands of concurrent TLS connections on TCP port 443, use mutual TLS (mTLS) so the client certificate is validated by the backend service, and allow the EKS cluster to autoscale via Cluster Autoscaler and HPA without breaking connectivity or requiring TLS termination in the load balancer.
Numbered approach:
- Deploy the service with pod-level TLS and mutual authentication implemented in the application or in a sidecar that does not terminate end-to-end TLS for external clients. Store server/client certificates in AWS Secrets Manager and mount via Kubernetes CSI secrets store or use a cert distribution mechanism that works with pod lifecycle.
- Use the AWS Load Balancer Controller to create a Network Load Balancer by annotating the Service (service.beta.kubernetes.io/aws-load-balancer-type: “nlb”) and set the target type to IP (service.beta.kubernetes.io/aws-load-balancer-target-type: “ip”), then create a TCP listener on port 443. This ensures the NLB performs L4 passthrough and does not terminate TLS.
- Configure the NLB target group with protocol TCP and register pod IPs dynamically (the Load Balancer Controller will call CreateTargetGroup and RegisterTargets). Ensure health checks are set to TCP or a custom TCP-based health probe at a short interval so targets are marked healthy quickly during autoscaling (aws elbv2 create-target-group –protocol TCP –port 443 –target-type ip; aws elbv2 create-listener –protocol TCP –port 443 …).
- Tune the Amazon VPC CNI to support high pod density and reduce ENI churn: enable prefix delegation if supported (set ENABLE_PREFIX_DELEGATION=true in the aws-node ConfigMap), configure WARM_IP_TARGET to maintain spare addresses, and monitor aws-node metrics (kube-system daemonset logs and CloudWatch custom metrics). If node-level IP limits are a concern, consider Cilium with eBPF for higher pod density and reduced ENI operations.
- Scale the cluster safely: ensure Cluster Autoscaler has proper node group tags and IAM permissions, set PodDisruptionBudgets, and verify target group health checks and NLB connection draining are configured to avoid dropping long-lived gRPC connections during scale-down.
- Secure certificate rotation and trust: automate cert rotation with ACM Private CA or Secrets Manager and ensure pods retrieve updated trust bundles without requiring NLB reconfiguration. Use Kubernetes readiness/liveness probes that reflect mTLS handshake readiness.
AWS rationale: A Network Load Balancer in IP target mode preserves TLS to the pod (true end-to-end encryption) and supports millions of persistent TCP connections, making it appropriate for thousands of concurrent gRPC sessions. Registering pod IPs directly avoids per-node hostPort or instance target registration complexity and works cleanly with Cluster Autoscaler/HPA because the AWS Load Balancer Controller will register and deregister pod IPs as pods scale. Tuning the VPC CNI or adopting an eBPF-based data plane prevents IP exhaustion and reduces ENI attach/detach latency, which is essential for fast autoscaling and high-connection workloads. Storing and delivering mTLS artifacts via Secrets Manager or a CSI provider makes certificate lifecycle manageable without touching the load balancer.
Practice these questions → · Timed practice on ExamRoll.io →
Pass the whole exam — not just this question
You found this answer. Get every verified question and explanation in one place, and save hours of prep. Free to start.
Pass your exam →