October 2026 | ~16 min read
The Line Item Nobody Checked
We were celebrating. Posts #7, #8, and #9 had taken our EKS compute costs from $14,200/month down to $3,800/month. Then finance called. Our overall AWS bill had only dropped by $6,000/month, not the $10,400 we expected. Where was the rest of the money going?
The answer was buried in a Cost Explorer category called "EC2-Other" — a catch-all that AWS uses for data transfer charges. We'd never drilled into it. When we did: $22,400/month. NAT Gateway alone was $9,400/month — our second-largest line item after EC2 compute, and nobody on the engineering team even knew it existed.
We spent four weeks fixing it. Data transfer dropped from $22,400/month to $8,300/month — a 63% reduction, $14,100/month in savings, $169,200 annualized. Here's everything we did.
The Numbers That Matter
┌──────────────────────────────────────────────────────────────────┐ │ DATA TRANSFER COST REDUCTION │ ├──────────────────────────┬──────────┬──────────┬─────────────────┤ │ Category │ Before │ After │ Savings │ ├──────────────────────────┼──────────┼──────────┼─────────────────┤ │ NAT Gateway │ $9,400 │ $4,100 │ -$5,300 (-56%) │ │ Cross-AZ traffic │ $5,200 │ $1,400 │ -$3,800 (-73%) │ │ Internet egress │ $4,800 │ $1,600 │ -$3,200 (-67%) │ │ S3 data transfer │ $3,000 │ $1,200 │ -$1,800 (-60%) │ ├──────────────────────────┼──────────┼──────────┼─────────────────┤ │ TOTAL │ $22,400 │ $8,300 │ -$14,100 (-63%) │ ├──────────────────────────┼──────────┼──────────┼─────────────────┤ │ New infra cost │ │ $180/mo │ VPC endpoints │ │ Net monthly savings │ │ │ $13,920 │ │ Annualized │ │ │ $169,200 │ └──────────────────────────┴──────────┴──────────┴─────────────────┘ Table of Contents
- Understanding Your Data Transfer Bill
- NAT Gateway — The Silent Budget Killer
- Cross-AZ Traffic — The Per-Gigabyte Tax
- Internet Egress — CloudFront as a Cost Optimizer
- S3 Data Transfer
- Monitoring Data Transfer Costs
- Results and ROI
- Lessons Learned
- Action Items Checklist
Understanding Your Data Transfer Bill
The first problem: finding data transfer costs at all. AWS doesn't group them in one place. They're spread across multiple line items, and the biggest ones hide under "EC2-Other" in Cost Explorer.
Here's what we found when we finally broke it down:
┌──────────────────────────────────────────────────────────────────┐ │ 5 TYPES OF DATA TRANSFER CHARGES │ ├────────────────────────┬──────────────┬───────────────────────────┤ │ Type │ Rate │ Where It Hides │ ├────────────────────────┼──────────────┼───────────────────────────┤ │ NAT Gateway processing │ $0.045/GB │ EC2-Other → NAT Gateway │ │ NAT Gateway hourly │ $0.045/hr/AZ │ EC2-Other → NAT Gateway │ │ Cross-AZ transfer │ $0.01/GB/dir │ EC2-Other → Data Transfer │ │ Internet egress │ $0.09/GB* │ EC2-Other → Data Transfer │ │ S3 transfer out │ $0.09/GB* │ S3 → Data Transfer Out │ │ Inter-region │ $0.02/GB │ EC2-Other → Data Transfer │ ├────────────────────────┴──────────────┴───────────────────────────┤ │ * First 10 TB/month. Tiered pricing after. │ └──────────────────────────────────────────────────────────────────┘ To actually see these numbers, you need to go beyond Cost Explorer's default view. We wrote a script that pulls data transfer costs from Cost and Usage Reports (CUR) via Athena:
#!/usr/bin/env python3 """ Pull data transfer cost breakdown from CUR via Athena. Requires: CUR export to S3, Athena table configured. """ import boto3 import time import csv import io ATHENA_DATABASE = "cur_database" ATHENA_TABLE = "cost_and_usage_report" ATHENA_OUTPUT = "s3://your-athena-results-bucket/data-transfer/" QUERY = """ SELECT line_item_usage_type, product_transfer_type, CASE WHEN line_item_usage_type LIKE '%NatGateway%' THEN 'NAT Gateway' WHEN line_item_usage_type LIKE '%DataTransfer-Regional%' OR line_item_usage_type LIKE '%IntraRegion%' THEN 'Cross-AZ' WHEN line_item_usage_type LIKE '%DataTransfer-Out%' THEN 'Internet Egress' WHEN line_item_usage_type LIKE '%InterRegion%' THEN 'Inter-Region' WHEN product_product_name = 'Amazon Simple Storage Service' AND line_item_usage_type LIKE '%DataTransfer%' THEN 'S3 Transfer' ELSE 'Other' END AS transfer_category, SUM(line_item_unblended_cost) AS total_cost, SUM(line_item_usage_amount) AS total_gb FROM {database}.{table} WHERE line_item_product_code IN ('AmazonEC2', 'AmazonS3') AND line_item_usage_type LIKE '%DataTransfer%' OR line_item_usage_type LIKE '%NatGateway%' AND month = '{month}' AND year = '{year}' GROUP BY 1, 2, CASE WHEN line_item_usage_type LIKE '%NatGateway%' THEN 'NAT Gateway' WHEN line_item_usage_type LIKE '%DataTransfer-Regional%' OR line_item_usage_type LIKE '%IntraRegion%' THEN 'Cross-AZ' WHEN line_item_usage_type LIKE '%DataTransfer-Out%' THEN 'Internet Egress' WHEN line_item_usage_type LIKE '%InterRegion%' THEN 'Inter-Region' WHEN product_product_name = 'Amazon Simple Storage Service' AND line_item_usage_type LIKE '%DataTransfer%' THEN 'S3 Transfer' ELSE 'Other' END ORDER BY total_cost DESC """.format( database=ATHENA_DATABASE, table=ATHENA_TABLE, month="09", year="2026", ) def run_query(): client = boto3.client("athena") response = client.start_query_execution( QueryString=QUERY, QueryExecutionContext={"Database": ATHENA_DATABASE}, ResultConfiguration={"OutputLocation": ATHENA_OUTPUT}, ) query_id = response["QueryExecutionId"] while True: result = client.get_query_execution(QueryExecutionId=query_id) state = result["QueryExecution"]["Status"]["State"] if state in ("SUCCEEDED", "FAILED", "CANCELLED"): break time.sleep(2) if state != "SUCCEEDED": raise RuntimeError(f"Query {state}: {result}") paginator = client.get_paginator("get_query_results") rows = [] for page in paginator.paginate(QueryExecutionId=query_id): for row in page["ResultSet"]["Rows"][1:]: # skip header rows.append([col.get("VarCharValue", "") for col in row["Data"]]) # Print summary by category categories = {} for row in rows: cat = row[2] cost = float(row[3]) if row[3] else 0 gb = float(row[4]) if row[4] else 0 if cat not in categories: categories[cat] = {"cost": 0, "gb": 0} categories[cat]["cost"] += cost categories[cat]["gb"] += gb print(f"\n{'Category':<20} {'Cost':>12} {'GB':>12}") print("─" * 48) total = 0 for cat, data in sorted(categories.items(), key=lambda x: -x[1]["cost"]): print(f"{cat:<20} ${data['cost']:>10,.2f} {data['gb']:>10,.1f}") total += data["cost"] print("─" * 48) print(f"{'TOTAL':<20} ${total:>10,.2f}") if __name__ == "__main__": run_query() When we ran this against September's bill, the output looked like this:
Category Cost GB ──────────────────────────────────────────────── NAT Gateway $9,413.28 209,184.0 Cross-AZ $5,218.40 260,920.0 Internet Egress $4,792.80 53,253.3 S3 Transfer $2,976.52 33,072.4 ──────────────────────────────────────────────── TOTAL $22,401.00 Why don't most teams notice? Because "EC2-Other" is a single line in Cost Explorer. If you're only looking at service-level breakdown (EC2, RDS, S3), data transfer hides inside EC2 and S3 totals. You have to filter by usage type to see it. Most teams don't — until the bill is thousands per month.
NAT Gateway — The Silent Budget Killer
NAT Gateway is the most expensive data transfer line item for most AWS accounts, and the least understood. Here's why.
Every private subnet in your VPC routes internet-bound traffic through a NAT Gateway. That means every API call to an AWS service (ECR, S3, CloudWatch, STS, SSM) from your EKS nodes goes through NAT — and you pay $0.045 per GB processed, on top of $0.045/hour per NAT Gateway per AZ.
Our bill:
NAT Gateway cost breakdown: Hourly charges: 3 AZs × $0.045/hr × 730 hrs/mo = $98.55/mo Processing: 209 TB × $0.045/GB = $9,405/mo ────────────────────────────────────────────────────────────── Total: $9,404/mo 209 TB per month through NAT Gateway. Where was it all going? We enabled VPC Flow Logs on the NAT Gateway ENI and analyzed the traffic:
NAT Gateway traffic breakdown (top destinations): S3 (ECR image layers + log archival) 89 TB 42% ECR API (image manifest pulls) 31 TB 15% CloudWatch Logs (log ingestion) 28 TB 13% CloudWatch Metrics (custom metrics) 19 TB 9% STS (IRSA token refresh) 14 TB 7% SSM (parameter store lookups) 11 TB 5% Other AWS APIs 17 TB 8% ────────────────────────────────────────────────────────── Total 209 TB 100% 89 TB to S3 alone — mostly ECR image layers (stored in S3 behind the scenes) and application log archival. Every container image pull, every log upload, every API call — all routing through NAT at $0.045/GB.
Fix 1: VPC Gateway Endpoints for S3 and DynamoDB
VPC gateway endpoints are free. They route traffic to S3 and DynamoDB directly through the AWS backbone instead of through NAT Gateway. There is no per-GB processing charge, no hourly charge, nothing. This should be the first thing you deploy in every VPC.
# modules/vpc-endpoints/gateway.tf # S3 Gateway Endpoint — free, saves NAT processing costs resource "aws_vpc_endpoint" "s3" { vpc_id = var.vpc_id service_name = "com.amazonaws.${var.region}.s3" vpc_endpoint_type = "Gateway" route_table_ids = var.private_route_table_ids tags = { Name = "${var.environment}-s3-gateway" Environment = var.environment Purpose = "cost-optimization" } } # DynamoDB Gateway Endpoint — also free resource "aws_vpc_endpoint" "dynamodb" { vpc_id = var.vpc_id service_name = "com.amazonaws.${var.region}.dynamodb" vpc_endpoint_type = "Gateway" route_table_ids = var.private_route_table_ids tags = { Name = "${var.environment}-dynamodb-gateway" Environment = var.environment Purpose = "cost-optimization" } } variable "vpc_id" { type = string } variable "region" { type = string default = "us-east-1" } variable "environment" { type = string default = "production" } variable "private_route_table_ids" { type = list(string) description = "Route table IDs for private subnets" } The key detail: you must associate the gateway endpoint with every private subnet's route table. If you miss one, traffic from that subnet still goes through NAT.
# Verify the endpoint is working — run from an EC2 instance in a private subnet # Before: traffic routes through NAT Gateway traceroute s3.us-east-1.amazonaws.com # After gateway endpoint: traffic routes through VPC endpoint (no NAT hop) traceroute s3.us-east-1.amazonaws.com Result: -$2,800/month. All S3 traffic (ECR image layers + log archival) now bypasses NAT entirely. This single change took 10 minutes to deploy and saved more than most of our individual EKS optimizations.
Fix 2: VPC Interface Endpoints for High-Traffic AWS Services
Gateway endpoints only work for S3 and DynamoDB. For everything else (ECR, CloudWatch, STS, SSM), you need interface endpoints. These cost $0.01/GB processed + $7.20/month per AZ — but if you're pushing more than 160 GB/month per service through NAT, the interface endpoint is cheaper.
Our decision framework:
Interface endpoint break-even analysis: NAT processing cost: $0.045/GB Interface endpoint cost: $0.01/GB + $7.20/AZ/month Break-even per AZ: $7.20 / ($0.045 - $0.01) = 205 GB/month With 3 AZs: $21.60/month fixed + $0.01/GB Rule: if a service pushes >205 GB/month through NAT, deploy an interface endpoint. Below that, NAT is cheaper. We deployed interface endpoints for four services:
# modules/vpc-endpoints/interface.tf locals { # Services where our NAT traffic exceeded the break-even threshold interface_endpoints = { ecr_api = { service = "com.amazonaws.${var.region}.ecr.api" name = "ecr-api" monthly_gb = 31000 # 31 TB — well above break-even } ecr_dkr = { service = "com.amazonaws.${var.region}.ecr.dkr" name = "ecr-dkr" monthly_gb = 31000 } logs = { service = "com.amazonaws.${var.region}.logs" name = "cloudwatch-logs" monthly_gb = 28000 # 28 TB } sts = { service = "com.amazonaws.${var.region}.sts" name = "sts" monthly_gb = 14000 # 14 TB } ssm = { service = "com.amazonaws.${var.region}.ssm" name = "ssm" monthly_gb = 11000 # 11 TB } } } resource "aws_vpc_endpoint" "interface" { for_each = local.interface_endpoints vpc_id = var.vpc_id service_name = each.value.service vpc_endpoint_type = "Interface" private_dns_enabled = true subnet_ids = var.private_subnet_ids security_group_ids = [aws_security_group.vpc_endpoints.id] tags = { Name = "${var.environment}-${each.value.name}" Environment = var.environment Purpose = "cost-optimization" MonthlyGB = each.value.monthly_gb } } resource "aws_security_group" "vpc_endpoints" { name_prefix = "${var.environment}-vpc-endpoints-" vpc_id = var.vpc_id ingress { from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = [var.vpc_cidr] description = "HTTPS from VPC" } tags = { Name = "${var.environment}-vpc-endpoints" Environment = var.environment } } variable "private_subnet_ids" { type = list(string) description = "Subnet IDs for interface endpoints" } variable "vpc_cidr" { type = string description = "VPC CIDR block for security group rules" } Cost of interface endpoints: 5 endpoints × 3 AZs × $7.20/month = $108/month fixed, plus $0.01/GB processing. Against $0.045/GB through NAT for the same traffic, the savings are massive.
Result: -$1,900/month (after subtracting endpoint costs).
Fix 3: ECR Pull-Through Cache
Every time a pod starts, it pulls its container image. If you're running 200+ pods across 3 AZs with Karpenter scaling nodes up and down, that's a lot of redundant image pulls. ECR pull-through cache stores upstream images (Docker Hub, public ECR, GitHub Container Registry) in your private ECR registry, so pulls stay local.
# Create pull-through cache rules for Docker Hub and public ECR aws ecr create-pull-through-cache-rule \ --ecr-repository-prefix docker-hub \ --upstream-registry-url registry-1.docker.io \ --region us-east-1 aws ecr create-pull-through-cache-rule \ --ecr-repository-prefix ecr-public \ --upstream-registry-url public.ecr.aws \ --region us-east-1 Update your Kubernetes deployments to use the cached path:
# Before: pulls from Docker Hub through NAT every time # image: nginx:1.25 # After: pulls from ECR cache (stays in VPC via endpoint) # image: <account-id>.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/nginx:1.25 We automated this with a mutating webhook that rewrites image references at admission time, so developers didn't need to change their manifests:
# ecr-cache-mutating-webhook.yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: ecr-pull-through-cache webhooks: - name: ecr-cache.k8s.io admissionReviewVersions: ["v1"] sideEffects: None clientConfig: service: name: ecr-cache-webhook namespace: kube-system path: /mutate rules: - operations: ["CREATE", "UPDATE"] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] namespaceSelector: matchExpressions: - key: ecr-cache operator: NotIn values: ["disabled"] Result: -$600/month. Image pulls dropped by 78% because most pulls now hit the local ECR cache instead of traversing NAT to Docker Hub.
NAT Gateway Summary
NAT Gateway optimization results: Before: $9,400/mo After gateway endpoints (S3/DDB): -$2,800 After interface endpoints: -$1,900 After ECR pull-through cache: -$600 ────────────────────────────────────────────── After: $4,100/mo (-56%) Cross-AZ Traffic — The Per-Gigabyte Tax
AWS charges $0.01/GB in each direction for traffic between Availability Zones — $0.02/GB round trip. This sounds negligible until you run microservices that make hundreds of thousands of cross-AZ calls per hour.
Our setup: 14 microservices running across 3 AZs, with Kubernetes service routing randomly selecting pods regardless of AZ. A request from a pod in us-east-1a to a pod in us-east-1b incurs a $0.02 round-trip charge. Multiply by millions of requests per day and our cross-AZ bill was $5,200/month.
Fix 1: Topology-Aware Routing in EKS
Kubernetes topology-aware routing (previously called topology hints) tells kube-proxy to prefer pods in the same AZ when routing service traffic. This keeps most service-to-service calls in-zone, avoiding cross-AZ charges.
# Enable topology-aware routing on high-traffic internal services apiVersion: v1 kind: Service metadata: name: payments-api namespace: payments annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: payments-api ports: - port: 8080 targetPort: 8080 --- apiVersion: v1 kind: Service metadata: name: user-service namespace: platform annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: user-service ports: - port: 8080 targetPort: 8080 --- apiVersion: v1 kind: Service metadata: name: inventory-service namespace: platform annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: inventory-service ports: - port: 8080 targetPort: 8080 Important caveat: topology-aware routing requires at least 3 ready endpoints per zone to activate. If a service has fewer replicas than zones, Kubernetes falls back to cluster-wide routing. We ensured every high-traffic service had at least 3 replicas per AZ:
# Ensure enough replicas for topology hints to activate apiVersion: apps/v1 kind: Deployment metadata: name: payments-api namespace: payments spec: replicas: 9 # 3 per AZ minimum template: spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: payments-api After enabling topology-aware routing on our top 8 services, 70% of service-to-service traffic stayed in-zone.
Fix 2: Single-AZ for Non-Critical Batch Workloads
Our data pipeline ran batch jobs across all 3 AZs by default. These jobs didn't need multi-AZ redundancy — they were retryable. We pinned them to a single AZ using a dedicated Karpenter NodePool:
# karpenter-nodepool-batch.yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: batch-single-az spec: template: spec: requirements: - key: topology.kubernetes.io/zone operator: In values: ["us-east-1a"] # Pin to single AZ - key: karpenter.sh/capacity-type operator: In values: ["spot"] -...
