Your AWS bill has a line item called "EC2-Other". Nobody looks at it. On a mid-sized Kubernetes or OpenShift platform it routinely accounts for 20–30% of total spend, and most of it is two things: NAT Gateway data processing and inter-Availability-Zone data transfer.

Unlike EC2 or RDS, these charges have no tag, no resource owner, and no obvious dashboard. They are the byproduct of architectural decisions made months ago — a pod scheduled in eu-west-1c talking to a Redis primary in eu-west-1a, a CI job pulling 400 GB of container images through a NAT Gateway, an S3 client that never got a Gateway Endpoint.

This article shows how to isolate these costs with Cost Explorer, quantify them precisely with VPC Flow Logs and Athena, and then kill them with VPC endpoints and zone-aware routing.

Context: why these costs are invisible

AWS charges network traffic in three independent ways that stack on top of each other:

Charge Typical price (eu-west-1) Billed on
NAT Gateway hourly ~$0.045/h per NAT GW Per AZ, per NAT
NAT Gateway data processing ~$0.045/GB Every byte in and out
Inter-AZ data transfer $0.01/GB in + $0.01/GB out Both sides charged
Internet egress ~$0.09/GB (first 10 TB) Outbound only

Two properties make this dangerous:

  1. Double billing on inter-AZ. A 1 GB transfer between two AZs costs $0.02, not $0.01. Both the sender and the receiver are charged.
  2. NAT data processing applies to private AWS traffic too. A pod in a private subnet calling s3.eu-west-1.amazonaws.com without a Gateway Endpoint pays $0.045/GB plus the NAT hourly cost — for traffic that never leaves the AWS backbone.

On a Kubernetes cluster spread across three AZs with a default scheduler and no topology constraints, roughly 2/3 of all pod-to-pod traffic crosses an AZ boundary. A service mesh with mTLS sidecars, a Kafka cluster, or a chatty microservice architecture turns that into real money.

Concrete example: a platform pushing 30 TB/month through NAT and 50 TB/month of cross-AZ service traffic.

  • NAT processing: 30,000 GB × $0.045 = $1,350/month
  • NAT hourly (3 AZs): 3 × 730 × $0.045 = $98/month
  • Inter-AZ: 50,000 GB × $0.02 = $1,000/month

That is ~$29k/year for traffic nobody asked for.

Step 1: Isolate the cost in Cost Explorer

Start with the top-down view. The relevant usage types are NatGateway-Bytes, NatGateway-Hours, and DataTransfer-Regional-Bytes.

# NAT Gateway cost breakdown, last 3 months, grouped by usage type aws ce get-cost-and-usage \ --time-period Start=2024-03-01,End=2024-06-01 \ --granularity MONTHLY \ --metrics "UnblendedCost" "UsageQuantity" \ --group-by Type=DIMENSION,Key=USAGE_TYPE \ --filter '{ "Dimensions": { "Key": "USAGE_TYPE_GROUP", "Values": ["EC2: NAT Gateway - Data Processed", "EC2: NAT Gateway - Running Hours"] } }' \ --output table

For inter-AZ transfer, filter on the regional data transfer usage type (the exact prefix depends on your region code):

aws ce get-cost-and-usage \ --time-period Start=2024-05-01,End=2024-06-01 \ --granularity MONTHLY \ --metrics "UnblendedCost" "UsageQuantity" \ --group-by Type=DIMENSION,Key=USAGE_TYPE \ --filter '{ "Dimensions": { "Key": "USAGE_TYPE", "Values": ["EUW1-DataTransfer-Regional-Bytes"] } }'

This tells you how much. It does not tell you who. For that you need flow logs.

Step 2: Quantify with VPC Flow Logs + Athena

Enable flow logs with a custom format that includes the fields you actually need for cost attribution: pkt-srcaddr/pkt-dstaddr (the real endpoints behind a NAT), az-id, and flow-direction.

VPC_ID=vpc-0a1b2c3d4e5f67890 BUCKET=s3://my-flowlogs-bucket/vpc-flow-logs/ aws ec2 create-flow-logs \ --resource-type VPC \ --resource-ids $VPC_ID \ --traffic-type ALL \ --log-destination-type s3 \ --log-destination $BUCKET \ --max-aggregation-interval 60 \ --destination-options FileFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=true \ --log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${az-id} ${flow-direction} ${pkt-srcaddr} ${pkt-dstaddr} ${traffic-path}'

Use Parquet + Hive-compatible partitions. On a busy cluster, text-format flow logs will cost you more in Athena scans than the traffic you are trying to optimize.

Create the Athena table:

CREATE EXTERNAL TABLE IF NOT EXISTS vpc_flow_logs ( version int, account_id string, interface_id string, srcaddr string, dstaddr string, srcport int, dstport int, protocol bigint, packets bigint, bytes bigint, start bigint, `end` bigint, action string, log_status string, vpc_id string, subnet_id string, instance_id string, az_id string, flow_direction string, pkt_srcaddr string, pkt_dstaddr string, traffic_path int ) PARTITIONED BY ( aws_account_id string, aws_service string, aws_region string, year string, month string, day string, hour string ) STORED AS PARQUET LOCATION 's3://my-flowlogs-bucket/vpc-flow-logs/AWSLogs/' TBLPROPERTIES ('projection.enabled'='true', 'projection.year.type'='integer','projection.year.range'='2024,2030', 'projection.month.type'='integer','projection.month.range'='1,12','projection.month.digits'='2', 'projection.day.type'='integer','projection.day.range'='1,31','projection.day.digits'='2', 'projection.hour.type'='integer','projection.hour.range'='0,23','projection.hour.digits'='2', 'projection.aws_account_id.type'='injected', 'projection.aws_service.type'='injected', 'projection.aws_region.type'='injected', 'storage.location.template'='s3://my-flowlogs-bucket/vpc-flow-logs/AWSLogs/${aws_account_id}/${aws_service}/${aws_region}/${year}/${month}/${day}/${hour}' );

Query A: top NAT Gateway consumers

traffic_path = 2 marks traffic routed through a NAT Gateway. Group by the real destination to find out what is being pulled.

SELECT srcaddr AS pod_or_node_ip, pkt_dstaddr AS real_destination, SUM(bytes) / 1024.0 / 1024 / 1024 AS gb, ROUND(SUM(bytes) / 1024.0 / 1024 / 1024 * 0.045, 2) AS nat_cost_usd FROM vpc_flow_logs WHERE year='2024' AND month='05' AND traffic_path = 2 AND flow_direction = 'egress' GROUP BY 1, 2 ORDER BY gb DESC LIMIT 30;

Typical findings: *.amazonaws.com IP ranges (S3, ECR, STS, CloudWatch) at the top. Every one of those is a VPC endpoint away from being free or near-free.

Query B: inter-AZ traffic between private IPs 

WITH enriched AS ( SELECT f.az_id AS src_az, f.pkt_dstaddr, f.bytes, f.srcaddr FROM vpc_flow_logs f WHERE year='2024' AND month='05' AND flow_direction = 'egress' AND regexp_like(pkt_dstaddr, '^10\.') -- stay inside the VPC CIDR ) SELECT src_az, SUBSTR(pkt_dstaddr, 1, 7) AS dst_prefix, SUM(bytes) / 1024.0 / 1024 / 1024 AS gb, ROUND(SUM(bytes) / 1024.0 / 1024 / 1024 * 0.02, 2) AS interaz_cost_usd FROM enriched GROUP BY 1, 2 ORDER BY gb DESC LIMIT 50;

To map IPs back to workloads, join against a dump of your pod inventory:

kubectl get pods -A \ -o jsonpath='{range .items[*]}{.status.podIP}{","}{.metadata.namespace}{","}{.metadata.name}{","}{.spec.nodeName}{"\n"}{end}' \ > pod-inventory.csv # Upload to S3 and create an Athena table, or just grep locally for the top talkers grep -F "10.0.34.112" pod-inventory.csv

Step 3: Kill NAT costs with VPC endpoints

Gateway endpoints (free)

S3 and DynamoDB Gateway Endpoints cost nothing — no hourly fee, no per-GB fee. There is no reason not to have them. For a cluster pulling images from ECR, the S3 endpoint alone often removes 40–60% of NAT traffic (ECR image layers live in S3).

aws ec2 create-vpc-endpoint \ --vpc-id vpc-0a1b2c3d4e5f67890 \ --service-name com.amazonaws.eu-west-1.s3 \ --vpc-endpoint-type Gateway \ --route-table-ids rtb-0private1 rtb-0private2 rtb-0private3

Interface endpoints (priced, but usually cheaper)

Interface endpoints cost ~$0.011/h per AZ + ~$0.01/GB. Compare against $0.045/GB through NAT: break-even is reached quickly, but create the endpoint in every AZ where you have workloads, otherwise you trade NAT cost for inter-AZ cost.

for SVC in ecr.api ecr.dkr sts logs monitoring elasticloadbalancing ec2 kms secretsmanager; do aws ec2 create-vpc-endpoint \ --vpc-id vpc-0a1b2c3d4e5f67890 \ --service-name com.amazonaws.eu-west-1.$SVC \ --vpc-endpoint-type Interface \ --subnet-ids subnet-0priv-a subnet-0priv-b subnet-0priv-c \ --security-group-ids sg-0vpce \ --private-dns-enabled \ --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=cost-center,Value=platform},{Key=managed-by,Value=terraform}]' done

Verify from inside the cluster that DNS now resolves to a private IP:

kubectl run netshoot --rm -it --restart=Never --image=nicolaka/netshoot -- \ dig +short ecr.dkr.eu-west-1.amazonaws.com # Expect 10.x.x.x, not a public IP

Step 4: Make the cluster zone-aware

Topology-aware routing for Services

Kubernetes 1.27+ (OpenShift 4.14+) supports trafficDistribution / topology-aware hints. When enabled, kube-proxy prefers endpoints in the same zone as the client.

apiVersion: v1 kind: Service metadata: name: payments-api namespace: billing annotations: # Legacy mechanism, still honoured on older clusters service.kubernetes.io/topology-mode: Auto spec: # Kubernetes >= 1.31 / OpenShift 4.18 — preferred field trafficDistribution: PreferClose selector: app: payments-api ports: - name: http port: 8080 targetPort: 8080

Check that hints are actually populated — if there are not enough endpoints per zone, the control plane silently disables them:

kubectl get endpointslice -n billing -l kubernetes.io/service-name=payments-api -o yaml | grep -A4 hints

Spread workloads so hints can work

Topology-aware routing only helps if every zone has endpoints. Enforce it:

apiVersion: apps/v1 kind: Deployment metadata: name: payments-api namespace: billing spec: replicas: 6 selector: matchLabels: app: payments-api template: metadata: labels: app: payments-api spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: payments-api - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: payments-api containers: - name: api image: registry.example.com/billing/payments-api:1.8.3 ports: - containerPort: 8080 resources: requests: cpu: 200m memory: 256Mi limits: memory: 512Mi

Keep stateful traffic in one AZ

For a cache or a non-replicated datastore, cross-AZ reads are pure waste. Pin consumers to the data:

apiVersion: apps/v1 kind: Deployment metadata: name: session-worker namespace: billing spec: replicas: 3 selector: matchLabels: app: session-worker template: metadata: labels: app: session-worker spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: redis-primary topologyKey: topology.kubernetes.io/zone containers: - name: worker image: registry.example.com/billing/session-worker:2.1.0 resources: requests: cpu: 100m memory: 128Mi

For Ingress, make sure the AWS Load Balancer Controller uses IP targets with cross-zone balancing disabled when traffic volume justifies it:

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: payments namespace: billing annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/load-balancer-attributes: >- load_balancing.cross_zone.enabled=false spec: ingressClassName: alb rules: - host: payments.example.com http: paths: - path: / pathType: Prefix backend: service: name: payments-api port: number: 8080

Disabling cross-zone balancing on an ALB is free of charge for the ALB itself but requires balanced target counts per AZ — which the topologySpreadConstraints above give you.

Security best practices

Cost optimization must not weaken your perimeter. A few rules:

  • Never replace a NAT Gateway with a public subnet. Moving nodes to public subnets eliminates NAT cost and your network isolation at the same time. Use VPC endpoints instead.
  • Scope endpoint policies. A default VPC endpoint policy is "Effect": "Allow", "Principal": "*", "Action": "*". Restrict it: 
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*", "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"], "Resource": [ "arn:aws:s3:::my-app-artifacts", "arn:aws:s3:::my-app-artifacts/*", "arn:aws:s3:::prod-eu-west-1-starport-layer-bucket/*" ], "Condition": { "StringEquals": {"aws:PrincipalOrgID": "o-exampleorgid"} } }] }
  • Lock down the endpoint security group. Allow only TCP/443 from the VPC CIDR or from the node security group — not 0.0.0.0/0.
  • Keep egress filtering. If you remove NAT for AWS services, you still need an explicit egress path for third-party APIs. Use NetworkPolicy (or EgressFirewall on OpenShift) so pods cannot reach arbitrary destinations: 
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: restrict-egress namespace: billing spec: podSelector: matchLabels: app: payments-api policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: openshift-dns ports: - protocol: UDP port: 5353 - to: - ipBlock: cidr: 10.0.0.0/16 # VPC, covers the interface endpoints ports: - protocol: TCP port: 443
  • Protect the flow log bucket. It contains your internal topology. Enable SSE-KMS, block public access, and apply a lifecycle rule (30–90 days) — flow logs grow fast and become their own cost problem.
  • Least privilege for the flow logs role. The IAM role used by the VPC Flow Logs delivery service should only allow s3:PutObject on the specific prefix, with aws:SourceAccount / aws:SourceArn conditions to prevent the confused deputy problem.

Conclusion

Hidden network costs are not an AWS pricing anomaly; they are a direct reflection of your architecture. Three takeaways:

  1. Measure before optimizing. Cost Explorer gives you the magnitude (NatGateway-Bytes, DataTransfer-Regional-Bytes). VPC Flow Logs in Parquet + Athena give you the attribution. Without pkt-dstaddr and traffic_path, you are guessing.
  2. Gateway endpoints for S3 and DynamoDB are free and non-negotiable. Interface endpoints for ECR, STS, CloudWatch Logs and KMS pay for themselves above roughly 300–500 GB/month per service. Always deploy them in every AZ you run workloads in.
  3. Zone awareness is a scheduling problem. trafficDistribution: PreferClose only works when topologySpreadConstraints guarantee endpoints in every zone. Pair them, and verify the hints are actually present in the EndpointSlices.

Build the Athena queries into a monthly report and attach the output to your FinOps review. The first pass typically removes 30–50% of NAT spend within a sprint; the inter-AZ part takes longer because it requires touching deployments — but it is the part that keeps growing with your traffic.