Kubernetes networking looks simple from the outside. You apply a Deployment, expose it with a Service, add an Ingress rule, and traffic flows. The confusion starts the first time traffic does not flow, because the system that moves packets is not one component. It is several layers that each solve a different problem: pod to pod communication, stable access to a changing set of pods, external entry into the cluster, and the plugin that wires the whole thing to the host network. Understanding each layer separately makes debugging much faster when something breaks.

Every pod gets an IP, and that changes everything

In Kubernetes, every pod receives its own IP address from a flat network space, and every pod can reach every other pod directly, without network address translation in between. A container does not need to know which node its peer runs on, and two pods can both listen on port 8080 without conflict, because their IPs differ.

The consequence is that pod IPs are the real unit of networking, while everything else is a layer of indirection on top. The problem is that pod IPs are unstable. Pods are created and destroyed by Deployments, autoscalers, and node failures. A client that hardcodes a pod IP will break the first time that pod is replaced. That instability is exactly what Services exist to hide.

Services give you a stable address for an unstable set of pods

A Service is a stable virtual IP, called the ClusterIP, plus a DNS name, backed by a set of pods selected by labels. When a pod dies and a replacement starts with a new IP, the Service updates its endpoint list automatically. Clients keep calling the same address and never notice the churn underneath.

There are four main Service types, and each builds on the previous one:

  1. ClusterIP is the default. It exposes the Service on an internal IP reachable only from inside the cluster. This is the right choice for internal traffic, such as a backend calling a database or one microservice calling another.
  2. NodePort opens the same Service on a fixed port on every node, so traffic to any node IP on that port is forwarded to the Service. It is useful for quick testing, but it is a blunt tool for production, since it exposes a high numbered port on every node and gives you no routing based on hostname or path.
  3. LoadBalancer builds on NodePort and asks the cloud provider to provision an external load balancer that forwards to the NodePorts. This is the standard way to expose a single service to the internet on a cloud provider, but each Service gets its own load balancer, which becomes expensive and hard to manage when you have dozens of public services.
  4. ExternalName is a special case that maps a Service name to an external DNS name instead of to pods. It is handy when code inside the cluster needs to call an outside dependency through the same service discovery mechanism as everything else.

A headless Service, created by setting clusterIP to None, skips the virtual IP and returns pod IPs directly through DNS, which matters for stateful workloads where a client must reach a specific replica.

Here is a minimal Service definition for an API deployment:

apiVersion: v1 kind: Service metadata: name: api spec: selector: app: api ports: - port: 80 targetPort: 8080 type: ClusterIP

The selector is the whole mechanism. If the labels on your pods do not match the selector, the Service will have no endpoints and traffic will go nowhere, one of the most common causes of a Service that looks correct but never answers.

kube-proxy is where the virtual IP becomes real

A ClusterIP corresponds to no network interface, and nothing in the kernel knows how to route to it by default. That job belongs to kube-proxy, which runs on every node and programs the packet filtering rules that make Service IPs work.

In iptables mode, kube-proxy writes rules redirecting Service traffic to a backing pod IP chosen at random. IPVS mode uses the kernel IPVS subsystem instead, which scales better and supports more balancing algorithms. Either way, the Service IP is maintained by rules on every node, updated whenever the endpoint list changes.

This explains a failure mode that confuses people. If kube-proxy rules are stale on one node, traffic from pods on that node can fail while traffic from other nodes works. Checking endpoints with kubectl get endpoints separates a selector problem from a proxy problem: if the endpoints are present and correct, the selector is fine and the fault lies further down the stack.

Ingress is the front door, not just another Service

A LoadBalancer Service gives each public service its own external address. Ingress solves the problem differently. You run one Ingress controller, backed by a single LoadBalancer Service, and it routes incoming HTTP and HTTPS traffic to the right internal Service based on hostname and path. One public IP can serve many applications, with TLS termination handled in one place.

An Ingress resource is a set of routing rules, not a running process. The controller watches those rules and configures a reverse proxy, usually NGINX or HAProxy, to implement them. If the resource exists but traffic returns a 404, the controller may not be installed, may watch a different ingress class, or may have failed to reload.

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: app.example.com http: paths: - path: /api pathType: Prefix backend: service: name: api port: number: 80 - path: / pathType: Prefix backend: service: name: frontend port: number: 80

A few practical points follow from this design. First, Ingress only handles HTTP and HTTPS; raw TCP or UDP from outside needs a LoadBalancer Service or a controller extension. Second, path matching has sharp edges: Prefix matching behaves differently from exact matching, and rewrite annotations change the path the backend sees. Third, TLS comes from a Secret referenced by the Ingress resource, and certificate rotation should be automated with a tool like cert-manager rather than done by hand.

CNI is the plugin that builds the pod network

So far we have assumed pods can reach each other. That assumption is implemented by the Container Network Interface, or CNI, plugin chosen when the cluster was built. When the kubelet starts a pod, it calls the CNI plugin to attach the pod to the network: allocate an IP, set up a virtual interface, and configure routing so that the pod IP is reachable from other nodes.

Different CNI plugins make different tradeoffs, and the choice is hard to change later. Flannel is simple and reliable but offers no network policy enforcement. Calico adds rich network policy support, letting you restrict which pods can talk to which. Cilium uses eBPF in the kernel for high performance networking, observability, and policy, and it is increasingly the default on managed platforms.

The practical implication is that two clusters running the same manifests can behave differently. A NetworkPolicy that works on Calico or Cilium does nothing on a cluster whose CNI ignores policy, so verify that your CNI actually enforces policy before relying on it, and test a deny by default posture in staging first.

DNS ties it together

Service discovery is DNS based. CoreDNS answers names like api.default.svc.cluster.local, with short forms like api working within the same namespace. DNS failures look like application bugs, so before blaming application code, check whether resolution works from inside a pod and whether CoreDNS itself is healthy.

A debugging order that saves time

When traffic fails, work from the top of the stack down, and verify each layer before moving on:

  1. Confirm the pods are running and ready, and that the application listens on the expected port inside the container, not just on localhost.
  2. Check the Service endpoints. Empty endpoints mean a selector or label problem, not a network problem.
  3. Test pod to pod connectivity directly by pod IP. If that fails, the CNI or node routing is suspect.
  4. Test the Service ClusterIP from a pod on the same node and from a pod on another node. A difference between the two points at kube-proxy on one node.
  5. Test DNS resolution of the Service name from inside a pod.
  6. For external traffic, check the Ingress controller logs and confirm the controller is running and watching the correct ingress class, then check the Service and endpoints behind it.

Most incidents resolve at one of these steps, and the order matters because each step rules out an entire layer.

Kubernetes networking is several systems cooperating: the CNI builds the pod network, Services provide stable addresses, kube-proxy implements them in the kernel, DNS makes them discoverable, and Ingress provides a single managed front door. Learn each layer on its own terms, and the system becomes something you can reason about and fix.