Kubernetes networking trips up more engineers than almost any other part of the platform. The layering—node networking, pod networking, Service networking, Ingress—each layer adding abstraction on top of the previous one, with different components responsible for different pieces—makes it difficult to form a complete mental model.
This post is a systematic walkthrough of how traffic flows in a Kubernetes cluster, from container to internet and back.
The Core Network Requirement
Kubernetes mandates a specific networking model:
- Every Pod gets its own IP address
- Pods on the same node can communicate without NAT
- Pods on different nodes can communicate without NAT
- The IP that a Pod sees itself as is the same IP that other Pods see it as
This “flat network” model—no NAT between pods—is the foundation everything else builds on. It’s implemented by your CNI plugin.
CNI: The Foundation
The Container Network Interface (CNI) plugin is responsible for:
- Assigning IP addresses to pods
- Creating network interfaces in pod namespaces
- Programming routes so pods can communicate across nodes
- Enforcing NetworkPolicies
Common CNI plugins:
Flannel: Simple overlay network. No NetworkPolicy support natively. Good for basic homelab use.
Calico: Full-featured CNI with native NetworkPolicy support. Can run in eBPF mode. BGP routing option for datacenter environments. Production-grade.
Cilium: eBPF-based CNI with rich NetworkPolicy (L7 policies), built-in load balancing, and Hubble for network observability. The most powerful option, higher complexity.
Weave Net: Overlay network with encryption. Simpler to operate than Calico/Cilium.
For production use: Calico or Cilium. Cilium’s eBPF-based approach provides better performance and more observability, but Calico is simpler to operate and extremely stable.
Services: Stable Endpoints for Dynamic Pods
Pods are ephemeral. Their IP addresses change when they restart. Services solve this by providing a stable IP and DNS name that routes to a set of pods.
apiVersion: v1
kind: Service
metadata:
name: my-app
namespace: my-app
spec:
selector:
app.kubernetes.io/name: my-app
ports:
- port: 80
targetPort: 8080
protocol: TCP
type: ClusterIP
This creates a stable ClusterIP (e.g., 10.96.45.23) and DNS name (my-app.my-app.svc.cluster.local) that kube-proxy maps to the backing pods via iptables or IPVS rules.
Service Types
ClusterIP (default): Internal only. Accessible within the cluster. Use for inter-service communication.
NodePort: Exposes the service on each node’s IP at a static port (30000-32767). Useful for development, not ideal for production.
LoadBalancer: Provisions a cloud load balancer. In bare-metal environments, requires MetalLB or similar.
ExternalName: DNS alias to an external hostname. Useful for wrapping external services in a Kubernetes-native interface.
DNS: Service Discovery
CoreDNS runs in the cluster and answers DNS queries for Kubernetes resources. The naming convention:
<service>.<namespace>.svc.cluster.local
Within a namespace, you can use just <service>. From a different namespace, use <service>.<namespace>.
Pods get DNS resolution automatically. A pod in namespace-a can reach a service in namespace-b at service-name.namespace-b.
Ingress: External Traffic Entry
ClusterIP Services are not accessible from outside the cluster. Ingress resources, managed by an Ingress controller, route external HTTP/HTTPS traffic to Services.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app
namespace: my-app
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: nginx
tls:
- hosts:
- app.example.com
secretName: my-app-tls
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app
port:
number: 80
The Ingress controller (nginx, Traefik, etc.) watches Ingress resources and configures itself to route accordingly. cert-manager handles TLS certificate provisioning automatically.
NetworkPolicies: Micro-Segmentation
By default, every pod can reach every other pod in the cluster. This is fine until it isn’t. NetworkPolicies restrict traffic flows using label selectors:
# Default deny all ingress in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: my-app
spec:
podSelector: {}
policyTypes:
- Ingress
# Allow traffic from ingress controller and specific namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-and-monitoring
namespace: my-app
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: my-app
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- port: 8080
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- port: 8080
Applying a default-deny policy per namespace and then explicitly allowing required traffic is the security baseline. Without NetworkPolicies, a compromised pod has unrestricted network access to everything else in the cluster.
MetalLB for Bare Metal LoadBalancer Services
Cloud providers automatically provision load balancers for type: LoadBalancer services. On bare metal (homelab, on-prem), MetalLB fills this gap:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: main-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.200-192.168.1.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2advertisement
namespace: metallb-system
spec:
ipAddressPools:
- main-pool
MetalLB assigns IPs from your pool to LoadBalancer services and responds to ARP requests so traffic routes to the cluster. For BGP environments, MetalLB can advertise service IPs via BGP for more production-grade behavior.
Debugging Network Issues
When networking breaks, this is your debugging toolkit:
# Can pod A reach pod B?
kubectl exec -it pod-a -n namespace-a -- curl http://pod-b-ip:port
# DNS resolution working?
kubectl exec -it pod-a -- nslookup service-name.namespace-b
# Are NetworkPolicies blocking traffic?
# Check Cilium (if using Cilium)
kubectl exec -it -n kube-system cilium-pod -- cilium policy trace \
--src-k8s-pod namespace/pod-a \
--dst-k8s-pod namespace/pod-b \
--dport 80
# Capture traffic on pod interface (ephemeral debug container)
kubectl debug -it pod-a --image=nicolaka/netshoot -- tcpdump -i eth0 -w /tmp/capture.pcap
# Check kube-proxy iptables rules
kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl exec -it kube-proxy-pod -n kube-system -- iptables -L -n | grep my-service
The netshoot image (packed with networking tools: tcpdump, dig, curl, iperf, traceroute, etc.) running as an ephemeral debug container is indispensable.
Understanding the network stack matters more for operations than for development. When an application works perfectly in development and fails in the cluster, the answer is almost always networking. Knowing how to trace the path from pod to service to ingress to external is the skill that separates engineers who can debug Kubernetes from engineers who just hope it works.
