Getting Started with Cilium: The CNI That Does Everything

If you’re spinning up a new Kubernetes cluster and haven’t decided on a CNI yet, Cilium is the answer I’d give in 2026. It’s the default in EKS (when you choose the AWS VPC CNI with Cilium policy enforcement), it’s the recommended CNI for many distributions, and it’s becoming the de facto standard for production Kubernetes networking.

Here’s why, and how to actually set it up.

Why Cilium Over Other CNIs

The CNI landscape has options: Flannel (simple, limited), Calico (established, iptables or eBPF mode), WeaveNet (aging), Antrea (VMware-backed), and Cilium. The consolidation toward Cilium has been happening for several reasons:

eBPF-native: Cilium was built on eBPF from the start, not retrofitted. This means the performance and observability properties are fundamental to the architecture, not add-ons.

NetworkPolicy done right: Standard Kubernetes NetworkPolicy is powerful but limited. Cilium’s CiliumNetworkPolicy extends it with:

L7 policy (HTTP method/path-aware rules)
DNS-based policy (allow traffic to api.github.com)
Identity-based policy (based on pod labels, not IP addresses)
Host networking rules

Hubble observability: Cilium includes Hubble — a network observability platform that shows you exactly what traffic is flowing where, with no application instrumentation required.

Service mesh without sidecars: Cilium’s Service Mesh mode provides mTLS and L7 observability at the kernel level, eliminating the overhead of sidecar proxies.

BGP support: Cilium can handle BGP peering for load balancer IP allocation, making it a full networking solution without needing MetalLB.

Installing Cilium on RKE2

RKE2 comes with Canal (Flannel + Calico) by default. To use Cilium:

# Disable the default CNI when bootstrapping RKE2
# /etc/rancher/rke2/config.yaml
cni: none  # Don't install default CNI

# After RKE2 starts, install Cilium via Helm
helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --version 1.16.5 \
  --namespace kube-system \
  --set operator.replicas=2 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=$(hostname -I | awk '{print $1}') \
  --set k8sServicePort=6443 \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set prometheus.enabled=true \
  --set operator.prometheus.enabled=true \
  --set hubble.metrics.enableOpenMetrics=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2}" \
  --wait

For existing clusters with a different CNI, migration is more complex (you’ll need to drain nodes and swap the CNI), but possible.

Verifying the Installation

# Install the Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz

# Check Cilium status
cilium status --wait

# Run connectivity tests (comprehensive health check)
cilium connectivity test

Expected output:

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       OK
    \__/        ClusterMesh:        disabled

DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2

Replacing kube-proxy

One of Cilium’s most impactful features: completely replacing kube-proxy with an eBPF implementation:

# Verify kube-proxy replacement is working
kubectl exec -n kube-system ds/cilium -- cilium status | grep KubeProxyReplacement

# Output:
# KubeProxyReplacement: True

With kube-proxy replacement enabled, Cilium handles:

Service load balancing (ClusterIP, NodePort, LoadBalancer)
External traffic policies
Session affinity
All using eBPF maps instead of iptables chains

The performance difference is measurable at scale. At thousands of services, iptables rule processing becomes a bottleneck. Cilium’s O(1) hash map lookups do not.

CiliumNetworkPolicy Examples

Standard NetworkPolicy works with Cilium, but CiliumNetworkPolicy is more powerful:

# L7 HTTP policy: only allow GET requests to /api/* from frontend pods
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-http-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend-api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/api/.*"
        - method: POST
          path: "/api/data"
        # Everything else is denied
---
# DNS-based egress policy: only allow traffic to specific external services
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: external-api-access
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: data-processor
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
    - matchName: "registry.npmjs.org"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
  - toEntities:
  - kube-apiserver  # Allow access to Kubernetes API

Hubble: Network Observability

Hubble is included with Cilium and gives you connection-level visibility:

# Install Hubble CLI
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin

# Port-forward Hubble relay
kubectl port-forward -n kube-system svc/hubble-relay 4245:80 &

# Watch live traffic
hubble observe --namespace production

# Watch only dropped traffic (policy violations)
hubble observe --namespace production --verdict DROPPED

# Watch traffic to a specific pod
hubble observe --pod production/backend-api-xxx

# HTTP request details
hubble observe --namespace production --http-status-code 500

The Hubble UI provides a graphical view:

# Access Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
# Open http://localhost:12000

The service map view shows every connection between services in real-time, including traffic that was allowed vs. denied by network policy.

BGP and LoadBalancer IPs

For bare metal or homelab clusters that need LoadBalancer IP assignment (instead of MetalLB):

# Enable BGP control plane in Cilium
# (Add to Helm values)
bgpControlPlane:
  enabled: true
---
# Configure BGP peering with your router
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: bgp-peering-policy
spec:
  nodeSelector:
    matchLabels:
      bgp-policy: default
  virtualRouters:
  - localASN: 65001
    exportPodCIDR: false
    neighbors:
    - peerAddress: "192.168.1.1/32"
      peerASN: 65000
    serviceSelector:
      matchExpressions:
      - key: somekey
        operator: NotIn
        values: ['never-used-value']
---
# Load balancer IP pool
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
  - cidr: "192.168.10.0/24"

Monitoring with Prometheus

Cilium exports comprehensive metrics:

# ServiceMonitor for Prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cilium-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: prometheus
    interval: 30s
  - port: hubble-metrics
    interval: 30s

Key Cilium metrics to alert on:

# Drop rate (policy violations or other drops)
rate(cilium_drop_count_total[5m]) > 0

# Failed policy resolutions
rate(cilium_policy_endpoint_enforcement_status{status="failed"}[5m]) > 0

# BPF map pressure (approaching limits)
cilium_bpf_map_pressure{name="ct4"} > 0.8  # Conntrack table 80% full

Conclusion

Cilium has earned its position as the premier CNI for production Kubernetes. The combination of eBPF-native performance, superior NetworkPolicy implementation, Hubble observability, and kube-proxy replacement make it the right default for any cluster where performance and security matter.

For a homelab running RKE2, the setup is straightforward and the operational overhead is low once you understand how to use the Cilium CLI and Hubble. The observability alone — being able to watch exactly what traffic is flowing between your pods — is worth the migration from a simpler CNI.

Start with Cilium installed with Hubble enabled, get familiar with the network flow visibility, then progressively implement CiliumNetworkPolicy for your applications.