I run Kubernetes. I’ve been running it for years. I believe in the model and I think, for the right problems, it’s the best solution available. I also think the industry significantly underrepresents what it actually costs to operate.
This is the post I wish I’d read before I started.
The Complexity Is Real
Kubernetes is complex. Not “learning curve” complex—intrinsically, architecturally complex. The control plane alone—API server, etcd, controller manager, scheduler—is a distributed system with its own failure modes. Adding worker nodes with kubelet, kube-proxy, CNI plugins, CSI drivers, and your workloads, and you have a system with a large number of components that can fail in interesting combinations.
This complexity serves real purposes. The abstractions that make complexity are the same abstractions that give you scheduling, healing, scaling, and declarative configuration. You can’t have the benefits without the complexity that produces them.
But the complexity means: something will always be wrong. At any given moment in a moderately sized Kubernetes cluster, there’s probably a pod in CrashLoopBackOff, a node with a disk pressure warning, a certificate expiring in 30 days, or a HelmRelease in a degraded state. The question isn’t “will things break?” but “is the right thing broken?” Learning to distinguish “this is a problem I need to fix immediately” from “this is noise I can investigate later” is a skill that takes time to develop.
The Operational Overhead
A Kubernetes cluster is not a one-time setup. It’s a continuously operated system that requires ongoing attention:
Kubernetes version upgrades: Kubernetes releases new minor versions every 4 months. Each version is supported for about 14 months (three minor versions). Falling behind means running out-of-support software with known vulnerabilities. Upgrading Kubernetes is not trivial—especially if you have Helm charts or operators that need to stay in sync with the Kubernetes API version.
Node OS upgrades: Your nodes run an operating system that also needs patching. With Talos, this is better—immutable OS, atomic upgrades. With traditional Linux, it’s apt upgrade on a machine running your production workloads.
Certificate rotation: Kubernetes uses a lot of TLS certificates. Most rotate automatically (kubelet serving certs, internal Kubernetes PKI), but some don’t. Setting up kube-cert-manager and monitoring cert expiration is not optional—expired certs bring clusters down.
etcd maintenance: etcd needs occasional compaction, defragmentation, and backup verification. It’s not high-maintenance, but it’s not zero-maintenance either.
Helm chart updates: If you’re running 20 Helm charts (ingress, cert-manager, monitoring, vault, longhorn, and all your applications), they all release updates. Major version upgrades often require manual migration steps. Staying on old versions accumulates security debt.
This ongoing work is real. It’s not unreasonable—mature tooling and GitOps help enormously—but it doesn’t go to zero.
The Learning Curve Is Longer Than Advertised
“Getting started with Kubernetes” means getting a cluster running and deploying a Hello World app. That’s achievable in an afternoon.
Truly understanding Kubernetes—understanding what happens when a node goes down, why a pod can’t pull its image, how network policies interact with your CNI, why your HPA isn’t scaling—takes months to years of hands-on experience.
The concepts that seem optional when you start become essential later:
- etcd understanding (what it stores, what happens when it’s unhealthy, backup and restore)
- Kubernetes networking at the IP/iptables level (essential for debugging)
- Security contexts and admission controllers (essential for compliance)
- StorageClass and PV/PVC lifecycle (essential for running databases)
- Custom Resource Definitions and operators (you’ll install them without understanding them; eventually you need to understand them)
The learning curve is a real investment. It pays off—these skills transfer and compound. But don’t let anyone tell you it’s a weekend project.
When Kubernetes Is the Right Answer
Kubernetes earns its complexity cost when:
- You have multiple services that need to run reliably and scale independently
- You want the operational discipline of GitOps, declarative configuration, and drift detection
- You need to run across multiple environments with consistency
- Your team has or is building Kubernetes expertise
- You want the ecosystem—the tooling, the community, the standard patterns
Kubernetes is genuinely excellent for these use cases. The operational model it enforces (declarative, reconciliation-based, immutable infrastructure) is the right model for production systems.
When Kubernetes Is Not the Right Answer
Kubernetes adds unjustified cost when:
- You’re running a single small application
- Your team has no Kubernetes experience and no path to building it
- Your workload isn’t containerized and you’re not planning to containerize it
- You need to move fast and don’t have the luxury of time to do it right
- Your infrastructure won’t benefit from the abstraction (specialized hardware, legacy applications with host-level dependencies)
The siren call of Kubernetes is strong because it’s modern and well-regarded. But “we should use Kubernetes” is not a business requirement. “We need to scale these services independently” or “we need consistent deployments across dev, staging, and production” might be requirements that Kubernetes satisfies. Start from the requirement.
The Right Mindset
Running Kubernetes well is a craft. It rewards investment. The engineers who get the most value from it are the ones who engage with it seriously—who understand the control plane, who debug networking issues from first principles, who read release notes and understand why the API changed.
If you approach it as “I’ll run Kubernetes because everyone else does and figure it out as I go,” you’ll spend a lot of time confused and frustrated by a system you don’t understand.
If you approach it as “I’m investing in building deep expertise in this platform,” you’ll find it pays dividends for years—both in what you can build and in career value.
I don’t regret running Kubernetes. I do regret some of the ways I learned it—more reading, less trial-by-fire would have saved me time. The costs are manageable when you go in clear-eyed about what they are.
