Kubernetes is the AI OS — but teams are stuck on inference

If you’re a student or early in DevOps, you’ve probably heard: “Kubernetes is becoming the operating system for AI.”

CNCF survey data backs the headline — 66% of orgs hosting generative AI use Kubernetes for some of that work. What the slide rarely says: 52% of organizations don’t train models at all.

So most real teams aren’t building foundation models. They’re running someone else’s model — answering user requests. That job is called inference. That’s where the pain shows up first.

What you get taught vs what shows up at work

In class or a bootcamp, “AI on Kubernetes” often looks like: pull a container, expose a port, maybe attach a GPU label. On a real team, the questions sound different:

• Who restarts the model when it crashes at 2am?
• Why is the GPU “allocated” but barely used?
• Who pays when usage spikes — platform, ML, or the app team?
• How do we roll back a bad model the same way we roll back an app?

Those aren’t algorithm questions. They’re platform / DevOps questions — ownership, cost, and change safety.

A simple mental model (keep this)

Three layers, three owners (ideally written down):

1. Model — which weights / version (often ML or data science)
2. Serving path — how requests reach the model (platform + app)
3. Cluster & accelerators — nodes, GPUs, quotas (platform)

When those lines are blurry, you get a “shadow platform”: everyone invents their own scripts, dashboards, and on-call habits.

Before you learn DRA, GPU slicing, or AI Conformance — learn who owns inference. Tools come after ownership.

Three basics that matter more than hype

1. Idle GPUs — a Pod can look “Running” while the accelerator sits quiet. Healthy ≠ useful. Ask: how do we see utilization?
2. Useful throughput — flashy tokens/sec on a demo isn’t the same as requests that actually succeed for users. Platforms care about the second.
3. Same change discipline as apps — canary, rollback, SLOs. A model deploy is still a production change.

If you’re learning Kubernetes / DevOps, do this next

• Draw one diagram: request → Service/Ingress → model Pod → GPU (even if the GPU is imaginary for now)
• Write three bullets: who owns model, serving, cluster in a team you’d join
• When you see “AI on K8s” content, ask: is this about training or inference? Most jobs lean inference
• Later: explore Kubernetes AI Conformance and device management — after the mental model sticks