S02 E08 - Disaggregated Inference on Kubernetes

AI workloads break a lot of things with the way we’ve typically done work in Kubernetes. It’s well understood that the execution model for inference often depends on things happening outside the cluster and that the information a proxy needs is buried in a JSON body rather than being in headers. It’s also a significant problem that inference requests are best handled in a disaggregated fashion, splitting a single Kubernetes-hosted LLM request across a pre-fill pod and a decode pod – and Kubernetes proxies were never built to handle either half of that split cleanly.

That's the problem Morgan Foster works on. Foster spent years as a Site Reliability Engineer, first at Google (on the Global Cache CDN that serves YouTube and on the Google Payment Stack) and then at Twitter from 2021 to 2022, where she worked on rollouts to Twitter's bare metal hosts, at the time the company's single biggest source of outages. After Twitter, she built an LLM-driven web crawler that mapped roughly 13,000 US school board websites, pulled meeting records back to 2010 from whatever bespoke FTP server or WordPress plugin each board happened to be running, and handed the corpus to researchers and nonprofits. (One finding from that work is that when school budgets get cut, culinary programs go first and music programs eventually follow.)

Foster now works in the Office of the CTO at Red Hat, in emerging technologies, alongside IBM Research. She's involved in the llm-d project and the AI Gateway Working Group and spoke with William Morgan on the AI Kubernetes Show about two specific problems: how disaggregated inference splits a single request across two pods and why the proxies sitting in front of that traffic weren't built for any of this.

Why inference runs on Kubernetes in the first place

Where training a model is fundamentally batch work, inference requests happen in real time, and real-time requests are exactly what Kubernetes was built to handle. "A lot of the primitives that you would need to invent [to do inference on other platforms] are already invented and battle tested," Foster said. That's why companies like OpenAI run inference on Kubernetes despite the platform not being purpose-built for GPU-bound workloads: the primitives for serving requests reliably in real time already exist, and rebuilding them elsewhere isn't worth it.

Pre-fill and decode: the two phases inside every inference request

Every LLM inference request runs through two phases with completely different hardware profiles, and that split is the reason disaggregated inference exists at all.

Pre-fill processes the entire prompt at once to produce the KV cache, which tracks which parts of the input are most important. This is a parallel linear-algebra job, which is what GPUs (and TPUs) are designed for. Because pre-fill can use the full parallelism of the hardware, it's fast, and hardware utilization is high.

Decode, where the model is generating a response, is different. Most models in production today use autoregression, which limits them to producing only one token at a time (each new token depends on all the tokens before it). That serial dependency means decode can't use the hardware nearly as efficiently as pre-fill, which makes decode much slower and more expensive than pre-fill.

(Autoregressive models aren’t the only game in town: diffusion models, in particular, can use the hardware much more effectively. Foster hasn’t seen wide adoption here, though, because autoregressive models do a better job of working with text. Separating pipeline stages to manage bottlenecks is likely to be relevant no matter what changes we see in the models, though.)

How llm-d splits pre-fill and decode across pods

One of the core ideas of the llm-d project, which builds on vLLM, is to stop running pre-fill and decode in the same place. Its scheduling system can choose a pre-fill pod based on the state of KV caches across the cluster and transfer the resulting KV blocks to a decode pod using remote DMA as they're generated. Separating the two helps address the decode phase's weaker hardware utilization without dragging down pre-fill's, lowering the overall cost of inference by increasing overall hardware utilization.

This focus on performance is important because in agentic systems, more speed can mean more actual capability. Faster token generation lets a smaller model do more effective reasoning within a given time, or lets an agent do more effective exploration of a problem space (especially as agents increasingly write their own tools to run in a sandbox). Foster notes that the industry may be getting away with sloppier engineering decisions right now because inference is comparatively slow. As inference speeds up, the consequences of those decisions, particularly around networking, are going to surface. 

Why proxies weren't built for AI traffic

What’s hard is addressing the challenges without completely rebuilding the proxies already in use in Kubernetes. “You end up looking a little bit like a Rube Goldberg machine," Foster said, because current proxies have to pick both the pre-fill pod and the decode pod in a single pass, and if either fails there’s nothing to handle the fallout. If proxies had the right semantics for holding state through a multi-step decision and handling re-entrancy, you could build a clean failover path instead of a workaround.

This is where the AI Gateway Working Group's problem statement starts. A Kubernetes proxy's default posture is ingress: a request arrives, the proxy handles authentication and routing based on the headers, the request (including the body) gets passed to a workload in the cluster and the response gets handed back to the client.

AI traffic breaks this model, though. First, in many cases a workload will depend on inference outside the cluster, so proxies need support for governing egress as well as ingress, which is a gap across the whole ecosystem. Second and harder, routing AI requests depends on information in the body of the request, not just the headers (for example, the model name a request targets is in a JSON field in the body). Modern proxies are very good at streaming the body rather than buffering it, but AI requests require buffering – and the bodies can be large, with multimodal requests possibly running up to 40 MB by Foster’s estimate.

Getting all of this right requires handling a chain of dependent policies. You need to know the model before you can count tokens correctly, because different models use different tokenizers. A guardrail system might inject content into the context at the proxy, which changes the token count you'd otherwise calculate. Protocol translation adds another layer: a guardrail system built around one API style and a response that needs to come back in a different style. Today, all of this logic lives in external processors, similar to the ext_authz pattern, running as separate services with state passed between them by hand. There's no clean way to define sequencing, no clean semantics for what failing open or failing closed should mean at each step, and no re-entrancy story for cases like a step failing after credential injection, where you need to re-inject credentials for a fallback model.

This is what causes the pre-fill/decode failover gap described earlier. Nothing handles failover today because proxies don't support this concept of a policy pipeline that can hold state and orchestrate across multiple steps.

What’s being built for the future

The AI Gateway working group is designing two things in parallel: 1) a control plane that targets the specific places proxies are struggling with AI-specific traffic and 2) changes to the Gateway API itself, aimed at establishing an industry-wide baseline for how these policies compose.

The goal is to give proxies the semantics they're currently missing, so that problems like the pre-fill/decode failover gap can be solved directly in the proxy layer – but all the tooling around AI is changing very quickly, from early attempts at getting valid JSON out of a model in a loop to agent frameworks to the coding harnesses common today. That rapid evolution will continue, but the operational problem under the hood is here to stay. AI requests can’t have a single well-known execution path determined ahead of time, and that drives the need for sandboxing, governing inference, and everything covered above.

Foster recommends AI newbies build something themselves (e.g., using a Kind cluster on a laptop). Wire up agents that talk to each other, and build an event bus and a cron system instead of relying on whatever ships in a packaged tool. Multi-agent systems are still early, but that's exactly where hands-on experience compounds.

FAQ

What is disaggregated inference in Kubernetes?

Disaggregated inference splits an LLM request's pre-fill and decode phases across separate pods, since each phase has a different hardware profile. A scheduler picks the two pods, then transfers the KV cache between them.

What's the difference between pre-fill and decode in LLM inference?

Pre-fill processes the whole prompt in parallel, so hardware utilization is high. Decode generates one token at a time and utilization drops. The KV cache from pre-fill is what makes decode fast.

Why can't Kubernetes proxies handle AI traffic well today?

Proxies route on headers, but AI policy decisions depend on the request body, like a model name buried in streamed JSON that can total tens of megabytes. Proxies weren't built to buffer or hold that much state.

What happens if a pre-fill or decode pod dies during inference?

Nothing handles failover cleanly today. The pre-fill/decode pod choice happens once, in one pass through the proxy. If either pod dies afterward, the computed KV blocks get thrown away instead of recovered.

What is the AI Gateway Working Group building?

A control plane and Gateway API changes that give proxies the sequencing, state, and re-entrancy semantics they're missing, so problems like broken pre-fill/decode failover can be fixed in the proxy layer itself.