S02 E10 - Durable Execution for AI Agents in Kubernetes

According to Mark Fussell, durable execution, not new agent vocabulary, is what actually changes when Kubernetes teams start running AI agents in production. Mark spent 30 years building distributed systems, 20 of them at Microsoft, where he built the platform running Azure's SQL Server at scale. Nine years ago he started Dapr (Distributed Application Runtime), now a CNCF-graduated project. And four and a half years ago he left Microsoft to found Diagrid, which is Dapr's primary maintainer and sells a commercial runtime called Diagrid Catalyst.

An agent is a long running process, a tool call is an RPC call, and MCP is a service contract problem. Memory is state, and the context window is a cache. None of that requires new infrastructure thinking. What's new is non-determinism, and that's where the real engineering problem sits.

Why AI agents change the contract, not the architecture

There is a direct line between the rise of microservices and what's happening with agents now. Microservices split monoliths apart because businesses needed to ship faster, not because developers wanted more services to operate. Dapr's original assumption, made before the current AI wave, was that developers needed consistent APIs for service discovery, pub/sub messaging, state management, and workflow orchestration instead of every team hand-rolling their own.

That bet paid off in a specific way: Dapr became the contract between the platform team and the application team. For example, instead of deeply embedding the Kafka API and needing to rip out all that code to switch message busses, developers could instead embed the Dapr pub/sub API and trust it to be stable, no matter which message bus was used underneath.

What's different with agents isn't the topology; it's that repeating the same input can produce a different output. Flow control gets decided by the model at runtime instead of by the code you wrote at design time, and a single step can now take an hour instead of milliseconds. It's smaller in scope than the microservices shift, but deeper in kind: it's not a new architecture, it's a new definition of what "correct" means for a given request.

What durable execution actually does for agents

The core technical concept Fussell keeps returning to is durable execution, sometimes called workflow or orchestration. Think of it as a state machine moving through steps, saving state at each one. If the process dies at step 99 of 100, all the variables, inputs, and outputs are replayed to get it back to step 99, not step 0. He calls it "process reincarnation."

That matters more with LLM calls in the loop for two reasons: non-determinism and latency. A deterministic call is trivial to replay. An agent that decided, mid-run, which MCP server tool to call and why is much harder to resume cleanly if you don't know what it already did. And where most RPC calls run in milliseconds, a call that touches a language model can take 30 seconds or, in agent workflows, minutes to hours. Retry and recovery logic that was optional for microservices becomes load-bearing for agents.

Dapr's answer is its workflow engine plus an added conversation API for swapping language models, and Diagrid has since built an agent framework, Dapr Agents, on top of the same durable execution primitives.

When to keep your workflow deterministic

Fussell says you shouldn't replace working deterministic code with a model call just because it’s possible. It’s easy to replace a simple process with an agent, but you risk spending a lot of money to have something more expensive, less reliable, and slower. His preferred pattern is to call a model only where a decision genuinely has to be made – for example, an SRE might use a workflow where intake and routing are completely deterministic, calling a model only to parse which of a thousand playbooks applies to this incident. The workflow, not the model, owns the steps.

Fussell shared two customer examples. Zeiss, the lens manufacturer, built an agent that reads uploaded prescription data, fills out the associated paperwork, and tracks the request through the pipeline. A logistics company built a warehouse-manager agent that reads incoming shipment-delay data and drafts customer update emails for a human to approve. In both cases the agent proposes and a human or a workflow step still owns the outcome that matters to the business.

Agent identity, and why logs aren't attestation

Dapr assigns identity at the process level, not just the pod level, using the SPIFFE standard (also a CNCF project). Fussell says this was a deliberate decision from Dapr's first year. You can run 20 processes in a pod and give each one a distinct identity, which matters once an agent has broader, more unpredictable access across systems than a typical service does.

There is a second identity problem that's specific to agents acting on someone's behalf. If an agent books a flight or approves a customer refund "for" a user, who's accountable for that action? The user or the agent? He connects this to the EU AI Act's high-risk category, which he says will require what he calls tamper-proof attestation for systems touching EU citizen data, cryptographic proof of exactly what tool was called and when. He's explicit that this is different from writing logs where a log can be edited after the fact, so it isn't proof of what actually happened, only a record someone chose to write.

What platform engineers should actually do

Fussell's closing advice for platform engineers evaluating this space is to translate, not to relearn. Most of what's labeled new is a renamed version of a problem the field has already solved. Ignore the model release cycle, that's just  "leaderboard noise," and keep the attention on durability, retries, state, identity, observability, and cost control, which he says is becoming a real problem as teams see agents rack up runaway spend with no enforcement layer in place.

He estimates that roughly 90% of existing platform engineering skills transfer directly to agentic systems. The 10% that's genuinely new is the shift from deterministic, millisecond responses to non-deterministic responses that can take minutes and need to be recovered, verified, and scoped like any other production system, just on a different clock.

‍

FAQ

What is durable execution?

Durable execution saves a process's state at every step, so if it fails, it resumes from its last completed step instead of restarting from zero.  

Why do AI agents need durable execution?

Agent calls to a language model are non-deterministic and can take 30 seconds to hours, so replaying a failed run needs to know exactly what the agent already did, not just where it was in the code.

Is Dapr an agent framework?

Dapr is a runtime with APIs for service discovery, pub/sub messaging, state, and durable workflows. Dapr Agents is a separate framework built on those primitives, maintained by Diagrid.

How do AI agents get an identity in Kubernetes?

Dapr assigns identity at the process level using the SPIFFE standard, a CNCF project, so multiple processes in one pod can each carry a distinct identity instead of sharing the pod's.

What is tamper-proof attestation for AI agents?

Cryptographic proof of each step an agent executed. Unlike a log, which can be edited after the fact, attestation proves the step actually happened.