S02 E03 - Securing the AI Agent Ecosystem on Kubernetes

In this AI Kubernetes Show episode, host William Morgan spoke with Evaline Ju, software engineer at IBM Research, about her work in the AI space, specifically focusing on explainability, fairness algorithms, and the challenges of bringing LLMs and agents to production on Kubernetes. She also shared her thoughts on the open source project Kagenti, a platform or middleware for Kubernetes focused on security-first and governance-first patterns.

The evolution of trust in AI

This blog post was generated by AI from the interview transcript, with some editing.

Ju's early work focused on what she called "encoder models"—essentially, the foundational transformer models where text input yields numerical output. Initially, the goal was to support and implement explainability algorithms like SHAP and LIME across various applications. However, the rapid emergence of large language models (LLMs) required a shift in focus, mainly because many of those original algorithms weren't easily transferable to the LLMs and agent-based applications we see today.

Guardrails and explainability: Building trust in LLMs

Creating trust in AI is the ultimate goal when we talk about explainability and guardrails. For traditional algorithms, explainability is relatively straightforward; you can usually attribute a decision back to the original training data. With large language models (LLMs), however, it's much harder to pinpoint the exact piece of information that led to an output.

Guardrails is a broad term that can mean different things, like rate limiting. But for LLMs, the non-deterministic nature of the models requires specific content moderation guardrails at both the input and output levels.

At the input level, a guardrail is necessary to prevent confidential information from inadvertently being used to train the model. This could be as simple as tracking mentions of competitors. At the output level, operators need to check the model's response. This guards against threats like attempted SQL injections or random, disguised information that might be an attempt to exfiltrate more data from the system.

Guardrails are also important for detecting hallucinations, where the LLM returns something confidently. How do you detect if that is real or not? LLM guardrails can be creatively bypassed, such as transforming malicious directions into Leet speak.

Why Kubernetes needs agentic guardrails

When building complex multi-agent systems, the need for Kubernetes becomes clear since agents are workloads that frequently call out to various external services. The challenge, however, is that Kubernetes is currently more optimized for stateless workloads. This creates a gap in effectively scaling, observing, and managing these stateful agents and their complex interactions.

Identity and authorization in complex systems

In systems built with agent-tech, where agents interact with each other, call tools, and use LLMs, authorization is a critical concern. Ensuring the agent is acting on behalf of the right user requires robust workload identity. It’s essential to be able to distinguish the agent's identity and correctly map that back to the original user's identity.

The production gap

While a large majority of engineers (around 82%) are experimenting with AI agents, only about 7% are actually deploying them to production daily. This significant gap stems from the difficulty in ensuring consistency and the challenges of proper observation. Rolling out agents without established standards is a high-risk endeavor, primarily because the decisions made by LLMs tend to slowly change over time, making stable and consistent benchmarking a real headache.

Enforcing security at the platform layer

Platform owners need to be able to enforce a uniform layer of control and visibility that solves security problems developers shouldn't have to worry about. 

When it comes to securing agents, there are several critical platform patterns to adopt. For example, tool filtering is essential. The platform should reduce the attack surface by only allowing agents to access the tool servers they actually need. A summarization agent, for instance, has no business seeing tools for deleting or writing. 

Gateway enforcement is another key strategy, providing a great place to enforce tool authorization and content detection guardrails. Also, agents should use short-lived tokens instead of an API key sitting around as a pod or deployment secret. This significantly reduces the attack surface if an agent is compromised. 

Finally, platform-level visibility is important for detecting anomalous behavior, such as an agent that usually reads GitHub issues suddenly trying to invoke a tool to send an email or write to a database.

The AI Gateway WG is also focusing on payload processing, which involves inspecting the request and response body to block or transform content. This could include masking out PII before the data goes to an external tool. There's a trade-off to consider in implementation. Simple PII checks, like those for social security numbers, can use a basic regex for lower request latency. However, more general PII checks often require more sophisticated large language models, like Google Shield Gemma or Meta's Lama Guard. While more thorough, this introduces additional costs, such as GPU expenses and higher request latency.

Shaping the future of agentic networking

Ju is actively involved in the Kubernetes Special Interest Groups (SIGs) that are focused on establishing standards for running agent workloads. One of these is the Agentic Networking Group, which works on agent communication and networking use cases. The goal here is to create standards for securing agent workloads that can persist beyond protocols like MCP. The other is the Kubernetes AI Gateway Working Group, which focuses on managing traffic flow to AI models in general.

These groups are defining new resources built on the core Gateway API, including backend resources, which are used to represent an external tool or language model (LM) service outside the cluster. They are also working on access policies, which are used to define exactly which tools an agent is allowed to call, ensuring constraints are properly enforced.

The two-year outlook: Desired and observed Behavior

Looking ahead, the focus is shifting toward establishing observability conventions to standardize end-to-end tracing across agents, tools, and LLMs.

This evolution extends to Kubernetes. The traditional model of reconciling on a static desired state doesn't quite work with the non-determinism introduced by all these agents. The vision is to move toward reconciling desired and observed behavior. This change would allow for policy enforcement, like ensuring that PII never leaves the system, even when an agent is making real-time decisions about which L1 or tool to call. For platform engineers, the key areas to consider will be the egress boundary and this comprehensive observability.

FAQ

How does explainability for traditional algorithms differ from building trust in LLMs with guardrails?

For traditional algorithms, decisions can be attributed back to the original training data. For LLMs, it is harder to pinpoint the exact information leading to an output, requiring content moderation guardrails at the input and output levels.

What are the key challenges in bringing complex multi-agent systems and LLMs into production on Kubernetes?

Kubernetes is currently optimized for stateless workloads, creating a gap in effectively scaling, observing, and managing stateful agents. The difficulty in ensuring consistency and stable benchmarking of LLM decisions is likely why we aren't seeing widespread prod adoption yet. 

What are some critical platform-level security patterns that should be adopted to secure AI agents?

Critical patterns include tool filtering to reduce the attack surface, using short-lived tokens instead of API keys, and enforcing security policies at the gateway layer. Platform visibility is also key for detecting anomalous agent behavior.

What is the anticipated shift in Kubernetes' model for managing non-deterministic AI agents in the near future?

The vision is to move from reconciling on a static desired state to reconciling desired and observed behavior. This change allows for policy enforcement, ensuring, for example, that PII never leaves the system even when agents are making real-time decisions.