Agent Substrate: Advancing Kubernetes for Agents at Scale
Agent Substrate addresses Kubernetes limitations for idle AI agents by multiplexing sessions over warm worker pools. This CNCF Sandbox project, paired with NVIDIA Dynamo and KAI Scheduler, enables efficient stateful agent execution for coding assistants, reinforcement learning, and GPU research workloads.
Tags
Quick summary
Agent Substrate addresses Kubernetes limitations for idle AI agents by multiplexing sessions over warm worker pools. This CNCF Sandbox project, paired with NVIDIA Dynamo and KAI Scheduler, enables efficient stateful agent execution for coding assistants, reinforcement learning, and GPU research workloads.
Agent Substrate: Advancing Kubernetes for Agents at Scale
AI agents spend most of their time idle, waiting for a human prompt, a tool result, or a model response. Traditional Kubernetes infrastructure was not originally built to handle this specific pattern of intermittent activity. Running one Pod per session holds memory for sessions that are not doing anything, while deleting Pods between turns incurs a multi-second cold start penalty every time and loses the agent's state. Reinforcement learning (RL) post-training faces the same wall.
To address these challenges, NVIDIA has developed Agent Substrate, an open-source, security-hardened, zero-trust runtime now donated to the Cloud Native Computing Foundation (CNCF) Sandbox. This runtime is engineered to safely run millions of agents and tools with 10x higher density than standard containers. It multiplexes many sessions over a small pool of warm Kubernetes worker Pods. When an actor goes idle, it is snapshotted out of its sandbox—using gVisor or a microVM—and its worker is freed. When the actor is called again, it is restored onto whichever worker is free, with its memory and filesystem state intact.
As noted by NVIDIA, waking an actor does not require a new trip through the API server, scheduler, or kubelet. Instead, Agent Substrate restores the actor onto an existing worker. The worker pods remain ordinary Kubernetes Pods, ensuring that cluster scheduling, quotas, and placement policies still apply.
The Shape of an Agent Workload
The runtime targets three primary workload types that drive demand for this architecture:
- Coding Assistants and Agent Serving: A coding agent alternates between LLM calls and work in its own environment, such as cloning a repository, editing files, running builds, and executing tests. Users expect the environment to persist between turns and to respond immediately. A platform serving thousands of these sessions cannot afford a dedicated warm sandbox for each one.
- Reinforcement Learning (RL) Post-Training: RL with verifiable rewards runs GPU trainers and generation servers alongside a fleet of CPU-heavy environments, including code sandboxes, compilers, browsers, and simulators. These environments execute actions and check results. Sandbox demand arrives in bursts as rollout batches dispatch actions. A training run can grow from a few hundred sandboxes in early experiments to many hundreds of thousands in production. The slowest environment determines when the trainer gets its next batch, so cold starts on that path leave GPUs idle.
- GPU Sandboxes for Research: A researcher's coding agent needs a GPU, not to serve a model but to compile CUDA code, benchmark a kernel, or run a numerical calculation. It launches a GPU sandbox on a shared cluster, runs the job, and deletes the sandbox. Supporting this across an organization requires many short-lived, isolated GPU sandboxes that start quickly and share a limited pool of GPUs fairly.
How Agent Substrate Works
Agent Substrate uses Kubernetes for running a pool of ready pods. It creates a separate control plane that maps a large set of actors (agents and their environments) onto that smaller pool of workers. This approach solves the latency issue inherent in spinning up compute Pods for agents, which involves a trip through the API server, admission, etcd, the scheduler, the kubelet, image pulls, and network setup. That process takes seconds, which is fine for a service that runs for weeks but is far too slow for a sandbox that must appear, run a 10 ms bash command, and get out of the way, hundreds of times per second.
When every worker is busy, the next request waits instead of failing. In an oversubscribed system, such as a coding-assistant platform at peak or an RL rollout burst, a user's next turn can arrive while every worker is busy. Agent Substrate implements request parking in the router to queue turns when all workers are busy. This mechanism holds the request and retries the resume until a worker frees up, with a bounded wait and a cap on how many requests can wait at once. A follow-up contribution ensures that requests to already-running actors never get stuck in that line.
Upstream Work: Security and Resource Management
Each contribution to the project started from one of the use cases described above. All of this work is public in the Agent Substrate repository.
Device Pluggability: Giving agents a GPU inside their sandbox is essential for research workloads and GPU-backed RL environments like simulators. Agent Substrate is working on device-pluggability that adds device support on Kubernetes Dynamic Resource Allocation (DRA). Device vendors plug in their devices through a profile, and each container in a sandbox requests only the devices it needs. This design works for both gVisor and microVM sandboxes. NVIDIA GPUs are the first supported device.
Per-Container Limits: One container cannot starve or kill its neighbors. An agent sandbox often runs the agent's workload next to helper containers for logging, proxying, or metrics. Some of those may be untrusted. A runaway container shouldn't consume the whole sandbox's memory and take the agent down with it. Per-container CPU and memory limits for microVM sandboxes addresses that issue. A container that exceeds its memory limit is stopped on its own, and the rest of the sandbox keeps running.
Preserving GPU Sandboxes: When an agent pauses to wait for human review, its sandbox should release the GPU and resume later with its compiled artifacts and loaded data still in place. Supporting that requires saving and restoring both the agent's state and the GPU state. Restoring GPU state introduces hardware compatibility constraints; a GPU checkpoint can usually be restored only onto the same GPU model, and sometimes only with the same driver version. Agent Substrate proposes a design to address this. Device vendors describe how to pause their devices safely and which device properties a snapshot depends on. Agent Substrate records those properties with each snapshot and restores the sandbox on a matching device, or falls back to a clean cold start if none is available. The design is vendor-neutral and builds on DRA. NVIDIA GPUs are the first target; the design can be extended to other hardware types.
NVIDIA Dynamo: Connecting the Agent Loop
Agent Substrate manages where agents act; inference runtimes manage where they think. Both face a similar efficiency challenge: keeping idle sandboxes warm wastes CPU and memory, while rebuilding model context on repeated model calls wastes GPU cycles. Scaling each layer independently improves utilization, but coordinating them can also reduce latency and preserve reusable context as agents move between tool execution and model calls.
NVIDIA Dynamo, the open-source distributed inference framework, provides that coordination. Agent Substrate activates, suspends, and restores stateful agent environments across a pool of warm sandbox workers. Dynamo orchestrates inference across GPU workers. The Dynamo router sits between them and connects what the agent is doing with what is happening on the GPUs.
Traditional inference routing treats every model call as an independent request. In an agent, each request is one step in a longer loop, and consecutive requests often reuse the same system prompt, tool definitions, and conversational history. NVIDIA Dynamo coordinates inference across GPU workers while Agent Substrate manages agent environments. The Dynamo router uses session identity, KV cache location, and tool-call boundaries to keep related model calls on workers that already hold reusable context.
Dynamo exposes inference signals such as projected time to first token and GPU-pool saturation back to Agent Substrate. Agent Substrate uses these signals to decide whether to keep an actor resident, suspend it during long inference waits, or resume it as the model response nears completion.
Running Agent Substrate with KAI Scheduler
KAI Scheduler places the Substrate worker Pods alongside GPU trainers and inference servers. The combined stack runs on the NVIDIA DSX AI Factory Platform with Substrate as an independent CNCF Sandbox project integrating with DSX OS components. This integration allows teams to build agent infrastructure that coordinates execution, inference, and shared cluster resources effectively.
To deploy a coding assistant on Dynamo and Agent Substrate, teams can follow the sandboxed coding agents guide. This approach ensures that the infrastructure is robust and scalable, handling the complexities of stateful agent environments without the overhead of traditional Kubernetes pod management for every interaction.
Conclusion
Agent Substrate represents a significant advancement in how Kubernetes handles agent workloads. By multiplexing sessions over a small pool of warm workers and leveraging checkpointing technology, it eliminates the cold start penalties and state loss associated with traditional pod-per-session models. The integration with NVIDIA Dynamo ensures that the inference loop is as efficient as the execution loop, preserving context and reducing latency.
As the industry moves toward millions of agents, the ability to safely and efficiently manage these workloads on a shared cluster becomes critical. Agent Substrate, now part of the CNCF ecosystem, provides the open-source foundation for this future, allowing platform teams to focus on building intelligent applications rather than wrestling with infrastructure limitations.
Source:
- Agent Substrate: Advancing Kubernetes for Agents at Scale — NVIDIA AI Blog — https://developer.nvidia.com/blog/agent-substrate-advancing-kubernetes-for-agents-at-scale



