Build a Validated Kubernetes Runtime for NVIDIA Dynamo

Build a Validated Kubernetes Runtime for NVIDIA Dynamo

Teams building AI applications require a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference framework designed to coordinate model serving across GPU workers. However, deploying it on Kubernetes is merely the first step. A successful production deployment demands that GPU drivers, resource allocation, scheduling, networking, and monitoring function as a cohesive unit. Resolving these dependencies often diverts time away from testing model behavior and serving performance.

To address this complexity, the NVIDIA DSX AI Factory Platform introduces the NVIDIA AI Cluster Runtime (AICR). This platform packages NVIDIA Dynamo and its supporting infrastructure into validated recipes. Each recipe defines a specific configuration for an environment and workload, including pinned component versions, compatibility constraints, and validation checks. AICR provides platform teams with a reproducible method to deploy and verify the runtime, ensuring developers have a consistent foundation for serving models.

Why the Underlying Runtime Matters

Running a model with Dynamo on Kubernetes begins with a DynamoGraphDeployment that specifies how engines like vLLM, SGLang, or NVIDIA TensorRT LLM should run in pods. Yet, this graph represents only the top of the stack. A production inference cluster requires a robust runtime layer below it to answer critical infrastructure questions:

• Is the NVIDIA GPU Operator installed with a driver stack that matches the nodes?
• Does the cluster expose GPUs through the expected Dynamic Resource Allocation (DRA) driver?
• Can the scheduler place GPU workers with the required topology and gang scheduling semantics?
• Are node health signals available before workloads land on a degraded node?
• Is the Gateway API / Endpoint Picker (EPP) path configured when the deployment should be Kubernetes-native at ingress?

AICR addresses these concerns by packaging both Dynamo and the cluster runtime components it depends on into a single unit. AICR ships version 1.4.2 of the dynamo-platform chart, which includes the Dynamo operator, runtime images, and resources such as the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition (CRD). This version replaces the former v1alpha1 API version.

The recipe pins component versions and records configuration values, node selectors, tolerations, checksums, and validation expectations to ensure deployment reproducibility. Notably, with Dynamo 1.4, frontends, workers, and other services discover each other through Kubernetes resources. Consequently, the default AICR Kubernetes path no longer uses NATS, and the dynamo-platform chart no longer installs it by default. NATS remains available as an opt-in for deployments that specifically require it.

Routing Paths and Architecture

Dynamo 1.4.2 utilizes TCP for the request plane and ZMQ for KV-cache events. This architecture enables both the native Dynamo Frontend router and the Gateway API Inference Extension (GAIE) / Endpoint Picker Plugin (EPP) path to make cache-aware routing decisions. The choice between these paths depends on the specific topology and management requirements of the deployment.

Path 1: Dynamo Frontend Router

The default path exposes a Dynamo Frontend behind a Kubernetes Service. When running with DYN_ROUTER_MODE=kv, the Frontend tokenizes each request, scores eligible workers using prefix overlap and current load, selects a worker, and dispatches the request through the Dynamo request plane, which uses TCP by default.

This path is suitable when an application team owns a model-specific endpoint or an existing edge proxy only needs to forward traffic to the Dynamo Frontend Service. It represents the smallest topology for deploying and validating a Dynamo graph and is considered the most production-proven serving option. It does not require the additional gateway and EPP components used in Path 2.

In this configuration, the Dynamo Frontend selects a vLLM worker based on KV-cache overlap and load. Requests use the TCP request plane, while worker KV-cache events reach the Frontend through the ZMQ-backed event plane.

Path 2: Gateway API Inference Extension and Dynamo EPP

The second path integrates Dynamo with the Gateway API and Gateway API Inference Extension (GAIE). In the AICR example, agentgateway resolves an HTTPRoute to an InferencePool. The Dynamo operator creates this InferencePool from the DynamoGraphDeployment; its selector identifies eligible worker pods, while its endpointPickerRef identifies the Dynamo EPP Service.

The gateway invokes the Dynamo EPP through GAIE’s endpoint-selection interface. The EPP tokenizes the request and uses Dynamo’s KV router to select an endpoint. It returns the selection and routing metadata to the gateway implementation, which acts as the actual HTTP proxy. In this mode, each routable worker pod includes a Dynamo Frontend sidecar that receives the request from the gateway and forwards it to the local worker container. The Frontend runs with --router-mode direct, applying the EPP selection without rerouting.

This path is suitable when the platform team manages shared Kubernetes ingress and needs to configure model endpoint worker selection, timeouts, retries, and fallback through Gateway API mechanisms. Note that both routing approaches can work behind a Gateway API HTTPRoute. The difference lies in which component manages the transport to the worker: the Dynamo Frontend or the Gateway itself.

The AICR Recipe Flow

AICR separates runtime setup from application deployment through a five-step flow designed to ensure stability and correctness before application workloads are installed.

Step 1: Snapshot the Cluster

The process begins by capturing the current state of the cluster to establish a baseline for the deployment.

Step 2: Generate the Dynamo Inference Recipe

Using the snapshot, the system generates a Dynamo inference recipe tailored to the specific environment and workload requirements.

Optional Offline Check

Teams may perform an optional offline check to verify the recipe configuration before proceeding to deployment.

Step 3: Render the Runtime Bundle

The validated recipe is rendered into a deployment bundle, preparing all necessary resources and configurations.

Step 4: Deploy the Runtime

The runtime is deployed using tools such as Helmfile or GitOps. This step applies the configuration to the cluster, installing the operator, images, and supporting infrastructure.

Step 5: Validate the Installed Runtime

Before installing application workloads, the installed runtime must be validated to ensure all dependencies are met and the system is ready for inference.

What AICR Validation Checks

AICR validation ensures that the infrastructure meets the rigorous demands of distributed inference. The validation process checks for the presence and correctness of the NVIDIA GPU Operator, the matching driver stack, the exposure of GPUs through the Dynamic Resource Allocation (DRA) driver, and the availability of node health signals. It also verifies that the Gateway API or Endpoint Picker path is correctly configured as required by the chosen routing topology.

By automating these checks, AICR allows platform teams to focus on application logic rather than debugging infrastructure dependencies. This approach supports the broader goal of the NVIDIA DSX AI Factory Platform: to help the ecosystem design, build, and operate more efficient AI factories.

Conclusion

Building a validated Kubernetes runtime for NVIDIA Dynamo requires more than simply deploying a model graph. It necessitates a coordinated approach to infrastructure management, routing strategy, and validation. The NVIDIA AI Cluster Runtime (AICR) provides the necessary framework to package Dynamo and its dependencies into reproducible recipes. Whether utilizing the native Dynamo Frontend router for model-specific endpoints or the Gateway API Inference Extension for shared ingress, AICR ensures that the underlying runtime is robust, verified, and ready for production workloads.

For teams looking to streamline their AI factory operations, exploring the NVIDIA AI Cluster Runtime offers a structured path to configuring and validating inference infrastructure efficiently.

Source: https://developer.nvidia.com/blog/build-a-validated-kubernetes-runtime-for-nvidia-dynamo