Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

As AI workloads become more diverse, the demand for flexible GPU resource management in Kubernetes environments grows. Many applications require only a fraction of a GPU's capacity, yet traditional allocation methods often assign entire devices, leaving significant capacity idle. NVIDIA addresses this challenge with CUDA 13.4, introducing memory boundary enforcement that is transparent to applications. This update allows for exact memory amounts, guaranteed minimums with burstable maximums, and a choice between time-slicing and sm-sharing compute modes.

To operationalize these capabilities at scale, NVIDIA has released the open-source GPU Fractioning Operator. Hosted under the KAI Scheduler organization on GitHub, this operator runs alongside the NVIDIA GPU Operator. Its primary function is to set up and supervise Multi-Process Service (MPS) on every GPU node, apply per-workload memory limits and compute modes, and measure memory and compute usage per GPU fraction rather than per whole GPU.

The Architecture of Enforcement

CUDA 13.4 provides the underlying memory enforcement capability, but a Kubernetes cluster cannot utilize it independently. On every GPU node, a mechanism is required to apply each Pod container's memory limits and compute mode before the process starts and to report actual usage per fraction. This work belongs on the node, independent of whichever scheduler placed the workload.

The GPU Fractioning Operator fulfills this role. It reads resource and mode annotations written in the Pod YAML by the scheduler. Once the scheduler selects the GPUs for a workload and records its memory allocation and compute-sharing mode in per-container Pod annotations, the GPU Fractioning Operator configures the assigned GPUs to enforce those limits.

KAI Scheduler v0.18.0 is the first scheduler integrated with the GPU Fractioning Operator. Starting with this version, KAI Scheduler includes a GPU-sharing mode called NvFractions. This mode picks which slice of which GPU each workload gets and writes the necessary annotations. Administrators can enable NvFractions during KAI Scheduler installation with a Helm flag, which also installs the GPU Fractioning Operator as a dependency.

Configuring Memory Requests and Limits

Workloads can declare fractional GPU requirements using specific annotations. There are four critical annotations that answer four fundamental questions: how much memory is needed, how far the workload may grow, how compute is shared, and how many GPUs it spans.

Administrators can specify memory needs as an absolute size using standard Kubernetes units. The container name is part of the key to ensure the annotation targets the correct container. Alternatively, workloads can declare a portion of a GPU. This ability allows a workload to define a guaranteed minimum (the request) and a maximum it may burst into when memory is free.

If a limit annotation is omitted, the limit equals the request, and the workload is capped at exactly what it asked for. A portion limit must be greater than the gpu-fraction and less than 1.

Compute Modes: Time-Slicing vs. SM-Sharing

CUDA 13.4 introduces two distinct compute modes for sharing GPUs, each with different performance and isolation characteristics.

Time-Slicing is the default mode. It is serial, meaning workloads take turns on the whole GPU. When every workload uses its full slice, each gets roughly 1/n of the time. Any time left unused by one workload is shared equally between the others. Crucially, each workload runs in its own error domain, providing full error isolation. If one job crashes, it does not affect its neighbors.

SM-Sharing (Streaming Multiprocessor sharing) is parallel. In this mode, the GPU's streaming multiprocessors are split between workloads, and all run at once continuously. Nothing is carved out in advance; the GPU hardware allocates and releases SMs as workloads need them. When every workload is busy, the split settles at roughly 1/n. However, if one workload goes quiet, its SMs go to the others, allowing a single active workload to use them all. In this mode, workloads share a single error domain.

Because a shared GPU runs only one compute mode at a time, KAI Scheduler never places a time-slicing workload and an sm-sharing workload on the same GPU. The scheduler tracks this per GPU, not per node, so one node can host both modes on different GPUs.

Practical Implementation Examples

The following examples demonstrate how to configure Pods for different sharing scenarios using KAI Scheduler annotations.

Inference Server with SM-Sharing

This example represents an inference server split across four GPUs. It requires a guaranteed 40Gi on each GPU and can use up to 60Gi when neighbors are idle. It runs alongside other workloads instead of waiting its turn.

apiVersion: v1
kind: Pod
metadata:
  name: inference
  labels:
    kai.scheduler/queue: default-queue
  annotations:
    nvidia.com/container.app.gpu-memory.request: "40Gi"
    nvidia.com/container.app.gpu-memory.limit: "60Gi"
    nvidia.com/container.app.gpu-compute.mode: "sm-sharing"
    gpu-fraction-num-devices: "4"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: app
      image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04

Development Job with Time-Slicing

This example is suitable for a development or batch job, such as notebooks or small fine-tuning runs. It needs a guaranteed half of each of two GPUs and can grow to 80% when memory is free. Time-slicing ensures error isolation between jobs.

metadata:
  annotations:
    gpu-fraction: "0.5"
    gpu-fraction.limit: "0.8"
    nvidia.com/container.app.gpu-compute.mode: "time-slicing"
    gpu-fraction-num-devices: "2"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: app

Getting Started and Community

To utilize these features, administrators must ensure KAI Scheduler is installed with the global.nvFractions.set=true flag. This configuration enables the NvFractions mode and installs the GPU Fractioning Operator.

The GPU Fractioning Operator and KAI Scheduler are both available on GitHub under the KAI Scheduler organization. Installation and configuration instructions for all three sharing mechanisms are available in the admin and user guides. Developers can also join the Slack channel or attend bi-weekly community meetings to ask questions and contribute to the project.

For more information about CUDA 13.4, refer to the official announcement. Additionally, NVIDIA will be present at KubeCon + CloudNativeCon NA 2026 in Salt Lake City, Utah, from November 9-12.

Conclusion

Memory-enforced GPU sharing in Kubernetes represents a significant advancement in resource efficiency for AI workloads. By combining the GPU Fractioning Operator with KAI Scheduler v0.18.0 and CUDA 13.4, platform teams can enforce per-workload memory limits and choose between time-slicing and sm-sharing compute modes. This flexibility allows organizations to serve more workloads with existing infrastructure while maintaining strict control over memory usage and compute allocation.

Source: https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4