Impactful Scheduling for GPU Clusters: Queue Design, Preemption, and Fairness in Local Model Workloads

Impactful Scheduling for GPU Clusters: Queue Design, Preemption, and Fairness in Local Model Workloads

A GPU cluster's throughput is decided at admission time, not at kernel time. A perfectly fused attention kernel buys you a few percent; a queue that admits the wrong job at the wrong moment costs you an entire node for eight hours. Yet in most self-hosted AI environments, the scheduler is the least designed component: a default queue, a first-come-first-served policy, and a hope that demand stays below capacity.

This article is a practical synthesis of the design space around queue design, preemption, and fairness for local model workloads — fine-tuning runs, batch inference, evaluation sweeps, and interactive notebooks sharing the same physical GPUs. The Allen Institute for AI's post on the Hugging Face blog, Impactful scheduling for GPU clusters (link), frames scheduling as a lever for getting more useful work out of capacity you already own. That framing is the starting point here; the concrete patterns below are general operational practice, not claims extracted from that post.

Verified context and interpretation

Before the mechanics, a boundary note. The verified background for this article is a single primary source: the Hugging Face blog post linked above. It is a general treatment of GPU cluster scheduling rather than a specification, so nothing below should be read as a feature list, benchmark, or configuration reference taken from it. Where I describe Kubernetes objects, SLURM parameters, or Python patterns, these are widely used, independently documented tools. Treat the commands as illustrative skeletons to adapt to your own environment and to the documentation of whatever version you install.

Why local model workloads break naive queues

General-purpose batch schedulers were designed around a simple assumption: jobs are roughly interchangeable, and running them in arrival order is close enough to optimal. Local model workloads violate that assumption in three ways.

Jobs are heterogeneous in shape. A 7B LoRA fine-tune might want one GPU for forty minutes. A full fine-tune of a larger model may need eight GPUs on one node for a day, with all eight starting simultaneously or none starting at all. A batch inference sweep may want to fan out across every idle GPU for three minutes. A single FIFO queue serving all three will routinely head-of-line block.

Jobs are heterogeneous in elasticity. Inference throughput scales gracefully with GPU count; distributed training does not. Admitting half of a training job is worse than admitting none of it, because partially placed workers occupy memory and network bandwidth while making no progress.

Demand is bursty and correlated. Research groups tend to submit in waves — a deadline, a paper, an evaluation campaign. A cluster sized for average load will be visibly idle half the time and unusable the other half. Queueing theory does not fix this; admission policy does.

Requirements

Before changing scheduling behavior, confirm the substrate. The list below is generic enough to cover both Kubernetes-based and SLURM-based clusters.

• GPU nodes with working device exposure. On Kubernetes, the NVIDIA device plugin or GPU Operator must be running and advertising nvidia.com/gpu (or an equivalent MIG or time-sliced resource name). On SLURM, GresTypes=gpu must be configured and --gres=gpu:N must allocate correctly.
• A queueing or admission layer. On Kubernetes this is typically Kueue, Volcano, or a custom admission webhook. On SLURM it is partitions plus QOS rather than a separate component.
• Reliable checkpoint storage. Preemption without checkpointing converts a scheduling policy into a data-loss policy. NFS, a CSI-backed volume, or object storage mounted into the training container all work; the requirement is that writes are atomic and fast enough not to dominate step time.
• Metrics. Queue depth, admission latency, preemption counts, and per-namespace GPU-seconds consumed. Without these, fairness is a matter of opinion.
• Resumable workload code. Your training script must handle SIGTERM and persist state. This is a code requirement, not an infrastructure one, and it is the most commonly skipped step.

Queue design

Queue design answers a single question: when a job arrives and capacity is scarce, who gets to run, and in what order?

Make the queue hierarchy match the org chart

A flat queue forces every decision to be made by priority number alone, which is impossible to tune. A two-level hierarchy — a cluster-level resource pool and namespace-level queues — is easier to reason about and easier to audit. Each team gets a queue with a nominal quota; the cluster queue aggregates the quotas and defines what happens when a team exceeds its share.

Decide the admission unit before the priority scheme

If your queue admits pods, a distributed training job can be partially scheduled. If it admits whole jobs, it cannot. Gang admission — all-or-nothing placement — is mandatory for synchronous training and strongly recommended for anything using collective communication. The cost is that a gang job waits for the largest contiguous block of GPUs rather than the sum of scattered free ones; that is usually the right trade.

Separate nominal quota from borrowing

A useful pattern: each queue has a guaranteed share that it can always claim, and a borrowing limit that lets it consume idle capacity from other queues. Borrowing makes the cluster look busy without permanently reallocating it. The critical design question is whether borrowed capacity is reclaimable — whether the lending queue can preempt the borrower when it needs its own quota back. If yes, borrowers must be checkpointable. If no, "borrowed" is a polite word for "given away."

Watch for the head-of-line blocking anti-pattern

A single queue with a strict priority ordering and no borrowing produces a predictable pathology: one large low-priority job sits at the head, every small high-priority job queues behind it, and utilization collapses while the cluster looks full. The fix is structural — either priority-aware admission that allows lower-priority work to start when it fits, or work-conserving borrowing with reclamation.

Preemption

Preemption is how a priority scheme becomes real. Without it, priorities are advisory labels that influence nothing.

Three levels of preemption

• Pod-level eviction. The scheduler deletes individual pods to free a node. Cheap to implement, dangerous for gang jobs, because evicting one worker of a synchronous job leaves the rest idle until the job is rescheduled.
• Job-level preemption. The entire workload is suspended and requeued. Correct for training, slightly wasteful for inference.
• Checkpoint-based preemption. The preempted workload receives a termination signal, writes state, and exits cleanly. This is the only variant that does not implicitly accept lost work.

Preemption has a price — model it

The cost of preemption is not the eviction; it is the recomputation since the last checkpoint. If you checkpoint every thirty minutes and preempt a job that has run for twenty-nine, you have destroyed twenty-nine minutes of GPU-seconds to reclaim the node. Two levers control this:

1. Grace period. Give the workload enough time to checkpoint. A twenty-second grace period combined with a checkpoint that takes ninety seconds is a corrupted checkpoint waiting to happen.
2. Minimum runtime before preemption. Some schedulers let you require that a job run for at least N minutes before it becomes a preemption candidate. This trades a small amount of scheduling responsiveness for a large reduction in wasted work.

Make preemption a first-class contract with the workload

The scheduler cannot checkpoint for you. The workload must implement it. That means handling SIGTERM, writing atomically, and being honest about how long the save takes.

Fairness

Fairness is the most misunderstood of the three pillars, mostly because people evaluate it instantaneously.

Fairness is time-averaged, not instantaneous

A queue with zero GPUs right now is not being treated unfairly if it consumed 60% of the cluster yesterday. Fair-share schedulers track decayed usage: recent consumption counts heavily, old consumption decays toward zero. Two consequences follow. First, a team that has been idle for a week correctly has a higher effective priority today. Second, if you monitor fairness with instantaneous gauges, your dashboard will show constant alarm and your operators will constantly "fix" a system that is working.

Weighted shares over equal shares

Equal shares assume each team's work is equally valuable, which is rarely true. Weighted shares let you express "the research group gets twice the share of the platform team" as a number rather than as an argument. The scheduler then allocates proportionally to weight when demand exceeds supply.

Starvation and aging

Strict priority ordering with no aging starves low-priority work indefinitely. Aging — increasing effective priority with waiting time — bounds the maximum wait. Where aging is not available, set a maximum wait target per queue and treat breaches as a policy failure to investigate, not as an acceptable cost of doing business.

Fairness across resources, not just GPUs

A job can be GPU-light and CPU-heavy, or GPU-heavy and storage-heavy. Dominant-resource fairness extends naturally: compute each job's share as its maximum ratio across all constrained resources, not just GPUs. On a cluster where preprocessing saturates CPUs, GPU-only fairness accounting will produce surprising bottlenecks.

Step-by-step installation

The sequence below assumes a Kubernetes cluster with GPU nodes; a SLURM alternative follows. Adapt names and versions to your environment.

1. Confirm the GPUs are visible as schedulable resources.

kubectl get nodes -L nvidia.com/gpu

This lists nodes with a nvidia.com/gpu column showing allocatable GPU count. If the column is empty, fix device exposure before touching scheduling.

2. Install the queueing layer. The example uses Kueue, a Kubernetes SIG project for job queueing with quotas, borrowing, and preemption.

kubectl apply --server-side -k "github.com/kubernetes-sigs/kueue/config/default?ref=main"

This installs the controller and its CRDs. Pin ref to a released tag in production rather than tracking main.

3. Verify the controller is running.

kubectl get pods -n kueue-system

The controller manager pod should reach Running. Check its logs if admission webhooks fail to register.

4. Define a cluster queue with nominal quota and reclaimable borrowing. Replace the resource names and quantities with your own.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: default
          resources:
            - name: "cpu"
              nominalQuota: 200
            - name: "memory"
              nominalQuota: 800Gi
            - name: "nvidia.com/gpu"
              nominalQuota: 8

The nominalQuota values are the guaranteed floor. Groups of cluster queues can be placed in a cohort so idle quota is borrowable across them; check the CRD version installed in step 2, since field names differ between releases.

5. Create a namespaced queue that points at the cluster queue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: team-a
  namespace: team-a
spec:
  clusterQueue: cluster-queue

Teams submit to their local queue; the cluster queue enforces the global policy.

6. Mark jobs as queue-managed and suspended. A batch Job with the queue label will be held by the controller until admission.

apiVersion: batch/v1
kind: Job
metadata:
  name: finetune-001
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: team-a
spec:
  suspend: true
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: trainer
          image: your-registry/trainer:latest
          resources:
            limits:
              nvidia.com/gpu: 1

The suspend: true field is what allows the queue controller to decide when the pod is created. Setting backoffLimit: 0 prevents Kubernetes from racing the scheduler with its own retry loop.

SLURM alternative. On a SLURM cluster, define priority tiers at the partition level and use QOS for preemption.

# slurm.conf fragment — partition priority drives preemption across tiers
# PreemptType=preempt/partition_prio
# PartitionName=interactive PriorityTier=1000 PreemptMode=CANCEL MaxTime=04:00:00 Nodes=ALL
# PartitionName=batch       PriorityTier=1    PreemptMode=REQUEUE MaxTime=7-00:00:00 Nodes=ALL

PreemptMode=REQUEUE returns the preempted job to the queue rather than killing it, which is the correct default for training workloads that cannot checkpoint quickly. Validate the exact parameter names against your SLURM version before rolling this out.

Usage examples

Example 1: submit and observe queue admission

kubectl -n team-a apply -f finetune-001.yaml
kubectl -n team-a get workloads
kubectl -n team-a describe workload finetune-001

The first command submits the job; the second shows whether it is admitted or pending; the third explains why — insufficient quota, unschedulable flavor, or waiting on a gang constraint.

Example 2: preemption-aware checkpointing

This pattern handles SIGTERM, writes atomically via a temporary file, and exits cleanly so the job can be requeued.

import os
import signal

import torch

CHECKPOINT_PATH = os.environ.get("CHECKPOINT_PATH", "/checkpoints/latest.pt")
_stop = False


def _handle_sigterm(signum, frame):
    global _stop
    _stop = True


signal.signal(signal.SIGTERM, _handle_sigterm)


def save(state: dict) -> None:
    """Write atomically so a crash mid-write never corrupts the checkpoint."""
    tmp = f"{CHECKPOINT_PATH}.tmp"
    torch.save(state, tmp)
    os.replace(tmp, CHECKPOINT_PATH)


def maybe_stop(model, optimizer, step: int) -> bool:
    if _stop:
        save({"step": step, "model": model.state_dict(),
              "optimizer": optimizer.state_dict()})
        return True
    return False

Call maybe_stop at the end of every step. The scheduler's grace period must exceed the time save takes; measure it rather than guessing.

Example 3: sanity-checking a weighted fair-share policy

Before trusting a scheduler's fairness behavior, verify your weights produce the ordering you expect. This small script is a teaching model, not a scheduler.

from dataclasses import dataclass


@dataclass
class Queue:
    name: str
    weight: float
    gpu_seconds_recent: float
    waited_seconds: float = 0.0


def admission_order(queues, aging_rate: float = 0.01):
    """Lower score = admitted first. Aging rewards long waits."""
    def score(q: Queue) -> float:
        share = q.gpu_seconds_recent / max(q.weight, 1e-9)
        return share - aging_rate * q.waited_seconds

    return sorted(queues, key=score)


queues = [
    Queue("team-a", weight=1.0, gpu_seconds_recent=3600),
    Queue("team-b", weight=2.0, gpu_seconds_recent=3600, waited_seconds=600),
    Queue("team-c", weight=1.0, gpu_seconds_recent=0, waited_seconds=120),
]

for q in admission_order(queues):
    print(q.name)

Run it with different weights and aging rates to see how sensitive the ordering is. A policy that flips ordering on a 0.001 change in aging_rate is not a policy you can explain to your users.

What to measure

Four signals tell you whether the design is working: p95 queue wait time per queue (responsiveness), GPU-seconds admitted per GPU-second available (utilization), preemptions per hour and checkpoint loss per preemption (the cost of your priority scheme), and maximum wait time for the least-served queue (starvation). If the fourth drifts upward while the first three look fine, your fairness policy has quietly become a priority policy.

Conclusion

Scheduling is the highest-leverage lever in a local GPU cluster because it acts before any workload runs. The three mechanisms — queue design, preemption, and fairness — are not independent knobs. Queues define the units that can be preempted; preemption defines whether a queue's guarantees are credible; fairness defines whether the queueing scheme survives contact with real teams over months rather than days.

Start with the smallest change that fixes a measured problem: gang admission for one synchronous training job, or a grace period long enough to cover a checkpoint. Then model the cost of preemption before enabling it, measure fairness over a decayed window rather than instantaneously, and treat the workload's ability to checkpoint as a prerequisite rather than an afterthought. The source post is a useful starting frame for that conversation; the specific policy is yours to design.