GPU Management: Why Idle GPUs Are the New Grounded Aircraft

GPU Management: Why Idle GPUs Are the New Grounded Aircraft

Walk into any AI lab today, and you'll see a familiar sight: racks of expensive GPUs, fans spinning softly at idle, waiting for a job that will never come. It is a scene eerily reminiscent of a fleet of grounded aircraft sitting on a tarmac — enormous capital assets burning value every second they sit unused. A plane flying generates revenue; a plane parked generates only costs. The same logic applies to GPUs. Yet most organizations manage their GPU fleets with the same sophistication a minor airline used for plane maintenance in the 1980s: lots of hardware, little telemetry, and no dynamic scheduling.

This article is a practical field guide to fixing that problem. We'll look at why idle GPUs are so expensive, what tooling you need to measure the problem, and how to install, configure, and use a lightweight monitoring and scheduling stack on your own infrastructure. By the end, you'll have concrete commands that turn your GPU fleet from a sunk cost into a measured, shared, and efficiently utilized asset.

The Hidden Cost of Parked Silicon

The analogy between idle GPUs and grounded aircraft is not an exaggeration. Consider what a modern AI accelerator actually represents. A high-end data-center GPU can cost tens of thousands of dollars. Its power supply, cooling, and network infrastructure double that cost. When you amortize that over a three-year hardware lifecycle and factor in electricity and system administration, every hour a GPU sits idle is a direct drain on your operational budget.

Yet the industry's attention is almost exclusively on *training big models*. The Hugging Face blog, one of the most consistent voices on GPU infrastructure, has repeatedly emphasized that efficient GPU management is as critical as model design. They've highlighted that the gap between theoretical peak performance and achieved utilization is massive, and that tooling — not architecture — is what closes it. The blog has become a reliable reference for the community's shift toward measuring, sharing, and optimizing every available accelerator.

A grounded aircraft at least has the excuse of a storm or a technical inspection. An idle GPU often has no excuse at all. It is idle because a queue is poorly engineered, because nobody knows it is free, or because a developer reserved it "just in case" and then went to lunch. That last one is the real killer. GPU reservation without scheduling is the equivalent of a pilot keeping a plane on the tarmac because they *might* want to fly later.

What "Idle" Actually Means

Before we fix the problem, we need to measure it. And for GPUs, "idle" is a more nuanced term than you might think.

• **SM Utilization**: The fraction of streaming multiprocessor cycles that are actually doing computation. A GPU showing 100% SM utilization is theoretically busy, but it may still be memory-bound.
• **Memory Utilization**: The percentage of device memory in use. High memory usage with low SM activity often indicates poor batching and a job waiting on data transfers.
• **Power Draw**: A good proxy for overall activity. An idle NVIDIA A100 might draw 30–50 watts; a fully loaded one draws 250–400 watts.
• **Compute vs. Copy Engines**: If the copy engines are saturated while the compute engines are idle, you are feeding the GPU too slowly.

The tools we install next will show you all these metrics. After a week of data collection, you'll see your own "grounded fleet" clearly: perhaps one GPU at 90% utilization and seven others at 5%, because your scheduler was a wiki page and a prayer.

Requirements

For this guide, you will need the following:

• One or more Linux servers with NVIDIA GPUs (Ampere or newer is recommended, but older architectures work too).
• NVIDIA driver installed and working. Verify with `nvidia-smi`. If the command is missing, install the driver from your distribution's repository first.
• Python 3.8 or later, plus `pip`.
• Basic familiarity with the command line and systemd.
• Optional: a Kubernetes cluster if you want to follow the advanced orchestration examples. For the core article, a single server is enough.

We will build our management stack from three components:

1. `nvidia-smi` — the standard NVIDIA command-line tool for querying GPU state. 2. `pynvml` — an official NVIDIA Python binding that lets us write monitoring scripts. 3. `cron` and `systemd` — the operating system's built-in schedulers, which become surprisingly powerful when combined with the GPU state we collect.

Optionally, we will install Ollama to serve local LLM inference as a way to fill idle gaps with useful work. Ollama's blog has documented how easy it is to turn a spare GPU into a self-hosted model server, and both Mistral AI News and the Meta AI Blog regularly announce new open-weights models that you can serve locally. Used as a periodic "background workload," this turns idle GPU time into a feature, not a cost.

Step-by-Step Installation

Step 1: Verify and install the NVIDIA driver stack

First, confirm your GPU is visible to the system. Run:

nvidia-smi

If the command is not found, install it. On Ubuntu 22.04 or later:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo reboot

After reboot, verify again:

nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used --format=csv

You should see a table listing each GPU index, name, current utilization, and memory usage. This is your baseline.

Step 2: Install the Python bindings

Next, install `pynvml`. This official NVIDIA library gives us programmatic access to all the telemetry visible in `nvidia-smi`. We'll also install `psutil` for system-level metrics:

pip install nvidia-ml-py psutil

Note: the package name is `nvidia-ml-py`, but the import is `pynvml`. This is a common source of confusion.

Step 3: Write a monitoring script

Create a file named `gpu_monitor.py`:

#!/usr/bin/env python3
import pynvml
import time
import csv
import os

pynvml.nvmlInit()
device_count = pynvml.nvmlDeviceGetCount()
log_file = "gpu_utilization.csv"

def log_gpu_state():
    with open(log_file, "a", newline="") as f:
        writer = csv.writer(f)
        writer.writerow([time.time()] + [get_device_state(i) for i in range(device_count)])

def get_device_state(index):
    handle = pynvml.nvmlDeviceGetHandleByIndex(index)
    util = pynvml.nvmlDeviceGetUtilizationRates(handle)
    memory = pynvml.nvmlDeviceGetMemoryInfo(handle)
    power = pynvml.nvmlDeviceGetPowerUsage(handle)
    return f"{util.gpu}%|{util.memory}%|{memory.used / 1024**2:.0f}MiB|{power / 1000:.1f}W"

if __name__ == "__main__":
    while True:
        log_gpu_state()
        time.sleep(30)  # Sample every 30 seconds

Test the script:

python3 gpu_monitor.py &

Let it run for a minute, then inspect the output:

cat gpu_utilization.csv

You'll see rows like `1724733600.0, 3%|5%|512MiB|38.2W, 98%|100%|79123MiB|392.1W`. If your GPUs look like the first entry rather than the second, you have a grounding problem.

Step 4: Set up a systemd service for persistent telemetry

A monitor running in a foreground shell is not a monitoring solution. Create a service so your telemetry survives reboots:

sudo nano /etc/systemd/system/gpu-monitor.service

Add the following:

[Unit]
Description=GPU utilization logger
After=multi-user.target

[Service]
WorkingDirectory=/opt/gpu-monitor
ExecStart=/usr/bin/python3 /opt/gpu-monitor/gpu_monitor.py
Restart=always

[Install]
WantedBy=multi-user.target

Now move your script, enable, and start the service:

sudo mkdir -p /opt/gpu-monitor
sudo mv gpu_monitor.py /opt/gpu-monitor/
sudo systemctl daemon-reload
sudo systemctl enable --now gpu-monitor

Congratulations. You now have a continuous, historical record of every GPU's utilization, memory, and power draw — the equivalent of a flight data recorder for your accelerator fleet.

Step 5: Install Ollama for background inference

Ollama is the fastest way to turn idle cycles into useful work. It is a single binary that wraps model serving for local LLMs, and its blog has documented a growing ecosystem of models you can pull and run. To install it, run:

curl -fsSL https://ollama.com/install.sh | sh

The installation script will set up a systemd service for you. Verify it:

sudo systemctl status ollama

Then pull a model. Because both Mistral AI and Meta publish open-weights models that Ollama supports, you can choose depending on your memory budget:

ollama pull llama3

Or, for a smaller model that runs on a single GPU with minimal memory:

ollama pull mistral

Once a model is pulled, the serving process remains resident in GPU memory, waiting for a request. When your training jobs are running, you can stop Ollama or migrate it to another GPU with `CUDA_VISIBLE_DEVICES`. This turns your "idle" GPUs into a private inference endpoint — valuable to developers, QA teams, and document-processing pipelines across the company.

Usage Examples

Example 1: Detect and kill zombie GPU processes

The most destructive pattern in any GPU fleet is a process that holds memory but does no computation — a landed aircraft with the engines off but the parking brake on. Use this command to list processes holding GPU memory:

nvidia-smi --query-compute-apps=pid,used_memory,process_name --format=csv

If you see stale `python3` processes, identify their parent and terminate them:

ps aux | grep <pid>
sudo kill <pid>

To make this a recurring routine, add it to `cron`:

crontab -e

Add the following line to run the kill script every hour:

0 * * * * /opt/gpu-monitor/cleanup_gpus.sh

The script contents can be a simple loop that finds processes using more than zero memory but with a GPU utilization of 0% for more than 10 minutes. This is the closest thing to an automated ground-crew inspection your GPUs have ever had.

Example 2: Dynamic scheduling with `CUDA_VISIBLE_DEVICES`

The single most effective discipline for GPU management is explicit device assignment. Instead of letting PyTorch or TensorFlow pick a GPU at random, always set the environment variable:

CUDA_VISIBLE_DEVICES=2 python3 train.py

If you run a multi-GPU training job, constrain the visible set:

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 train.py

Combine this with a quick Python check to find the least-loaded GPU before launching a job:

import pynvml
pynvml.nvmlInit()
best_idx = -1
best_util = 101
for i in range(pynvml.nvmlDeviceGetCount()):
    handle = pynvml.nvmlDeviceGetHandleByIndex(i)
    util = pynvml.nvmlDeviceGetUtilizationRates(handle).gpu
    if util < best_util:
        best_util = util
        best_idx = i
print(best_idx)

Save this as `pick_gpu.py` and wrap your training command:

export BEST_GPU=$(python3 pick_gpu.py)
CUDA_VISIBLE_DEVICES=$BEST_GPU python3 train.py

This simple pattern eliminates the worst form of idle GPU: the reserved-but-unused card.

Example 3: Fill idle windows with Ollama batch inference

Imagine you have a GPU that is idle every night from midnight to 6 AM. Instead of letting that asset sit on the "tarmac," schedule a batch job that uses Ollama to process a queue of pre-existing requests. Here is a small Python batch client:

import requests
import json

with open("texts.json") as f:
    texts = json.load(f)

for text in texts:
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": "mistral", "prompt": f"Summarize: {text}", "stream": False}
    )
    result = response.json()
    print(result.get("response", ""))

Run this with a cron job:

0 0 * * * /usr/bin/python3 /opt/batch_inference.py >> /var/log/batch_inference.log 2>&1

Your GPU is now doing useful work during an otherwise dead window. That is the exact difference between a grounded aircraft and a working one.

Example 4: Alerting on GPU starvation

Write a script that checks the last 10 lines of your CSV log and raises a metric if utilization has been below 10% for over an hour:

import sys
import csv

rows = list(csv.reader(open("/opt/gpu-monitor/gpu_utilization.csv")))
if len(rows) < 20:
    print("Not enough samples yet.")
    sys.exit(0)

# Parse GPU 0 utilization from the last 20 samples
recent_u = [float(r[1].split("|")[0].rstrip("%")) for r in rows[-20:]]
avg_util = sum(recent_u) / len(recent_u)
if avg_util < 10:
    print(f"ALERT: GPU 0 average utilization {avg_util:.1f}% over the last 10 minutes.")
else:
    print(f"OK: GPU 0 average utilization {avg_util:.1f}%.")

If you want a mature alerting pipeline, export these numbers to Prometheus and visualize in Grafana. The CSV approach, however, is elegant because it requires zero additional infrastructure — no Kubernetes, no exporters, no time-series database.

Beyond the Single Server: Fleet Management

The commands above work well for one machine. In a Kubernetes cluster, the problem is solved at the platform level. The official NVIDIA device plugin for Kubernetes exposes GPUs as schedulable resources, and operators like the NVIDIA GPU Operator extend this to MIG (Multi-Instance GPU) slicing. Hugging Face's blog has covered these patterns extensively, positioning them as core to cost-efficient model serving and training at scale.

If your organization is not on Kubernetes yet, the discipline is still applicable: standardize on `CUDA_VISIBLE_DEVICES`, enforce a wall-clock reservation system, and log utilization everywhere. The tooling is secondary to the habit of measurement.

Conclusion

An idle GPU is a grounded aircraft: an enormous capital asset performing no revenue-generating work while it burns electricity, occupies cooling capacity, and occupies the balance sheet. The fix is not to buy more GPUs. The fix is to install a monitoring loop, force explicit device scheduling, and fill genuinely idle windows with background inference jobs.

The commands in this article are deliberately simple. A Python script, a cron job, a systemd service, and a local model server. Together, they give you what most AI organizations lack: visibility. With visibility comes utilization; with utilization comes a return on the millions of dollars you've already spent. Your fleet is sitting on the tarmac, waiting for a pilot. You now have the cockpit instruments to fly it.