How to Choose Full-Stack Observability for NVIDIA AI Factories

How to Choose Full-Stack Observability for NVIDIA AI Factories

When an AI factory spans thousands of NVIDIA GPUs, an InfiniBand fabric, distributed storage, and a scheduler that launches batch jobs around the clock, an ordinary monitoring stack stops working. You no longer have a few servers with a handful of metrics; you have a machine where every layer — silicon, firmware, driver, container runtime, framework, and orchestration — can become a bottleneck on the same day. A single silent behavior, like a GPU that begins throttling memory clocks, can slow down an entire training run, yet none of your application-level dashboards will catch it.

Full-stack observability for NVIDIA AI factories is not just a bigger dashboard. It is a deliberate choice about what data to collect, how to join that data across layers, and who will act on it. This article lays out a practical decision framework and shows how to stand up a minimal, GPU-aware reference stack that demonstrates the full-stack pattern with real commands.

What "Full-Stack" Means in an AI Factory

In traditional IT, full-stack observability usually means linking the application, the runtime, and the host OS. An AI factory adds several layers that are easy to miss:

• The silicon layer: GPU utilization, memory utilization, temperature, power draw, clock throttling, and errors surfaced through NVIDIA's NVML and DCGM interfaces.
• The system and fabric layer: CPU and host memory, NVMe health, PCIe link issues, and the InfiniBand or RoCE fabric that connects GPUs to each other and to storage. A degraded link can cause collective operations to slow down without a single error appearing in application logs.
• The runtime and orchestration layer: Kubernetes or Slurm scheduling behavior, container health, GPU time slicing or MIG isolation, and the queue waits that precede every training run.
• The application and framework layer: how many training steps per second the workload is achieving, data loader throughput, loss curves, and the health of inference endpoints from Triton or a custom serving container.

The central design decision is correlation. Metrics collected independently at each layer are almost useless in an AI factory because production problems are usually cross-layer. A low GPU utilization number should be joined with the data-loader metrics to determine whether the GPUs are idle because of I/O starvation or because the scheduler is throttling the job. Choosing observability, therefore, means choosing a model for how telemetry from different layers will be connected.

Requirements to Settle Before You Compare Tools

Before evaluating any product, write down the concrete requirements. Four are especially important for AI factories.

First, define the SLOs you actually care about. For training, the main service-level objective is steady forward progress: steps per second must stay above a threshold, and the job must not be silently slowing down. For inference, the SLOs are latency, throughput, and the GPU utilization of the serving fleet. Observability tools should be assessed against whether they can produce these exact signals, not just a generic "system healthy" status.

Second, decide whether you need real-time or post-hoc visibility. An AI factory is a high-cardinality environment: every job has its own set of containers, GPUs, and processes. You need to decide how many metrics per second the pipeline can sustain and for how long data must be retained. Full-fidelity telemetry at 15-second intervals across 10,000 GPUs is very different from sampled telemetry that is kept for a month.

Third, settle the multi-tenancy question. In many factories, multiple teams share the cluster. Observability data can inadvertently leak information about other users' workloads — for example, the exact power draw of a neighboring job can reveal activity patterns. Choose a stack that supports tenant-scoped dashboards and that keeps detailed raw metrics in a datastore with permission controls.

Fourth, identify the operational owner. Full-stack observability fails when nobody owns the full stack. In practice, cluster administrators own the silicon and fabric metrics, ML platform teams own the framework metrics, and application teams own the training curves. The selected tooling must define clean ownership boundaries while still letting one team drill from a loss spike directly down to an XID error message.

What to Instrument: From XID Errors to Data-Throughput

A common mistake is to instrument only GPU utilization. NVIDIA's DCGM exposes a much richer set of metrics, including memory utilization, SM occupancy, power draw, temperature, clock throttle reasons, and error counters. On Linux, you can quickly inspect the health of the GPUs on your node with the NVIDIA System Management Interface:

# How to Choose Full-Stack Observability for NVIDIA AI Factories
nvidia-smi

# List detailed metrics for every GPU in JSON format
nvidia-smi --query-gpu=index,uuid,temperature.gpu,utilization.gpu,utilization.memory,power.draw,clocks.sm --format=csv

However, manual inspection is not observability. The important part is continuous collection. DCGM supports a more rigorous health-checking mode as well:

# Run DCGM's diagnostic suite once to identify hardware issues
dcgmi diag -r 1

These commands illustrate why the full-stack question is as much about the metric taxonomy as it is about the tools. Before choosing a vendor, establish which of these signals are mandatory:

• XID errors and driver-level faults, which indicate hardware or software fault conditions;
• throttle reasons, since a GPU can be at 100 % compute utilization but still throttled to 70 % of its clock;
• fabric errors, qualified as either corrected (benign) or uncorrected (fatal);
• PCIe and HBM read/write rates, which surface memory bandwidth saturation;
• job-level metadata from the scheduler, so every metric can be attributed to a workload owner.

If a proposed tool cannot capture these signals at a reasonable granularity, it is not a full-stack solution for an AI factory. The rest is decoration.

Step-by-Step Reference Installation

There is no single "correct" observability stack, but there is a well-trodden open-source pattern that demonstrates all of the principles above: DCGM-Exporter for GPU metrics, Prometheus for scraping and storage, and Grafana for dashboards. The commands below are an illustrative reference implementation. Use a dedicated monitoring namespace and pin the exact versions of the container images in your own environment, since versions change frequently.

Step 1 — Start a GPU metrics exporter.

NVIDIA publishes DCGM-Exporter as a container image. It reads telemetry from the host drivers and exposes them as Prometheus-formatted metrics on port 9400. The following command runs it on a node with NVIDIA drivers installed:

# Start the DCGM-Exporter on the default port
docker run -d --gpus all --rm \
  --name dcgm-exporter \
  -p 9400:9400 \
  nvcr.io/nvidia/k8s/dcgm-exporter:latest

If you prefer not to run a container, the DCGM-Exporter is also available as a standalone binary from the project's release page on GitHub. Verify that it is working by curling its metrics endpoint:

# Confirm that GPU metrics are being exported
curl -s http://localhost:9400/metrics | head -20

You should see metric names beginning with DCGM_FI_DEV_, such as DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_MEM_COPY_UTIL.

Step 2 — Configure Prometheus to scrape the exporter.

Create a minimal Prometheus configuration file. The key detail is to attach a label identifying the physical node, so that when a job runs on any node in the cluster, its metrics remain attributable to the correct host:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "dcgm"
    static_configs:
      - targets: ["gpu-node-01:9400", "gpu-node-02:9400"]
        labels:
          cluster: "ai-factory-east"

Then run Prometheus:

# Run Prometheus with the configuration above
docker run -d --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus:latest

Step 3 — Provision a Grafana dashboard.

Grafana can pull metrics directly from the data source, so no agent daemon is needed:

# Start Grafana and connect it to the Prometheus instance on port 9090
docker run -d --name grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  grafana/grafana:latest

After the containers are running, open the Grafana web interface on port 3000, log in, add a Prometheus data source pointing to http://prometheus:9090, and create a dashboard with panels that query the DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_POWER_USAGE metrics.

Usage Examples

The real value of the stack appears when you start asking cross-layer questions. Here are three useful patterns.

1. Detect GPU throttling that does not show up as low utilization.

A GPU running at 100 % of its current clock limit can still be running slower than its declared maximum rate. Python, using the Prometheus API, can expose this discrepancy. The example below queries the SM clock occupancy and throttle reasons:

import requests

prometheus_url = "http://localhost:9090/api/v1/query"

queries = {
    "throttle_reasons": 'DCGM_FI_DEV_CLOCK_THROTTLE_REASONS',
    "sm_clock": 'DCGM_FI_DEV_SM_CLOCK',
}

for name, query in queries.items():
    response = requests.get(prometheus_url, params={"query": query}).json()
    for result in response["data"]["result"]:
        print(name, result["metric"].get("gpu_uuid"), result["value"][1])

A nonzero throttle-reason bitmask tells you what is limiting the chip — thermal, power, or other factors — and the SM clock value tells you what the GPU decided to do about it.

2. Correlate a slow job with the data-loading layer.

Suppose your job runs at 300 steps per second instead of the expected 450. GPU utilization looks fine, which is suspicious. The query below computes the average memory copy utilization of every GPU over the last five minutes, which often surfaces a data-loading stall:

import requests

prom = "http://localhost:9090/api/v1/query"
query = 'avg_over_time(DCGM_FI_DEV_MEM_COPY_UTIL[5m])'

response = requests.get(prom, params={"query": query}).json()
for result in response["data"]["result"]:
    metric = result["metric"]
    print(metric.get("kubernetes_pod_name"),
          metric.get("gpu_uuid"),
          round(float(result["value"][1]), 2))

If the memory copy utilization is high while compute utilization is only moderate, the workload is almost certainly spending its time copying tensors between host and device or inside the fabric, rather than computing.

3. Build a simple alert for silent GPU faults.

A reliable alert can be written directly in PromQL, using the dcgm_exporter metrics to flag a GPU whose error counter increases over time:

increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0

This alert is far more actionable than a generic "node down" alert because it names the GPU and the job sharing it via the attached labels.

What to Standardize and Who Owns It

The hardest part of choosing full-stack observability is not the installation; it is the standardization of names and the organization of ownership.

First, standardize labeling across all metrics. If your GPU exporter, your Kubernetes exporter, and your scheduler all use the same job identifier, you can join them at query time. Choose a small set of labels — job_id, cluster, node, gpu_uuid — and enforce the convention at ingestion time.

Second, decide who deletes the metrics. An AI factory generates a continuous stream of telemetry that is valuable during an active incident and mostly useless after the incident is closed. Set a retention policy per tier: high-resolution metrics for the last few days, downsampled metrics for a few months, and raw GPU error logs for a year to support hardware warranty disputes.

Third, think carefully about the trade-off between collection overhead and fidelity. DCGM metrics are cheap to scrape, but the fabric and frame-level metrics are more expensive. Instrument the deep layers only on nodes where you have active training jobs, not on the entire cluster, and you will keep your overhead near zero.

Pitfalls to Avoid

Several common mistakes explain why AI factories end up with monitoring stacks that look impressive but fail in practice. One is instrumenting GPUs as if they were CPUs, which produces dashboards showing average utilization — a metric that is meaningless when one job occupies four GPUs far more heavily than another. Another is scraping only one destination, such as a single Prometheus instance, and losing data when the datacenter network blips. A third is choosing a tool that provides beautiful visualization but forces your telemetry through a proprietary agent that your security team rightly refuses to install.

There is also a purely administrative pitfall: buying an observability platform and assuming it will solve the organizational problem. The tool will not decide who remedies a fabric error; the factory operations team must. If your organization cannot already answer the question "Who owns the network fabric?", no product will help you see the network clearly enough to fix it.

Conclusion

Full-stack observability for NVIDIA AI factories is a layered decision. Define your SLOs, decide what telemetry you can afford and how long you need to store it, confirm that the candidate tool surfaces the low-level NVIDIA signals such as throttle reasons and XID errors, and then correlate those signals with scheduler and application data. A minimal reference stack built from the DCGM-Exporter, Prometheus, and Grafana is a reliable way to test these requirements in your own environment before committing to a wider deployment. Start with the silicon-level metrics, add the fabric and job metadata, and enforce a strict labeling convention from the first day. The GPU factory runs at a different pace than normal IT infrastructure, and its observability must be built to keep up.