Diagnose Distributed AI Workload Failures with NVIDIA Attribution

Diagnose Distributed AI Workload Failures with NVIDIA Attribution

In the realm of distributed artificial intelligence, a single hardware fault can trigger a cascade of symptoms that obscure the true origin of a failure. When a GPU encounters a hard fault, downstream ranks may report communication errors, scheduler states may shift, and thousands of log messages may appear across the cluster. The reported exception often indicates where the workload stopped, not where the failure began. NVIDIA Attribution addresses this complexity by correlating workload-scoped logs, metrics, telemetry, and scheduler state to produce a structured diagnosis.

According to NVIDIA, this open-source tool allows infrastructure teams to turn raw workload evidence into a reproducible diagnosis on their own clusters. The system operates through two deployable components: the Attribution Service, which exposes a REST API for the core attribution engine, and the Attribution Blueprint, a Python reference orchestrator that connects cluster schedulers and telemetry sources to the Service.

Architecture and Core Components

The architecture of NVIDIA Attribution is designed to separate the core logic from environment-specific integration. The Attribution Service analyzes one workload attempt at a time, making attribution available through a REST API. The Attribution Blueprint, conversely, handles cluster-specific integration. It discovers workloads, resolves the execution context, runs collectors, calls the Service, enriches the verdict, and writes the result to output sinks, including a read-only dashboard.

This separation allows teams to apply the same attribution logic across clusters while adapting scheduler, telemetry, and output connectors to each environment. The Blueprint ships with execution-context connectors for Slurm, Kueue, and static inputs. Additionally, it includes log and metric connectors for Loki, local files, Kubernetes pod logs, and PromQL metrics.

NVIDIA notes that the system has been used in production across NVIDIA AI infrastructure for years. In one large RDMA-backed cluster, it correlated port errors, connection resets, and scheduler events to identify patterns around specific host and switch ports. This capability helps teams target fabric repairs and congestion tuning effectively.

The Four-Stage Attribution Process

The attribution process follows a deterministic four-stage workflow designed to separate downstream symptoms from the event that caused the failure.

1. Scope the Workload: The Blueprint identifies the workload attempt from scheduler identifiers or from an explicit execution window and node allocation.
2. Collect Observations: Log and metric connectors produce workload-scoped Observation records from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics.
3. Extract and Correlate Events: The Service invokes the attribution engine to identify significant events such as GPU XIDs, NCCL failures, scheduler transitions, and metric conditions. Deterministic rules evaluate their timing and interaction.
4. Return a Structured Result: The Blueprint combines the verdict with the workload outcome (terminalstate), attributed domain (subdomain), symptoms, affected nodes, and a record of collector coverage and errors.

This process ensures that results are reproducible and customizable. The structured output enables integration with existing health and recovery systems, allowing downstream systems to consume terminalstate and subdomain instead of parsing product-specific logs.

Interpreting Evidence: A Worked Example

To illustrate how the system functions, consider a synthetic multi-rank training workload where one GPU encounters a hard fault. In this scenario, application logs contain three different NCCL errors, while system logs on node-02 contain XID 149 with error words that decode to an NVLink signal integrity issue.

The Extraction phase runs first on the raw input logs, producing structured events to be processed by later stages. As described in NVIDIA's documentation, the EARLY-stage HandleXidBurstEvents rule reads the XID burst evidence, decodes the error words, and emits an attribution event for a signal integrity issue on node-02. During this step, the three NCCL events remain unchanged.

The framework then materializes events into per-node symptoms and runs per-node scrubbing. In the example provided, the Xid149AndNvlinkFiltering rule retains the signal integrity symptom and demotes the other node symptoms. The Gb200PriorityRootCause rule further reduces the active job-wide symptom set. Finally, outcome mapping produces the hardware-failure attribution.

Crucially, the system retains the NCCL reports as "interesting" evidence. While they are demoted to support the primary diagnosis, their original sources are preserved so they can still be inspected. This approach ensures that the diagnosis reflects the full picture: the root cause is the hardware failure on node-02, while the NCCL errors are preserved as supporting symptoms explaining the distributed nature of the failure.

Handling Uncertainty and Quality

When multiple causes fit the evidence, the sub_domain may be ambiguous. NVIDIA Attribution correctly reflects this uncertainty instead of converting incomplete evidence into a definitive diagnosis. The quality of attribution is dependent on the quality and completeness of the underlying telemetry.

The system distinguishes between pretraining and fine-tuning contexts implicitly by analyzing the specific events and logs associated with the workload. However, it is important to note that a model demonstration is not an independently verified measurement. The diagnosis records the distinction, ensuring that teams understand the confidence level of the attribution.

Extending Attribution to Infrastructure

Open sourcing NVIDIA Attribution enables infrastructure teams to encode a failure pattern once and apply it across workloads and clusters. Developers can add rules for new failure modes and implement connectors for their schedulers and telemetry backends. Because extensions use the same ExecutionContext, Observation, and result contracts, new integrations feed the same diagnosis pipeline.

The Attribution Blueprint is intended to be a reference implementation for practitioners to customize the solution to their environment. Blueprints can run near each cluster’s data sources, with either a dedicated Attribution Service for each Blueprint or a shared Service for multiple Blueprints. There is flexibility on how teams want to customize their deployment, such as choosing a one-to-one or one-to-many configuration.

Conclusion

NVIDIA Attribution provides a robust framework for diagnosing distributed AI workload failures by moving beyond isolated node events to a workload-centric analysis. By correlating logs, metrics, and scheduler state through deterministic rules, the system identifies root causes such as hardware faults while preserving downstream symptoms as evidence.

Teams can leverage the Attribution Service and Blueprint to gain visibility into failure patterns, target fabric repairs, and improve the resilience of their AI infrastructure. Whether analyzing NCCL errors or NVLink signal integrity issues, the tool offers a structured path from raw evidence to actionable diagnosis.

Source: https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution