Unlocking Tensor Core Performance for Machine Learning Interatomic Potentials

Unlocking Tensor Core Performance for Machine Learning Interatomic Potentials

Atomistic simulations are fundamental to understanding how molecules and materials behave, ranging from chemical reactions and phase transitions to mechanical failure. However, these simulations present a significant cost-accuracy trade-off. Quantum-mechanical methods, such as density functional theory (DFT), offer high accuracy but are computationally expensive, often limiting their use to small systems or short timescales. Conversely, classical interatomic potentials are fast but lack the generalizability required for diverse atomic environments. Machine learning interatomic potentials (MLIPs) bridge this gap by using neural networks to predict energies and forces at a fraction of the cost, enabling the study of larger systems while maintaining necessary accuracy.

Despite their efficiency, running MLIPs efficiently remains a challenge. Atomistic simulations are sensitive to small numerical errors that accumulate over time, particularly when computing forces derived from energy differentiation. During molecular dynamics, these forces guide atomic updates over many timesteps, making numerical precision critical for both physical accuracy and trajectory stability. Consequently, many MLIPs continue to run inference in FP32, even though modern NVIDIA GPUs offer substantially higher matrix-multiplication throughput through low-precision Tensor Core operations.

To address this, researchers have evaluated Tensor Core FP32 methods. These techniques decompose FP32 operands into lower-precision components, evaluate their cross-products on Tensor Cores, and recombine the results in FP32. This approach increases matrix-multiplication throughput while retaining accuracy close to native FP32, without requiring model retraining or architectural changes. The following analysis evaluates the performance of these methods on the Point Edge Transformer (PET), a transformer-based MLIP developed by Lab COSMO at EPFL.

How Tensor Core FP32 Closes the Hardware Gap

Modern NVIDIA GPUs deliver substantially higher Tensor Core throughput for BF16 and TF32 than for FP32 matrix multiplication. This creates a gap between available throughput and the precision requirements of scientific workloads. Tensor Core FP32 methods help narrow this gap by leveraging the hardware's strength in lower precision while maintaining the output precision required for stability.

On NVIDIA Blackwell GPUs (compute capability 10.0 or 10.3), the BF16x9 algorithm is supported via NVIDIA cuBLAS. As illustrated in the documentation, each FP32 operand is decomposed into three BF16 components, producing nine pairwise products that are accumulated in FP32 along five anti-diagonal bands. NVIDIA cuBLAS enables this path through the CUBLAS_EMULATE_SINGLE_PRECISION=1 environment variable, allowing compatible PyTorch GEMMs to utilize the path without modifying the model architecture or implementing custom kernels.

On NVIDIA Ampere GPUs, the TF32x3 method applies a similar principle. Each FP32 operand is represented using a leading TF32 component and a residual TF32 component. The implementation evaluates three cross-products—leading-by-leading and two leading-by-residual products—and accumulates them in FP32, omitting the residual-by-residual term. NVIDIA CUTLASS provides this approach through the OpMultiplyAddFastF32 operator, where inputs and outputs remain in FP32.

Benchmarking Tensor Core FP32 Matrix Math on PET

The evaluation focused on the 26M-parameter PET-MAD small checkpoint (pet-mad-s-v1.6.0-nn-32) on a 192-atom water box and a 17,280-atom supercell, as well as the 255M-parameter large checkpoint (pet-mad-m-v1.6.0) on the 192-atom system. Predicted energies, forces, and stresses were compared against the native FP32 path.

BF16x9 Method on NVIDIA Blackwell GPUs

On Blackwell GPUs, the BF16x9 method demonstrated significant acceleration for large GEMMs within the PET model. For the 17,280-atom workload, large GEMMs were accelerated by 2.2–2.5x. For the 192-atom workload, the acceleration ranged from 1.2–1.4x.

When evaluating end-to-end molecular dynamics performance, two strategies were tested:

• Performant Strategy: Uses BF16x9 when cuBLAS expects it to be faster.
• Eager Strategy: Uses BF16x9 whenever the operation supports it.

On the 192-atom system, the performant strategy sped up the MD step by 1.2x, while the eager strategy achieved 1.1x. On the 17,280-atom system, the performant strategy achieved a 1.7x speedup, while the eager strategy achieved 1.4x. Importantly, the error introduced by the BF16x9 Tensor Core method was found to be below the model's own level of non-determinism. This result is significant because PET already expresses much of its runtime as optimized GEMM work, yet the BF16x9 path exposed performance headroom without retraining the model or changing its transformer architecture.

TF32x3 Method on Ampere GPUs

On Ampere GPUs, the TF32x3 method was implemented by replacing PET's standard nn.Linear layers with a custom TF32Linear layer that calls a CUTLASS C++ TF32x3 kernel. The key setting in the PET implementation selects CUTLASS's fast, accurate FP32 multiplication mode. The kernel represents each FP32 operand as leading and residual TF32 components, and Tensor Cores execute the three TF32 products with accumulation in FP32.

For the 17,280-atom workload on an NVIDIA A100 Tensor Core GPU, TF32x3 accelerated large PET SGEMMs by approximately 1.7–1.9x relative to native FP32. In terms of full molecular dynamics steps, TF32x3 sped up the process by up to 1.5x while keeping force error near the FP32 baseline.

Validating Physical Consistency

A critical aspect of using lower-precision arithmetic in scientific simulations is ensuring physical consistency. The evaluation tested the TF32x3 method for potential-energy smoothness, energy conservation, and structural relaxation.

Energy Conservation and Smoothness

TF32x3 preserves potential-energy smoothness and energy conservation in NVE trajectories, matching FP32 behavior. In contrast, one-pass TF32 showed drift and roughness in these trajectories. This distinction highlights the importance of the specific decomposition method used to maintain numerical stability over long simulations.

Structural Relaxation

The methods were further tested during the structural relaxation of a 539-atom Cantor-alloy slab. In this scenario, TF32x3 converged in 218–222 steps with a 1.5x per-step speedup. Conversely, one-pass TF32 failed to reach the force tolerance even after 1,000 steps. This demonstrates that while standard low-precision methods may offer raw speed, they can compromise the convergence and reliability required for structural optimization tasks.

Limitations and Decision Criteria

While Tensor Core FP32 methods offer substantial performance gains, they are not a universal solution. The evaluation did not measure the combined gain when these methods are used alongside optimized graph construction, fused operations, custom kernels, and NVIDIA CUDA graphs; such end-to-end gains should be established for each specific workload.

Furthermore, the implementation details differ between hardware generations. On Blackwell GPUs, the approach relies on cuBLAS environment variables to enable the path for single-precision general matrix multiplication (SGEMM) without code changes. On Ampere GPUs, a custom C++ implementation using CUTLASS is required to replace standard linear layers. Users must weigh the performance benefits against the implementation complexity and ensure that the specific error margins remain acceptable for their scientific domain.

Conclusion

Tensor Core FP32 methods provide a viable pathway to accelerate machine learning interatomic potentials without sacrificing the numerical stability required for atomistic simulations. By decomposing FP32 operands into lower-precision components, methods like BF16x9 on Blackwell GPUs and TF32x3 on Ampere GPUs can leverage higher Tensor Core throughput.

The evidence indicates that for the Point Edge Transformer, BF16x9 can accelerate large GEMMs by up to 2.5x on Blackwell hardware, while TF32x3 achieves similar gains on Ampere GPUs. Crucially, these methods maintain physical consistency, preserving energy conservation and convergence in structural relaxation tasks where one-pass low-precision methods fail. Researchers can unlock these performance gains by utilizing the documented tools and environment variables, enabling faster exploration of materials and molecules while retaining the accuracy essential for scientific discovery.

Source: https://developer.nvidia.com/blog/unlocking-tensor-core-performance-for-machine-learning-interatomic-potentials