NVIDIA H100 GPUs Are Here: Powering the Next Wave of AI

NVIDIA H100 GPUs Are Here: Powering the Next Wave of AI

The arrival of NVIDIA H100 GPUs marks one of the most significant shifts in the AI hardware landscape in years. Across the industry, the signal is unmistakable: OpenAI's news page is full of large-scale model launches, Google's AI blog chronicles increasingly ambitious multimodal systems, and Microsoft's AI blog describes enterprise platforms that put frontier-scale compute into ordinary business workflows. Underpinning much of that progress is the new generation of accelerators, with the H100 leading the charge. Even GPU-focused research and deployment platforms such as Replicate have turned their attention to H100s for serving large open-source models in production. The GPU is no longer just a piece of hardware; it is the engine that determines what kinds of AI problems we can solve at all.

For developers, researchers, and infrastructure teams, the question is no longer *whether* to adopt H100-class hardware, but *how*. This article walks through the practical side of standing up an H100-based AI environment, from hardware requirements and driver installation to running cutting-edge training and inference workloads. By the end, you will have a working, reproducible setup that lets you take full advantage of the next wave of AI.

The H100 in Context

The H100 is built on NVIDIA's Hopper architecture, a design aimed squarely at the era of large Transformer models. Its most distinctive feature is the Transformer Engine, which can automatically switch between precision formats—including the FP8 and FP16 mixed-precision modes that have become standard for large language models—to accelerate training and inference without sacrificing accuracy. Support for HBM3 memory, second-generation NVLink, and high-bandwidth interconnects make it possible to scale from a single GPU to a multi-node cluster without the bottleneck that plagued earlier systems.

From an industry perspective, the H100 has quickly become the default compute resource for serious AI work. Cloud providers now offer H100 instances, and the major AI laboratories depend on clusters of thousands of these GPUs. What this means for you is simple: if you want to reproduce modern research, fine-tune a large model, or serve an open-source LLM at scale, the H100 is the reference hardware you should be developing against.

That said, raw hardware is only half the story. The real power of the H100 is unlocked by a software stack that has matured rapidly around it: CUDA 12.x, cuDNN, PyTorch, and containerized environments delivered through NVIDIA's NGC catalog. The installation process described below reflects that modern, container-first workflow.

Requirements

Before you begin, make sure your system meets the following criteria. The H100 comes in two primary form factors—SXM5 modules that plug into an NVIDIA baseboard, and PCIe cards that fit into standard server slots. This guide assumes a PCIe H100 in a compatible x86 server, but the software steps are identical for both.

**Hardware requirements:**

• An NVIDIA H100 GPU with at least 80 GB of HBM3 memory.
• A server with a Gen 4 or Gen 5 PCIe x16 slot for the PCle variant.
• At least 64 GB of system RAM for light workloads; 256 GB or more is recommended for large-model training.
• Sufficient power delivery. The PCIe H100 draws around 350 W, while the SXM5 module can draw up to 700 W. Plan your power budget accordingly.
• Adequate cooling. H100s are designed for server chassis with aggressive airflow; do not attempt to install one in a workstation without verified thermal design.

**Software requirements:**

• A supported Linux distribution, such as Ubuntu 22.04 LTS or Rocky Linux 9.
• NVIDIA driver version 535 or newer for full H100 support.
• CUDA Toolkit 12.1 or later if you are installing outside a container.
• Docker with the NVIDIA Container Toolkit for containerized deep learning.
• A recent version of PyTorch (2.0 or newer) with CUDA 12.x support.

If you do not have local hardware, the same steps apply to H100 cloud instances from the major cloud providers. The commands below assume you have root access via `sudo` and are working on Ubuntu 22.04.

Step-by-Step Installation

The fastest path to a working H100 environment is a containerized one. NVIDIA maintains prebuilt PyTorch containers that bundle the driver, CUDA libraries, and optimized deep learning frameworks. We will start there, with a brief stop at the host driver.

1. Verify the GPU

First, confirm that the operating system sees the GPU. The following command lists all PCI devices with the NVIDIA vendor ID:

lspci | grep -i nvidia

You should see output similar to `NVIDIA Corporation Device 2322` (the PCI device ID of the H100 PCIe card). If the GPU is not listed, shut down and reseat the card or check your PCIe slot configuration.

If a driver is already installed, you can run `nvidia-smi` directly:

nvidia-smi

At this point the output may show no driver or version errors. That is fine. We will install the correct driver next.

2. Install the NVIDIA Driver

Ubuntu LTS releases provide NVIDIA drivers through their package repositories. For the H100, install version 535 or later. The commands below update the package index and install the driver:

sudo apt update
sudo apt install -y nvidia-driver-535

After installation, load the kernel module and check that the GPU is visible:

sudo modprobe nvidia
nvidia-smi

Wait a few seconds for the driver to initialize. The output should display "NVIDIA H100 80GB HBM3" together with the driver version and a CUDA version. If `nvidia-smi` reports "No devices were found," reboot the machine once and run it again.

3. Install the NVIDIA Container Toolkit

We will use Docker to run a prebuilt deep learning environment, which avoids the famously tedious process of matching CUDA, cuDNN, and PyTorch versions by hand. First, install Docker:

sudo apt install -y docker.io
sudo systemctl start docker
sudo systemctl enable docker

Next, add the NVIDIA Container Toolkit repository and install the toolkit. These commands are taken from the official NVIDIA setup guide and configure the repository with the correct GPG signing:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Now update the package list and install:

sudo apt update
sudo apt install -y nvidia-container-toolkit

Finally, configure Docker to use the new runtime and restart it:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test the containerized GPU access with a minimal CUDA image:

docker run --rm --gpus all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi

If everything is correct, this command prints the same `nvidia-smi` output as before, but from inside the container. This confirms that the entire GPU software stack, from driver to container runtime, is operational.

4. Pull the PyTorch Container

The NVIDIA NGC catalog provides a prebuilt PyTorch container that is tuned specifically for H100 hardware. Pull it and run an interactive session:

docker pull nvcr.io/nvidia/pytorch:23.10-py3

docker run --gpus all -it --shm-size=16g --rm nvcr.io/nvidia/pytorch:23.10-py3

The `--shm-size=16g` flag is important. PyTorch DataLoader workers use shared memory for inter-process communication, and the default 64 MB limit will cause errors with any significant dataset. Once the container starts, you will be at a Bash prompt with Python, PyTorch, and CUDA already installed. Verify the GPU is accessible to PyTorch:

import torch
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.get_device_name(0))

You should see `True`, `1`, and `NVIDIA H100 80GB HBM3`. Congratulations—your H100 is ready for deep learning.

5. Installing PyTorch Without Docker (Alternative)

If you prefer not to use containers, install the NVIDIA driver as above, then install CUDA 12.x and PyTorch directly. On Ubuntu, installing the full CUDA Toolkit via the NVIDIA runfile is the most reliable method, but for simplicity the pip route is fastest. The official PyTorch binaries with CUDA 12.1 support are available through the PyTorch package index:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Afterwards, run the same three-line Python check above. This approach works, but for production work the NGC container is generally recommended because it is compiled with NVIDIA's own optimizations for newer architectures.

Usage Examples

With the environment running, let us look at three common operations: training a model with mixed precision, running inference from a pretrained transformer, and monitoring the GPU.

Example 1: Mixed-Precision Training

The H100 excels at reduced-precision training. PyTorch's `torch.autocast` and gradient scaling make this straightforward. The following script trains a small convolution network on random data, mixing FP16 and FP32 automatically:

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(3, 32, 3, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    nn.Flatten(),
    nn.Linear(32, 10),
).cuda()

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

x = torch.randn(64, 3, 224, 224, device="cuda")
y = torch.randint(0, 10, (64,), device="cuda")

scaler = torch.cuda.amp.GradScaler()

for step in range(10):
    optimizer.zero_grad()
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        out = model(x)
        loss = loss_fn(out, y)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    print(f"Step {step}: loss = {loss.item():.4f}")

With the Transformer Engine activated automatically in the background, you get near-linear speedups over pure FP32 training.

Example 2: Serving an LLM With Transformers

The most common task for an H100 today is running inference over a large language model. The Hugging Face `transformers` library works out of the box. Try a quick generation with GPT-2 as a sanity check:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").cuda().eval()

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

For models that are too large for a single GPU, PyTorch's distributed launch tool is the standard entry point. Create a training script, then run it across multiple GPUs with:

torchrun --nproc_per_node=8 train.py

This scales to multi-node clusters via NVLink and InfiniBand with minimal code changes.

Example 3: Monitoring Performance

While workloads run, check utilization and memory with the `nvidia-smi` monitoring loop. The `dmon` subcommand shows a live dashboard:

nvidia-smi dmon -s pucvmet

And to capture detailed utilization over time, use:

nvidia-smi --query-gpu=timestamp,utilization.gpu,power.draw,memory.used \
           --format=csv -l 1

Watching the power draw and GPU utilization during your first large training run is the quickest way to validate that the system is performing as expected. In a healthy setup, the H100 should sit near 100% utilization and reach specified power targets under sustained load.

What This Means for the AI Landscape

Looking at the broader ecosystem, the availability of H100 hardware intersects directly with the ambitions of the major AI players. OpenAI's ongoing releases of large models, Google's work on multimodal systems, and Microsoft's enterprise AI infrastructure all depend on exactly this class of accelerator. The Replicate blog's focus on H100-backed inference also points to a key trend: GPUs are moving from experimental laboratory equipment to production infrastructure. The fact that small teams can now rent H100 instances by the hour, when even two years ago nothing comparable existed, changes the economics of AI development substantially.

For individual engineers, this translates into a concrete advantage. You no longer need to be part of a large organization to train, fine-tune, or serve a model of meaningful scale. With one H100 in front of you, you can reproduce results that were exclusive to large labs only a few years ago. The steps in this article provide the foundation for that work. From here, the next steps are to explore quantization libraries, inference servers, and distributed training frameworks—all of which have first-class support for Hopper architecture.

Conclusion

NVIDIA H100 GPUs are not just another hardware upgrade. They represent the computational backbone of the current AI boom—the engine that powers the large language models, multimodal systems, and production AI services announced across the industry, from OpenAI and Google to Microsoft and beyond. The fact that major deployment platforms have rallied around this hardware tells you where the ecosystem is headed.

The practical path to getting started is now well-defined: verify the hardware, install the driver, set up the NVIDIA Container Toolkit, and pull a prebuilt PyTorch container. Within minutes, you can move from bare metal to a working mixed-precision training loop or a live inference endpoint. Whether you are running a single H100 for research or managing a cluster for production workloads, the same principles apply.

The next wave of AI will be built on this hardware. By configuring your own H100 environment today, you position yourself to be part of it.