NVIDIA H100 GPUs: Powering the Next Wave of AI Innovation

NVIDIA H100 GPUs: Powering the Next Wave of AI Innovation

The modern AI landscape is defined by astonishing progress. Every week, major research labs and cloud providers publish new breakthroughs—from large language models that write code to diffusion models that generate photorealistic images. The AI blogs from OpenAI, Google, and Microsoft offer a constant stream of these advances, while infrastructure companies like Replicate share practical guidance on deploying and running cutting-edge models. At the heart of this revolution sits the NVIDIA H100 GPU. Built on the Hopper architecture, the H100 is purpose-engineered to handle the enormous computational demands of training and serving state-of-the-art deep learning models.

The H100 is not just a faster version of its predecessor; it represents a fundamental shift in how we design AI systems. With its dedicated Transformer Engine, accelerated tensor operations, and enormous memory bandwidth, the H100 has become the de facto workhorse for organizations training frontier-scale models. But high-end hardware is only as useful as your ability to put it to work. In this article, I will walk through a complete, hands-on guide to getting an H100 GPU operational on a Linux server, from driver installation to running real training workloads.

Requirements

Before we dig into commands and configuration files, let’s establish what you need in place to follow along successfully.

Hardware Requirements

• **NVIDIA H100 GPU**: The H100 is available in several form factors, including the SXM5 module (typically found in HGX baseboards or NVIDIA DGX systems) and the PCIe variant for standard servers. Either works for the steps below, but SXM modules require a compatible server platform.
• **Host system**: An x86_64 or ARM64 server with at least one empty high-bandwidth PCIe slot (for the PCIe version) or a pre-built NVIDIA HGX system.
• **System memory**: At least 32 GB of RAM is recommended. For large batch sizes and multi-GPU training, consider 64 GB or more.
• **Storage**: A fast NVMe SSD with at least 100 GB free for the OS, drivers, CUDA toolkit, and sample models. Training datasets will require additional space.
• **Power supply**: H100 PCIe cards draw up to 350 W, and SXM modules can draw up to 700 W. Ensure your power budget and cooling are sufficient for sustained operation.

Software Requirements

• **Operating system**: Ubuntu 22.04 LTS is used in this guide, but the commands are broadly compatible with other modern Linux distributions.
• **NVIDIA driver**: A recent driver version (525.60.13 or later) is required for H100 support.
• **CUDA toolkit**: CUDA 12.x is recommended for full Hopper architecture support, including the Transformer Engine.
• **Container runtime**: Docker with the NVIDIA Container Toolkit is optional but strongly recommended for reproducible AI environments.

Access Requirements

You will need admin (sudo) access to the machine. If you are renting an H100 cloud instance, your provider will likely offer a base image with drivers pre-installed. Still, it is valuable to understand the setup process from scratch, especially if you are provisioning your own hardware.

Step-by-step installation

Now we get to the heart of the matter. The pipeline looks like this: identify the hardware, install the NVIDIA driver, install CUDA, add cuDNN and the container toolkit, and finally verify everything with `nvidia-smi`.

Step 1: Verify that the GPU is visible

Before installing anything, confirm that the operating system can see the H100. First, use `lspci` to list all PCI devices and filter for NVIDIA:

lspci | grep -i nvidia

You should see an output line that includes "NVIDIA" and a device name such as "H100" or "H100 SXM". If nothing appears, check the physical seating of the card, the power cables, and the BIOS settings (ensure PCIe slots are enabled).

Step 2: Update the system and install build tools

The NVIDIA driver ships as a kernel module, so you will need the Linux kernel headers and a C compiler. Update the package list and install the essentials:

sudo apt update && sudo apt upgrade -y

This ensures your system is fully patched before we proceed. Next, install the tools required for building kernel modules:

sudo apt install -y build-essential dkms linux-headers-$(uname -r)

The `linux-headers-$(uname -r)` package provides the exact headers for your current kernel, which the NVIDIA driver build process depends on.

Step 3: Install the NVIDIA driver

There are two common ways to install the driver: using the distribution’s package manager, or using the official NVIDIA runfile. For most users, the package manager approach is simpler and more maintainable. On Ubuntu, `ubuntu-drivers` can automatically select and install the recommended driver version:

sudo ubuntu-drivers install

The `ubuntu-drivers` tool reads the hardware database and selects the most appropriate proprietary driver for your GPU. After installation completes, reboot the system so the driver kernel module can load:

sudo reboot

Once the machine is back online, check the driver status with:

nvidia-smi

If the command runs successfully, you will see a table showing your H100, the driver version, and the amount of GPU memory. For production settings, you may want to install a specific driver version instead. In that case, download the desired `.run` file from NVIDIA’s official website and execute it with:

sudo sh NVIDIA-Linux-*.run

When using the runfile, ensure the NVIDIA kernel module is not already loaded, or the installer will warn you. The package manager approach is cleaner for most scenarios.

Step 4: Install the CUDA toolkit

With the driver installed, the next layer is the CUDA toolkit, which provides the `nvcc` compiler and CUDA libraries. On Ubuntu, you can install the toolkit directly from the NVIDIA package repository. First, add the NVIDIA CUDA repository keyring:

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb

Note that `apt-key` is deprecated on newer systems; a more modern approach involves downloading and installing the keyring package:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb

Then install the downloaded package and update the package list:

sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

Now install the full CUDA toolkit:

sudo apt install -y cuda-toolkit-12-4

This installs CUDA 12.4, which has full support for the H100 architecture. For the most recent version, replace `12-4` with the latest release number available. Once installation is complete, add the CUDA binaries and libraries to your PATH and LD_LIBRARY_PATH:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

To make these persistent across sessions, add them to your shell profile:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Verify the compiler with:

nvcc --version

You should see the version output that confirms CUDA 12.4 is active. Note that this installed the toolkit system-wide; if you prefer, you can install only the CUDA runtime via `libcudart` packages and manage CUDA dependencies within a Docker container. We will revisit that option later.

Step 5: Install cuDNN

cuDNN (CUDA Deep Neural Network library) provides highly optimized implementations of neural network operations like convolutions and recurrent layers. It is a prerequisite for most deep learning frameworks. Download the appropriate cuDNN debian package from the NVIDIA developer portal (registration is required), then install it:

sudo dpkg -i cudnn-linux-x86_64-*.deb

Alternatively, if you installed the CUDA toolkit via the repository, you can use `apt` directly:

sudo apt install -y libcudnn8 libcudnn8-dev

The `libcudnn8` package contains the runtime libraries, while `libcudnn8-dev` includes the headers needed to compile against cuDNN. After installation, verify that cuDNN is found by checking the library path:

ldconfig -p | grep cudnn

Step 6: Install the NVIDIA Container Toolkit (Docker)

If you plan to use Docker—and I highly recommend it for isolating your AI environments—you need the NVIDIA Container Toolkit so that containers can access the GPU. First, ensure Docker is installed:

sudo apt install -y docker.io
sudo systemctl enable --now docker

Next, install the NVIDIA Container Toolkit. Start by adding NVIDIA’s official repository to your apt sources:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

Then add the repository entry:

echo "deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(lsb_release -cs)/" | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update and install the toolkit:

sudo apt update
sudo apt install -y nvidia-container-toolkit

Now restart Docker for the changes to take effect:

sudo systemctl restart docker

Step 7: Verify everything

Finally, verify that the whole stack works together. Run a container that uses the GPU:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If the container prints the GPU table, your driver, CUDA, and container runtime are all correctly configured. You now have a fully equipped H100 machine ready for deep learning.

Usage examples

With a functional setup, let’s look at practical ways to use the H100. We will cover verifying GPU availability with PyTorch, training a small model, and launching a multi-GPU job.

Example 1: PyTorch GPU sanity check

PyTorch is the most widely used framework for AI research, and it ships pre-compiled with CUDA support. Begin by installing PyTorch in a fresh virtual environment:

python3 -m venv h100-env
source h100-env/bin/activate

Then install the latest PyTorch build with CUDA support:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

The `--index-url` flag ensures that you get the build bundled with CUDA 12.4 support. Now create a short Python script to check that the H100 is detected:

python <<EOF
import torch
print("CUDA available:", torch.cuda.is_available())
print("GPU count:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))
EOF

You should see output like `CUDA available: True` and `GPU name: NVIDIA H100`. Next, run a simple tensor operation on the GPU to confirm that computations are actually executed on the device:

import torch

# Create a large random tensor directly on the GPU
x = torch.randn(10000, 10000, device="cuda")
y = torch.randn(10000, 10000, device="cuda")

# Perform matrix multiplication on the H100
z = x @ y

# Synchronize and report the result shape
torch.cuda.synchronize()
print("Matrix multiplication complete. Result shape:", z.shape)

The mix of `device="cuda"` and `torch.cuda.synchronize()` ensures that operations are placed on the H100 and that the Python code waits for the computation to finish.

Example 2: Training a small neural network

Let’s train a lightweight neural network on the MNIST dataset to see the H100 in action. First, ensure you have `torchvision` installed (we already did above). Then save the following training script to a file:

# train_mnist.py
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

# Define a simple feed-forward network
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10)
        )

    def forward(self, x):
        return self.fc(x)

# Load the MNIST dataset
transform = transforms.ToTensor()
train_dataset = datasets.MNIST(root="./data", train=True, transform=transform, download=True)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)

# Create the model, loss, and optimizer, and move everything to the GPU
model = SimpleNet().cuda()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())

# Train for a few epochs
for epoch in range(3):
    total_loss = 0.0
    for images, labels in train_loader:
        images, labels = images.cuda(), labels.cuda()
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()

    print(f"Epoch {epoch + 1}, Loss: {total_loss / len(train_loader):.4f}")

Run the script with:

python train_mnist.py

Even though this model is small, you will notice that training is extremely fast. The H100 is overkill for MNIST, but the workflow demonstrates how your framework interacts with the GPU.

Example 3: Multi-GPU training with `torchrun`

Real-world AI models require multiple GPUs. The H100 is often deployed in clusters of eight or more. PyTorch’s `torchrun` utility simplifies distributed data-parallel training. Suppose you have a server with eight H100 GPUs and want to launch training across all of them. First, write a training function that uses `DistributedDataParallel`:

# ddp_train.py
import torch
import torch.distributed as dist
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP

def main():
    dist.init_process_group("nccl")
    local_rank = int(os.environ["LOCAL_RANK"])

    model = nn.Linear(1024, 1024).cuda(local_rank)
    model = DDP(model, device_ids=[local_rank])

    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    for step in range(100):
        x = torch.randn(256, 1024).cuda(local_rank)
        y = model(x).sum()
        optimizer.zero_grad()
        y.backward()
        optimizer.step()

    dist.destroy_process_group()

if __name__ == "__main__":
    import os
    main()

Then launch the job with `torchrun`, telling it to spawn one process per GPU:

torchrun --nproc_per_node=8 ddp_train.py

The NCCL backend will handle high-speed communication between the H100 GPUs, whether over NVLink inside the same node or over InfiniBand across nodes.

Example 4: Running a pre-built container

For reproducible experiments, pull a pre-built deep learning container that includes PyTorch and CUDA, then run your code inside it. The NGC catalog provides official containers:

docker run --gpus all -it --shm-size=8g nvcr.io/nvidia/pytorch:24.04-py3 bash

The `--shm-size=8g` flag increases the shared memory available to DataLoader workers, which is a common source of crashes in training containers. Once inside the container, you can run `python` and access the GPU exactly as you would on the host. This approach is particularly useful for sharing environments with collaborators.

Conclusion

The NVIDIA H100 GPU is more than just a piece of hardware; it is the engine driving the current wave of artificial intelligence innovation. The progress visible across the AI blogs of OpenAI, Google, and Microsoft—new models with emergent reasoning, tools that generate code and art, and systems that understand multimodal inputs—would be unimaginable without the computational capacity that GPUs like the H100 provide. Meanwhile, platforms like Replicate, which have written about how they deploy H100s at scale, make this technology accessible to developers who do not own their own data centers.

In this article, I have taken you from a bare server to a fully configured deep learning workstation. We verified the GPU, installed the NVIDIA driver, set up CUDA and cuDNN, configured Docker, and ran both single-GPU and multi-GPU workloads. The H100’s power is astonishing, but it is also demanding: it requires careful power management, a modern software stack, and a disciplined approach to containerization to be used efficiently.

As AI research accelerates, the gap between models that are merely possible and models that are practical will continue to narrow. With an H100 in your toolkit, you are well positioned to be on the leading edge of that curve. The commands and examples in this article are your starting point. Now go and build something remarkable.