Run AI Workloads on Any Cloud, Store on Hugging Face: Zero-Egress Storage with SkyPilot

Run AI Workloads on Any Cloud, Store on Hugging Face: Zero-Egress Storage with SkyPilot

Data transfer costs—commonly known as egress fees—have long been a hidden tax on cloud-based AI development. Moving terabytes of training data, model checkpoints, and inference artifacts between cloud providers or from cloud to on-premises systems can quickly inflate budgets. But what if you could decouple compute from storage, running GPU-intensive workloads on any cloud while keeping your data safely and cheaply on Hugging Face? That is precisely the promise of zero-egress storage with SkyPilot.

In this article, we will explore how to combine SkyPilot’s multi-cloud orchestration with Hugging Face Hub’s storage capabilities to eliminate egress costs. We will walk through installation, configuration, and practical examples so you can start saving money and simplifying your AI infrastructure today.

What Is Zero-Egress Storage and Why Does It Matter?

When you run AI workloads on a cloud provider like AWS, GCP, or Azure, you typically store your data in that provider’s object storage (S3, GCS, Azure Blob). Moving data out of that storage to another cloud or to a different region incurs egress fees—often $0.05–$0.12 per GB. For datasets in the hundreds of gigabytes or terabytes, these costs become significant.

Zero-egress storage means you store your data in a location that does not charge for egress when accessed from compute resources. Hugging Face Hub, while not a full cloud provider, offers free egress for public datasets and models. For private repositories, Hugging Face does not charge egress fees either—only storage fees apply. By storing your data on Hugging Face and running compute on any cloud via SkyPilot, you effectively eliminate egress costs.

How SkyPilot Enables Multi-Cloud AI

SkyPilot is an open-source framework that abstracts away cloud provider differences. You define your workload (e.g., a training job or inference server) in a YAML file, and SkyPilot automatically provisions the cheapest or most suitable cloud resources across AWS, GCP, Azure, or even on-premises clusters. It handles networking, storage mounting, and job orchestration.

Key features:

• **Multi-cloud support:** Run the same YAML on any cloud.
• **Automatic failover:** If one cloud is out of capacity, SkyPilot tries another.
• **Cost optimization:** Choose the cheapest region or spot instances.
• **Storage integration:** Mount cloud buckets or Hugging Face repositories as volumes.

By combining SkyPilot with Hugging Face Hub, you get a truly portable AI stack: your data lives on Hugging Face, your compute runs wherever is cheapest or most available, and you never pay egress.

Requirements

Before you begin, ensure you have the following:

1. **A Hugging Face account** (free tier works for public repos; paid tier for private storage). 2. **Cloud provider credentials** for at least one of AWS, GCP, or Azure. (We will use AWS in examples, but the process is similar for others.) 3. **Python 3.8+** installed on your local machine. 4. **`git` and `git-lfs`** installed (for pushing large files to Hugging Face). 5. **Basic familiarity with the command line.**

Step-by-Step Installation

1. Install SkyPilot

SkyPilot is available via pip. We recommend installing it in a virtual environment.

python3 -m venv skypilot-env
source skypilot-env/bin/activate
pip install skypilot

Verify the installation:

sky --version

You should see output like `sky 0.6.0` (or newer).

2. Configure Cloud Credentials

SkyPilot needs access to your cloud accounts. For AWS, you can use the AWS CLI.

pip install awscli
aws configure

Enter your AWS Access Key ID, Secret Access Key, default region, and output format (json). SkyPilot will automatically detect these credentials.

For GCP or Azure, follow the respective CLI setup and ensure your credentials are available in the environment.

3. Install Hugging Face Hub Libraries

We will use the `huggingface_hub` Python library to interact with the Hub.

pip install huggingface_hub

Log in to your Hugging Face account:

huggingface-cli login

Follow the prompt to paste your access token (create one at `huggingface.co/settings/tokens`).

4. Set Up Git LFS for Large Files

If your datasets or models are larger than 10 MB, you need Git LFS.

git lfs install

5. Create a Hugging Face Repository for Your Data

You can create a repository via the web UI or the CLI. For example, to create a dataset repository:

huggingface-cli repo create my-ai-dataset --type dataset

This creates a public repository at `https://huggingface.co/datasets/your-username/my-ai-dataset`.

Usage Examples

Now we will walk through a concrete scenario: training a machine learning model on a dataset stored in Hugging Face, using GPU instances on AWS via SkyPilot.

Example 1: Training with a Dataset from Hugging Face

Assume you have a dataset `my-ai-dataset` on Hugging Face. You want to train a model using a GPU instance on AWS.

**Step 1: Write a SkyPilot YAML**

Create a file `train.yaml`:

name: train-on-hf-data

resources:
  accelerators: V100:1

workdir: .

setup: |
  pip install torch torchvision transformers datasets huggingface_hub

run: |
  # Download dataset from Hugging Face to local disk (no egress cost)
  huggingface-cli download your-username/my-ai-dataset --local-dir /tmp/dataset

  # Run training script
  python train.py --data-dir /tmp/dataset --epochs 10

This YAML tells SkyPilot:

• Use one V100 GPU.
• Copy the current working directory to the remote VM.
• On setup, install Python dependencies.
• On run, download the dataset from Hugging Face and execute `train.py`.

**Step 2: Launch the Job**

sky launch -c my-training-cluster train.yaml

SkyPilot will provision an AWS instance, run the setup, and execute the training. The dataset is downloaded directly from Hugging Face to the VM’s local disk—no S3 bucket, no egress fees.

**Step 3: Save Model Checkpoints Back to Hugging Face**

After training, you can upload the model to Hugging Face from within the same job:

# Inside train.py or a post-training script
from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="./output_model",
    repo_id="your-username/my-trained-model",
    repo_type="model"
)

This keeps your entire workflow within the zero-egress ecosystem.

Example 2: Multi-Cloud Failover with Zero-Egress

Suppose you want to run inference on the cheapest available cloud. SkyPilot can automatically try GCP, Azure, or AWS.

Create `infer.yaml`:

name: multi-cloud-inference

resources:
  accelerators: T4:1
  use_spot: true

file_mounts:
  /model:
    name: your-username/my-trained-model
    source: huggingface

run: |
  python serve.py --model-path /model

The `file_mounts` section mounts a Hugging Face model repository directly into the VM. This is even more elegant: the model is streamed from Hugging Face without any download step.

Launch with automatic failover:

sky launch -c inference-cluster infer.yaml --cloud aws --cloud gcp --cloud azure

If AWS spot instances are unavailable, SkyPilot will try GCP, then Azure. The model data always comes from Hugging Face, so egress costs are zero regardless of which cloud wins.

Example 3: Sharing Artifacts Across Teams

In a team setting, you can store shared datasets or models on Hugging Face as private repositories. Each team member can then run jobs on their preferred cloud without worrying about egress.

# Team member A on AWS
sky launch -c team-train train.yaml

# Team member B on GCP (same YAML)
sky launch -c team-train train.yaml --cloud gcp

Both jobs access the same Hugging Face repository. The only cost is Hugging Face storage (typically $0.01/GB/month for private repos), which is far cheaper than cloud egress.

Best Practices for Zero-Egress Workflows

1. **Use Hugging Face as your primary data lake.** Upload all datasets, pre-trained models, and checkpoints there. 2. **Leverage SkyPilot’s `file_mounts` for streaming.** When possible, mount Hugging Face repos directly instead of downloading to disk. 3. **Optimize for spot instances.** Since you are not paying egress, the compute cost becomes your only variable. Use spot instances aggressively. 4. **Implement caching.** If you repeatedly access the same data, cache it locally on the VM using `--local-dir` with huggingface-cli. 5. **Monitor storage costs.** Hugging Face storage is cheap but not free. Regularly clean up old model versions or datasets.

Potential Limitations and Workarounds

**Bandwidth:** Downloading large datasets from Hugging Face can be slower than from a cloud object store in the same region. For latency-sensitive workloads, consider caching or using Hugging Face’s CDN.

**Private repositories:** While Hugging Face does not charge egress, private repos have storage limits on the free tier. Paid plans increase these limits.

**SkyPilot’s cloud coverage:** Currently SkyPilot supports AWS, GCP, Azure, and Lambda Labs. If you need other providers, you may need to extend it.

Conclusion

Zero-egress storage with SkyPilot and Hugging Face Hub is a game-changer for AI teams tired of cloud lock-in and hidden data transfer costs. By storing your data on Hugging Face and using SkyPilot to orchestrate compute across any cloud, you achieve true portability without financial penalty.

The setup is straightforward: install SkyPilot, configure your cloud credentials, push your data to Hugging Face, and define your workloads in YAML. From there, you can train, fine-tune, or serve models on the cheapest or most available cloud resources, all while paying zero egress fees.

This approach aligns with the broader industry trend toward open, portable AI infrastructure—where data and models live in a neutral repository, and compute is a fungible commodity. As cloud providers continue to compete on GPU pricing and availability, the ability to switch between them seamlessly will become a competitive advantage. Start implementing zero-egress storage today, and let your AI workloads run free.