SkyPilot and Hugging Face bring zero-egress storage to multi-cloud AI

SkyPilot and Hugging Face have integrated Hugging Face Storage as a native SkyPilot backend, letting teams keep models and datasets on the Hub while running compute on whichever cloud has available GPUs. A new store: hf option mounts a Hugging Face Bucket or any model/dataset repo into a job with a single hf:// URL and an existing HF_TOKEN, using either MOUNT or COPY mode.

The key change is cost: Hugging Face charges no egress or CDN fees, so reading data onto GPUs is free regardless of which of the 20+ supported clouds, Kubernetes, Slurm, or on-prem cluster SkyPilot schedules the job on. This removes the usual penalty of pinning workloads to whichever vendor holds a data copy, letting teams use all their reserved GPU capacity across providers.

MOUNT mode uses the hf-mount FUSE driver for lazy reads, pulling only the bytes a process actually touches so training can start almost immediately. Buckets are built on Xet, so checkpoints and model variants only store and transfer the chunks that changed. In a benchmark fine-tuning job run identically on AWS, GCP, and Lambda, model loading took about 30 seconds at no cost, while checkpoint writes ran at roughly 110-170 MB/s per cloud.