Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

在任意云端运行 AI 工作负载，并存储于 Hugging Face：SkyPilot 实现零出口流量存储

For most teams, models and datasets live in a bucket in one region of one cloud. The GPUs you can get, whether for development, training, or serving, increasingly sit on a different cloud than your data. The moment those two come apart, you pay a cross-cloud transfer tax just to read your own data onto your own GPUs. 对于大多数团队而言，模型和数据集通常存储在某个云厂商特定区域的存储桶中。然而，无论用于开发、训练还是推理，你所能获取的 GPU 往往位于与数据不同的云平台上。一旦两者分离，你仅仅为了将数据读取到 GPU 上，就必须支付跨云传输费用。

Together with Hugging Face, we’ve joined the two halves: your models and datasets stay on the Hub, and SkyPilot runs the compute (dev, training, or serving) on whatever cluster has the GPUs. Mount a Hugging Face Bucket or any Hub repo into a SkyPilot job with one hf:// URL and the HF_TOKEN you already have, then launch it wherever capacity is. Hugging Face charges no egress, so reading your data onto those GPUs costs nothing, on any cloud. 我们与 Hugging Face 携手将这两部分结合起来：你的模型和数据集保留在 Hugging Face Hub 上，而 SkyPilot 则在任何拥有 GPU 的集群上运行计算任务（开发、训练或推理）。只需使用一个 hf:// URL 和你现有的 HF_TOKEN，即可将 Hugging Face 存储桶或任何 Hub 仓库挂载到 SkyPilot 任务中，并在任何有算力的地方启动它。由于 Hugging Face 不收取出口流量费，因此在任何云端将数据读取到 GPU 上都是免费的。

Here’s what’s new:

新功能亮点：

Your Hub data in any job. store: hf mounts a Hugging Face Bucket (read-write) or any model / dataset / Space repo (read-only) into a SkyPilot task with one hf:// URL and your existing HF_TOKEN, via MOUNT or COPY. Run it on any GPU, on any cloud. 在任意任务中使用 Hub 数据。 通过 store: hf，你可以使用一个 hf:// URL 和现有的 HF_TOKEN，通过 MOUNT 或 COPY 方式，将 Hugging Face 存储桶（读写）或任何模型/数据集/Space 仓库（只读）挂载到 SkyPilot 任务中。在任何云端的任何 GPU 上运行任务。

SkyPilot finds that job compute across 20+ clouds, Kubernetes, Slurm, and on-prem, so the same run uses whichever of your reserved or on-demand GPUs is available, on any vendor. SkyPilot 可在 20 多种云平台、Kubernetes、Slurm 和本地机房中寻找算力， 因此同一个任务可以使用你预留或按需获取的任何可用 GPU，无论其属于哪个云厂商。

No egress to read your data. Hugging Face Storage charges no egress or CDN fees, so wherever SkyPilot lands the job, it reads your models and datasets straight from the same bucket, with no per-cloud copies and no egress bill to pull them in. 读取数据零出口费用。 Hugging Face Storage 不收取出口流量费或 CDN 费用，因此无论 SkyPilot 将任务调度到哪里，它都能直接从同一个存储桶读取模型和数据集，无需在每个云端进行备份，也无需支付数据拉取的出口账单。

Xet-backed dedup. Buckets are built on Xet, so incremental checkpoints and model variants only store and transfer the chunks that changed. 基于 Xet 的去重技术。 存储桶基于 Xet 构建，因此增量检查点和模型变体仅存储和传输发生变化的数据块。

Built together. Hugging Face and SkyPilot shipped this jointly, and the Hugging Face team upstreamed the hf-mount FUSE fixes that make it work in unprivileged containers. 共同开发。 Hugging Face 和 SkyPilot 联合发布了此功能，Hugging Face 团队还向上游提交了 hf-mount FUSE 修复程序，使其能够在非特权容器中正常工作。

Hugging Face Storage is now a first-class SkyPilot backend

Hugging Face Storage 现已成为 SkyPilot 的一级后端

SkyPilot tasks already read and write cloud object stores (S3, GCS, Azure, R2, and many more) by mounting them at a local path. Hugging Face Storage now joins that list as store: hf, reached through the hf:// scheme: SkyPilot 任务此前已支持通过挂载到本地路径来读写云对象存储（如 S3、GCS、Azure、R2 等）。现在，Hugging Face Storage 作为 store: hf 加入了该列表，通过 hf:// 协议访问：

file_mounts:
  # A Hugging Face Bucket, read-write, for checkpoints, logs, processed data.
  /checkpoints:
    source: hf://buckets/my-org/qwen-sft
    store: hf
    mode: MOUNT # or COPY
  # A model repo, mounted read-only.
  /base-model:
    source: hf://Qwen/Qwen3.5-4B
    store: hf
    mode: MOUNT
  # A dataset repo, pinned to a revision, read-only.
  /data:
    source: hf://datasets/my-org/my-dataset@main
    store: hf
    mode: MOUNT

That one hf:// scheme covers the whole lifecycle: read the model and dataset from their repos, write checkpoints to a Bucket while you train, publish the finished model back to a repo, and pull it onto inference servers when you serve. Most teams already keep their models and datasets on the Hub, so there is no migration step and no new storage account to create. 这一个 hf:// 协议涵盖了整个生命周期：从仓库读取模型和数据集，在训练时将检查点写入存储桶，将完成的模型发布回仓库，并在推理时将其拉取到推理服务器。大多数团队已经将模型和数据集保存在 Hub 上，因此无需迁移步骤，也无需创建新的存储账户。

MOUNT uses Hugging Face’s hf-mount FUSE backend, so a bucket or repo shows up as a local path next to SkyPilot’s other FUSE mounts (gcsfuse, blobfuse2, rclone, goofys). The fetching happens at the filesystem layer: when your code issues a read(), the driver pulls just those bytes from the Xet backend, so only the data you actually touch crosses the network, and hf-mount keeps an on-disk cache so repeat reads stay local. MOUNT 使用 Hugging Face 的 hf-mount FUSE 后端，因此存储桶或仓库会像 SkyPilot 的其他 FUSE 挂载（gcsfuse, blobfuse2, rclone, goofys）一样显示为本地路径。数据获取发生在文件系统层：当你的代码发出 read() 请求时，驱动程序仅从 Xet 后端拉取所需字节，因此只有你实际访问的数据才会通过网络传输。同时，hf-mount 会保留磁盘缓存，确保重复读取时数据保持在本地。

That on-disk cache is the behavior SkyPilot gives its other backends under MOUNT_CACHED, where a plain MOUNT instead streams every read from the bucket with nothing kept locally. For the hf store, MOUNT and MOUNT_CACHED behave the same, so either mode keeps the cache. Because reads are lazy, a process can start working through a large file before the whole file has downloaded, instead of blocking on a full copy first. That keeps the GPU busy almost immediately, training on data as it streams in rather than sitting idle (and billing) while a dataset or checkpoint copies down. It pays off most on the first epoch, when nothing is cached yet. 这种磁盘缓存行为与 SkyPilot 在 MOUNT_CACHED 下为其他后端提供的功能一致，而普通的 MOUNT 则会从存储桶流式传输每次读取，不在本地保留任何内容。对于 hf 存储，MOUNT 和 MOUNT_CACHED 的行为相同，因此两种模式都会保留缓存。由于读取是惰性的，进程可以在整个文件下载完成之前就开始处理大文件，而不必先等待完整拷贝。这使得 GPU 能够几乎立即投入工作，在数据流式传输时就开始训练，而不是在等待数据集或检查点下载时处于空闲（并产生费用）状态。这在第一个 epoch（此时尚未缓存任何内容）时效果最为显著。

COPY takes the other route and downloads through huggingface_hub up front, with no special requirements. COPY 则采用另一种方式，通过 huggingface_hub 预先下载，没有任何特殊要求。

Authentication is the token you already have. Set HF_TOKEN in your environment and hand it to a run with --secret HF_TOKEN; SkyPilot uses it for the mount on whatever cloud the job lands. One token works whether the job lands on AWS, GCP, Azure, Nebius, Lambda, or your own Kubernetes cluster, so there are no per-cloud bucket keys to juggle. 身份验证使用你现有的令牌。在环境中设置 HF_TOKEN 并通过 --secret HF_TOKEN 传递给运行任务；无论任务落在哪个云端，SkyPilot 都会使用它进行挂载。无论任务是在 AWS、GCP、Azure、Nebius、Lambda 还是你自己的 Kubernetes 集群上运行，同一个令牌都适用，因此无需管理各个云厂商的存储桶密钥。

No egress: storage stops deciding where you run

零出口流量：存储不再限制你的运行位置

GPU capacity rarely comes from one place anymore. To get enough H100s and H200s, teams hold reserved and committed capacity across several vendors at once (a block on a hyperscaler, a cluster on a neocloud, maybe an on-prem rack) and run wherever they have allocation. SkyPilot is built for this: one job spec, scheduled across 20+ clouds, Kubernetes, and on-prem, landing on whichever reserved cluster is free. GPU 算力现在很少来自单一来源。为了获得足够的 H100 和 H200，团队通常同时在多个供应商处持有预留和承诺的算力（超大规模云厂商的资源块、新一代云厂商的集群，甚至是本地机架），并在有配额的地方运行。SkyPilot 正是为此而生：一个任务规范，可调度到 20 多个云平台、Kubernetes 和本地机房，并自动落在任何空闲的预留集群上。

Object storage has been the catch. Object stores are regional and per-cloud, so feeding a GPU or an inference server that sits in a different vendor’s data center means either keeping a copy of your data in every vendor’s bucket or paying to pull it across. Most clouds charge egress (around $0.09/GB out of AWS) the moment data leaves their network, and often between regions inside one cloud. Pulling a base model onto every inference node, or iterating a dataset for several epochs from a cluster on another cloud, adds a hefty bill on top of GPUs you have already reserved. Teams end up pinning each run to whichever vendor holds the data and leaving the rest of their capacity idle. 对象存储一直是痛点。对象存储是区域性和云厂商绑定的，因此要为位于不同供应商数据中心的 GPU 或推理服务器提供数据，要么在每个供应商的存储桶中保留一份数据副本，要么支付跨云拉取费用。大多数云厂商在数据离开其网络时就会收取出口费用（从 AWS 传出约为 $0.09/GB），甚至在同一云厂商的不同区域之间也会收费。将基础模型拉取到每个推理节点，或者从另一个云端的集群对数据集进行多次 epoch 迭代，会在你已经预留的 GPU 费用之上增加一笔巨额账单。最终，团队只能被迫将每个任务绑定到存储数据的供应商，导致其余算力闲置。

Hugging Face Storage takes that cost off the table where it bites: the read side. With no egress or CDN fees and storage at $12-18/TB/month (versus AWS S3 at roughly $23/TB plus egress), the same bucket is reachable from every one of those clusters, and reading from it is free no matter where the GPUs run. Writing back still costs your compute cloud’s… Hugging Face Storage 解决了最棘手的成本问题：读取端。由于没有出口流量费或 CDN 费用，且存储价格为 $12-18/TB/月（相比之下，AWS S3 约为 $23/TB 外加出口费），同一个存储桶可以从上述所有集群中访问，无论 GPU 在哪里运行，读取数据都是免费的。回写数据仍需支付你计算云厂商的费用……