The OlmoEarth Platform: Geospatial inference at planetary scale

OlmoEarth 平台：行星尺度的地理空间推理

The OlmoEarth models are our family of Earth observation foundation models, pretrained on roughly 10 terabytes of multimodal satellite data. Governments, NGOs, and other mission-driven organizations are already adapting OlmoEarth for applications including deforestation monitoring, food security, and wildfire risk. OlmoEarth 模型是我们推出的地球观测基础模型系列，基于约 10 TB 的多模态卫星数据进行预训练。目前，政府、非政府组织及其他使命驱动型机构已经开始将 OlmoEarth 应用于森林砍伐监测、粮食安全和野火风险评估等领域。

At Ai2, we know how to train and release powerful open models, and for organizations with strong engineering teams, an open model is all they need to run with. But most organizations in the environmental space – the ones best placed to apply these models – don’t have the infrastructure or engineering teams that can manage the full lifecycle: labeling data, fine-tuning models, and running large-scale inference. 在 Ai2，我们深知如何训练并发布强大的开源模型。对于拥有强大工程团队的组织来说，一个开源模型足以让他们开展工作。然而，环境领域的大多数组织——尽管他们最适合应用这些模型——却缺乏管理全生命周期（包括数据标注、模型微调和运行大规模推理）所需的基础设施或工程团队。

We’ve spent more than a decade operating platforms like Skylight and EarthRanger, software that users around the world rely on every day, so it has to work every day. That experience taught us what delivering impact takes: running models cost-effectively at the right time and place, monitoring performance, turning raw outputs into actionable insights, and verifying those outputs drive the outcomes partners want. 我们花费了十多年时间运营 Skylight 和 EarthRanger 等平台，这些软件是全球用户每天依赖的工具，因此必须保证每日稳定运行。这段经历让我们明白实现影响力需要什么：在正确的时间和地点以经济高效的方式运行模型、监控性能、将原始输出转化为可操作的洞察，并验证这些输出是否推动了合作伙伴期望的结果。

That’s why we built the OlmoEarth Platform: infrastructure for taking geospatial models from fine-tuning and evaluation to large-scale inference. Inference at this scale presents its own set of challenges. Satellite imagery must be found and accessed across multiple providers, aligned across projections and resolutions, and processed efficiently. Results then have to be stitched into geographically consistent maps while the infrastructure recovers from the routine failures of distributed computing. 正因如此，我们构建了 OlmoEarth 平台：这是一套旨在将地理空间模型从微调和评估阶段推向大规模推理的基础设施。这种规模的推理带来了独特的挑战。我们必须跨多个供应商查找并获取卫星影像，在不同的投影和分辨率之间进行对齐，并进行高效处理。随后，必须将结果拼接成地理一致的地图，同时确保基础设施能够从分布式计算的常规故障中恢复。

Today, the platform can run inference across continent-scale areas in roughly a day, processing dozens of terabytes of imagery at a cost of fractions of a penny per square kilometer. Developing it meant confronting a series of engineering challenges that others working on large-scale geospatial systems are likely to encounter as well. This post walks through those challenges and the solutions we arrived at. 如今，该平台可以在大约一天内完成洲际范围的推理任务，处理数十 TB 的影像，每平方公里的成本仅为几分之一美分。开发该平台意味着要面对一系列工程挑战，而其他从事大规模地理空间系统工作的人员也可能会遇到同样的问题。本文将详细介绍这些挑战以及我们得出的解决方案。

Why satellite inference is challenging

为什么卫星推理具有挑战性

Most ML models take in a few megabytes of data and produce a result in under a second—think LLMs processing a paragraph of text or computer vision models analyzing a photo from a smartphone. Earth observation inference operates at a very different scale—a single job fine-tuning a foundation model for maximum performance can move terabytes of data and run for hours. 大多数机器学习模型只需输入几 MB 数据，并在不到一秒的时间内产生结果——例如处理一段文本的大语言模型，或分析智能手机照片的计算机视觉模型。而地球观测推理的规模则完全不同——一个为了实现最佳性能而微调基础模型的单一任务，可能需要移动数 TB 的数据并运行数小时。

The inputs may span multiple spectral bands, sensor types, and time steps across a large geographic area. They can come from several providers, each using different projections and resolutions, and may include observations that are missing or obscured by clouds. The output is itself a map, so every prediction must remain precisely aligned with the same projection and coordinate grid as the areas around it. 输入数据可能跨越广阔地理区域的多个光谱波段、传感器类型和时间步长。它们可能来自多个供应商，每个供应商使用不同的投影和分辨率，并且可能包含缺失或被云层遮挡的观测数据。输出本身就是一张地图，因此每个预测结果都必须与周围区域保持精确的投影和坐标网格对齐。

Even acquiring the data can be a major challenge. Prediction jobs often spend more time downloading and preparing imagery than running the model itself, making efficient data pipelines critical. Those pipelines must handle high-volume I/O while providing the compute needed to reproject and resample imagery. 即使是获取数据也可能是一项重大挑战。预测任务往往花费在下载和准备影像上的时间比运行模型本身还要多，因此高效的数据流水线至关重要。这些流水线必须在处理高吞吐量 I/O 的同时，提供重投影和重采样影像所需的计算能力。

The right hardware for the right task

为任务匹配合适的硬件

Because data acquisition and preparation often dominate an inference job’s runtime, assigning that work to GPUs would leave the system’s most expensive hardware doing tasks better suited to CPUs. We therefore divide each job into three stages, each matched to a distinct hardware profile: 由于数据获取和准备往往占据了推理任务运行时间的大部分，如果将这些工作分配给 GPU，就会导致系统中最昂贵的硬件去执行更适合 CPU 的任务。因此，我们将每个任务分为三个阶段，并为每个阶段匹配不同的硬件配置：

Data acquisition and preprocessing (CPU, high I/O): Fetch, reproject, align, and normalize imagery, then write it in a format optimized for fast loading during inference.
数据获取与预处理（CPU，高 I/O）： 获取、重投影、对齐并归一化影像，然后将其写入一种针对推理期间快速加载而优化的格式。
Inference (GPU): Run the model’s forward pass and write minimally processed outputs directly to storage.
推理（GPU）： 运行模型的前向传播，并将经过最少处理的输出直接写入存储。
Postprocessing (CPU): Stitch the per-window outputs together, apply masks or rescaling, and export them in user-friendly formats such as Zarr, GeoTIFF, or GeoJSON.
后处理（CPU）： 将每个窗口的输出拼接在一起，应用掩码或重缩放，并以 Zarr、GeoTIFF 或 GeoJSON 等用户友好的格式导出。

The OlmoEarth Platform distributes these stages across many machines while keeping GPUs fully utilized. Multiprocess data loaders continuously feed each GPU, while completed outputs stream directly to blob storage. OlmoEarth 平台将这些阶段分布在多台机器上，同时保持 GPU 的充分利用。多进程数据加载器持续为每个 GPU 提供数据，而完成的输出则直接流式传输到对象存储中。

One request, hundreds of workers, and thousands of processes

一个请求，数百个工作节点，数千个进程

OlmoEarth Run, the platform’s execution layer for large-scale inference jobs, divides the geographic region covered by each job into partitions sized for individual compute instances (workers), then subdivides those partitions into smaller windows that the OlmoEarth models process. OlmoEarth Run 是该平台用于大规模推理任务的执行层，它将每个任务覆盖的地理区域划分为适合单个计算实例（工作节点）的分区，然后将这些分区进一步细分为 OlmoEarth 模型可以处理的更小窗口。

Because each window can be handled independently in a separate forward pass, work on one part of the map does not need to wait for another. In practice, a state-sized area might become a hundred or so partitions, while a continent-scale run can become thousands. Adjacent partitions overlap slightly, and we reconcile that overlap when the outputs are assembled so no seam appears in the final raster. 由于每个窗口都可以在独立的前向传播中单独处理，因此地图某一部分的工作无需等待其他部分。在实践中，一个州大小的区域可能会被划分为一百个左右的分区，而洲际规模的运行则可能产生数千个分区。相邻分区会有轻微重叠，我们在组装输出时会协调这些重叠，以确保最终的栅格图像中不会出现接缝。

Because the partitions are independent, the same stage can run across thousands of compute instances at once. We recently used this approach to generate a wildfire-risk map covering all of North America. At peak, the run used roughly 19,600 CPUs and 994 GPUs in parallel, with network throughput exceeding 168 GB/s. That level of parallelism reduced an estimated 4,737 hours of serial compute to about 30.5 hours of wall-clock time—a 155× speedup. 由于分区是独立的，同一阶段可以同时在数千个计算实例上运行。我们最近使用这种方法生成了一张覆盖整个北美的野火风险地图。在峰值时，该任务并行使用了约 19,600 个 CPU 和 994 个 GPU，网络吞吐量超过 168 GB/s。这种并行化水平将估计 4,737 小时的串行计算时间缩短至约 30.5 小时的实际运行时间——实现了 155 倍的加速。

Fan-out is not unbounded, though. More workers push against cloud quotas, so parallelism is a per-run knob, one of several that we can adjust on individual jobs. Output resolution trades data volume and compute for detail; model size trades GPU time for accuracy; caching raw imagery trades storage for speed across. 不过，扩展并非没有限制。更多的工作节点会触及云配额上限，因此并行度是每个任务的一个可调参数，也是我们可以在单个任务中调整的多个参数之一。输出分辨率是在数据量和计算量与细节之间进行权衡；模型大小是在 GPU 时间与准确性之间进行权衡；缓存原始影像则是在存储空间与速度之间进行权衡。