Mesh LLM: distributed AI computing on iroh
Mesh LLM: distributed AI computing on iroh
Mesh LLM：基于 iroh 的分布式 AI 计算
When people picture running a large language model, they picture a data center. Racks of GPUs that belong to someone else, a metered API, and a bill that grows every month you succeed. You send your prompts off to a black box and hope the price, the model, and the privacy policy all stay the way they were when you signed up. For a lot of teams that is a bad trade. You give up control over when models change, where your data goes, and what hardware runs your workloads. And as usage grows, so does the bill, with no lever to pull except “pay more.”
当人们想象运行大语言模型时，脑海中浮现的往往是数据中心：属于别人的 GPU 机架、按量计费的 API，以及随着业务增长而逐月增加的账单。你将提示词发送到一个“黑盒”中，并祈祷价格、模型版本和隐私政策能保持不变。对于许多团队来说，这并不是一笔划算的交易。你放弃了对模型更新时间、数据流向以及运行负载的硬件的控制权。随着使用量的增加，账单也随之水涨船高，除了“多付钱”之外，你别无选择。
Mesh LLM is a different shape. It pools the GPUs and memory you already have, across as many machines as you want to add, and exposes the whole thing as one OpenAI-compatible API. Start one node. Add more later. Let the mesh decide whether a model runs on the box in front of you, routes to a peer, or splits across several machines.
Mesh LLM 则呈现出不同的形态。它将你现有的 GPU 和内存汇聚起来，跨越任意数量的机器，并将整个集群统一呈现为一个兼容 OpenAI 的 API。你可以先启动一个节点，后续再添加更多。让网格（Mesh）自动决定模型是在你面前的机器上运行，还是路由到对等节点，亦或是拆分到多台机器上协同处理。
The problem: AI is expensive, and it is somebody else’s
问题所在：AI 昂贵且受制于人
The popular models are monoliths. Most people reach them through a UI or an API key and pay a large provider to run everything. That is convenient, and it is also a surrender. You do not control when the model gets updated, what memory it runs in, or what hardware sits underneath. Plenty of businesses and services that depend on these models want the opposite: more control, more pluggability, lower cost. They have GPUs sitting in offices, in closets, under desks. What they are missing is a way to make those machines act like one.
主流模型都是庞大的单体。大多数人通过 UI 或 API Key 访问它们，并向大型供应商付费以运行一切。这虽然方便，但也是一种妥协。你无法控制模型何时更新、在什么内存中运行，或者底层硬件是什么。许多依赖这些模型的企业和服务恰恰想要相反的东西：更多的控制权、更强的可插拔性以及更低的成本。他们的办公室、储藏室或桌子底下闲置着许多 GPU，他们所缺少的，是将这些机器整合为一体的方法。
Mesh LLM: run the models yourself
Mesh LLM：自己运行模型
The pitch is simple. Run bigger models without buying bigger GPUs. Share compute privately with your team, or publicly with the world, to power agents and chat. Point any OpenAI client at http://localhost:9337/v1 and stop caring where the work actually happens.
核心理念很简单：无需购买更强大的 GPU 即可运行更大的模型。你可以与团队私下共享算力，或向全球公开，为智能体和聊天应用提供支持。只需将任何 OpenAI 客户端指向 http://localhost:9337/v1，就不必再关心计算任务究竟在哪里执行。
Under the hood, Mesh LLM distributes model compute across a mesh of iroh endpoints. A request can be served three ways:
- Run it locally, on this machine’s GPU.
- Route it to a peer that already has the model loaded.
- Split a model too big for any single box across several machines, as a pipeline.
在底层，Mesh LLM 将模型计算分布在由 iroh 端点组成的网格中。请求可以通过三种方式处理：
- 在本地机器的 GPU 上运行。
- 路由到已经加载了该模型的对等节点。
- 将单机无法容纳的大模型拆分到多台机器上，以流水线方式运行。
How it works
工作原理
The architecture is pluggable. Plugins declare what they provide in a manifest, the runtime starts them, routes calls, and exposes their capabilities over MCP, HTTP, inference, and mesh events. The catalog ships with 40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants.
该架构具有可插拔性。插件在清单中声明其提供的功能，运行时负责启动插件、路由调用，并通过 MCP、HTTP、推理和网格事件暴露其能力。目录中预置了 40 多种模型，从适合笔记本电脑运行的 5 亿参数模型，到 2350 亿参数的混合专家（MoE）巨型模型。
For the giants, Mesh LLM has a split mode (internally, “Skippy”). A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Activations flow from one stage to the next, so several modest machines can run a model none of them could hold alone. The OpenAI client never sees any of this. It still just talks to localhost.
对于巨型模型，Mesh LLM 具备一种拆分模式（内部代号为“Skippy”）。模型按层范围被划分为多个阶段：例如节点 A 处理第 0-15 层，节点 B 处理第 16-31 层，以此类推。激活值在各阶段间流动，因此几台性能普通的机器就能运行单机无法承载的模型。OpenAI 客户端对此一无所知，它依然只与 localhost 通信。
How it uses iroh
如何使用 iroh
Every node, whether it serves models or only sends requests, boots an iroh endpoint. That endpoint is the node’s identity, a public key, and its only network surface. There is no central server. iroh handles the hole-punching, NAT traversal, and relay fallback needed to open a direct, authenticated QUIC connection between any two nodes, wherever they sit. To keep that working across the open internet, Mesh LLM runs two iroh relays in different regions, so nodes that cannot reach each other directly always have a fallback path nearby.
每个节点（无论是提供模型服务还是仅发送请求）都会启动一个 iroh 端点。该端点即节点的身份标识（公钥），也是其唯一的网络入口。这里没有中心服务器。iroh 负责处理打洞、NAT 穿透以及中继回退，从而在任意两个节点之间建立直接且经过身份验证的 QUIC 连接。为了确保在公网上稳定运行，Mesh LLM 在不同区域运行了两个 iroh 中继，确保无法直接连接的节点总能找到附近的备用路径。
The whole protocol rides on QUIC’s ALPN negotiation. There are three: 整个协议基于 QUIC 的 ALPN 协商，包含三种协议：
| ALPN | What it carries |
|---|---|
mesh-llm/1 | Main mesh: gossip, routing, HTTP tunnels, plugin channels |
mesh-llm-control/1 | Owner control plane (config sync, ownership attestation) |
skippy-stage/2 | Latency-sensitive activation transport for split models |
| ALPN | 承载内容 |
|---|---|
mesh-llm/1 | 主网格：Gossip 协议、路由、HTTP 隧道、插件通道 |
mesh-llm-control/1 | 所有者控制平面（配置同步、所有权认证） |
skippy-stage/2 | 针对拆分模型的低延迟激活值传输 |
Inside the main mesh-llm/1 connection, everything is a bidirectional QUIC stream tagged with a single leading byte that says what kind of stream it is. One connection carries gossip, inference, route queries, and peer-lifecycle events, all demuxed by that first byte:
在 main mesh-llm/1 连接内部，所有数据都是双向 QUIC 流，并由一个前导字节标记流的类型。一个连接可以同时承载 Gossip、推理、路由查询和对等节点生命周期事件，所有这些都通过第一个字节进行多路复用：
| Byte | Stream type | Description |
|---|---|---|
| 0x01 | GOSSIP | peer announcements (models, GPU, RTT, capabilities) |
| 0x04 | TUNNEL_HTTP | inference requests proxied to a peer |
| 0x05 | ROUTE_REQUEST | ”which models do you host?“ |
| 0x06 | PEER_DOWN | dead-peer notification |
| 0x07 | PEER_LEAVING | graceful shutdown |
| 0x08 | PLUGIN_CHANNEL | plugin RPC |
| 0x0e | DIRECT_PATH_REQUEST | share direct addresses for NAT traversal |
| 字节 | 流类型 | 描述 |
|---|---|---|
| 0x01 | GOSSIP | 对等节点公告（模型、GPU、RTT、能力） |
| 0x04 | TUNNEL_HTTP | 代理到对等节点的推理请求 |
| 0x05 | ROUTE_REQUEST | “你托管了哪些模型？” |
| 0x06 | PEER_DOWN | 节点离线通知 |
| 0x07 | PEER_LEAVING | 优雅关机 |
| 0x08 | PLUGIN_CHANNEL | 插件 RPC |
| 0x0e | DIRECT_PATH_REQUEST | 分享用于 NAT 穿透的直接地址 |
The neat part is what this buys you. iroh gives authenticated, NAT-traversing QUIC between any two machines, addressed by public key. So “route to a peer” and “stream activations to the next pipeline stage” become the same primitive as “talk to localhost,” just with a different endpoint ID. The networking stops being something you have to think about. iroh provides the secure transport. Mesh LLM builds its own gossip layer on top, so it controls exactly who gets admitted to the mesh, which versions are compatible, and which peers to trust.
最巧妙的地方在于它带来的收益。iroh 在任意两台机器之间提供了基于公钥寻址的、经过身份验证且支持 NAT 穿透的 QUIC 连接。因此，“路由到对等节点”和“将激活值流式传输到下一个流水线阶段”变成了与“与 localhost 通信”相同的原语，只是端点 ID 不同。网络配置不再是你需要操心的问题。iroh 提供了安全传输层，而 Mesh LLM 在其之上构建了自己的 Gossip 层，从而精确控制谁可以加入网格、哪些版本兼容以及信任哪些对等节点。
Getting started
开始使用
Users can install the lightweight software (about 18 MB) and either join the public mesh or configure private deployments. The system presents itself as localhost:9337/v1 to any standard OpenAI client. A mobile app is coming, built on iroh’s Swift SDK. The plan is to speak ACP, the emerging agent standard, so other clients can join the mesh too. The throughline is the same one that motivated the whole project: more peer to peer, fewer closed servers, and no lock-in.
用户可以安装这个轻量级软件（约 18 MB），既可以加入公共网格，也可以配置私有部署。该系统对任何标准 OpenAI 客户端都表现为 localhost:9337/v1。基于 iroh Swift SDK 开发的移动端应用也即将推出。未来的计划是支持新兴的智能体标准 ACP，以便其他客户端也能加入网格。贯穿始终的理念与该项目的初衷一致：更多的点对点连接、更少的封闭服务器，以及拒绝厂商锁定。
See the code | Mesh LLM Website
Iroh is a dial-any-device networking library that just works. Compose from an ecosystem of ready-made protocols to get the features you need, or go fully custom on a clean abstraction over dumb pipes. Iroh is open source, and already running in production on hundreds of thousands of devices. To get started, take a look at our docs, dive directly into the code, or chat with us in our discord channel.
Iroh 是一个能够连接任何设备的网络库，且开箱即用。你可以从现成的协议生态中组合所需功能，或者在简洁的抽象层上进行完全自定义开发。Iroh 是开源的，目前已在数十万台设备上投入生产使用。如需开始使用，请查看我们的文档、直接深入代码，或在我们的 Discord 频道中与我们交流。