ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
ZGateway：在 ZippyDB 前端部署代理的经验教训
By Rittik Banik, Yunhao Cao 作者：Rittik Banik, Yunhao Cao
We’re introducing ZGateway, the proxy we are using to unify traffic through ZippyDB, Meta’s most widely-used key value store. As a bonus, it also enables admission control, load balancing, cross-region resilience, and richer operations. 我们在此介绍 ZGateway，这是我们用于统一 Meta 最广泛使用的键值存储 ZippyDB 流量的代理。作为额外收益，它还实现了准入控制、负载均衡、跨区域弹性以及更丰富的运维能力。
ZippyDB is the most widely used key value store at Meta, backing product metadata, counters, and configuration, and can serve billions of operations per second across a globally distributed fleet. A previous post described how ZippyDB works. This post is about the layer in front of it: ZGateway, the proxy through which we are unifying ZippyDB client traffic. ZippyDB 是 Meta 最广泛使用的键值存储，支撑着产品元数据、计数器和配置，能够在全球分布式集群中每秒处理数十亿次操作。之前的文章介绍了 ZippyDB 的工作原理。本文将重点介绍其前端层：ZGateway，即我们用于统一 ZippyDB 客户端流量的代理。
ZGateway grew out of the need to manage the sprawl of ZippyDB’s client fleet, and its value turned out to be structural. A ZippyDB client might be one of more than a million hosts, owned by hundreds of teams, that we cannot change quickly. A proxy sits in a different position entirely — in the path of many clients at once — and that shared vantage point lets it do what no single client could. Every capability below is easier, safer, or only possible at all in one managed tier rather than in a million client binaries. We tell the story through the two clearest cases, connection management and request batching. ZGateway 的诞生源于管理 ZippyDB 庞大客户端集群的需求，其价值最终被证明是结构性的。ZippyDB 的客户端可能分布在超过一百万台主机上，由数百个团队拥有，我们无法对其进行快速变更。代理则处于完全不同的位置——同时位于许多客户端的路径上——这种共享的视角使它能够完成任何单一客户端无法做到的事情。以下提到的每一项功能，在统一管理的层级中实现都比在百万个客户端二进制文件中实现更简单、更安全，甚至只有前者才可行。我们将通过两个最清晰的案例——连接管理和请求批处理——来讲述这个故事。
Why ZippyDB Needed a Proxy Layer
为什么 ZippyDB 需要代理层
Proxies show up wherever a large, diverse client population talks to a shared backend: a connection pooler, a service mesh sidecar, a CDN edge, an API gateway. Interposing between many callers and a shared resource buys three things. It bounds the problem: The backend stops seeing the client population and starts seeing a fleet its own operators control. It creates a home for shared work — pooling, retries, routing, caching, admission control — solved once by the team that knows the backend best, instead of in every client. It creates a control point: The only place you can see the whole workload, attribute load to whoever generated it, and change behavior in minutes rather than waiting out a fleet-wide client rollout. The tradeoff is a hop and one more tier to operate; the trade pays off when the client population is large, diverse, and not yours to change. 无论是在大型、多样化的客户端群体与共享后端通信的场景中，还是在连接池、服务网格边车（sidecar）、CDN 边缘或 API 网关中，代理都无处不在。在众多调用者和共享资源之间插入代理可以带来三个好处。它界定了问题范围：后端不再直接面对庞大的客户端群体，而是面对由其自身运维人员控制的集群。它为共享工作（如连接池、重试、路由、缓存、准入控制）提供了一个归宿，由最了解后端的团队统一解决，而不是在每个客户端中重复实现。它创建了一个控制点：这是唯一可以查看整体工作负载、将负载归因于生成者，并在几分钟内改变行为的地方，而无需等待全集群范围的客户端部署。其代价是增加了一跳和需要运维的额外层级；但当客户端群体庞大、多样且无法由你直接控制时，这种权衡是值得的。
ZippyDB is an extreme case. In the direct-access model, every ZippyDB client connects to every database host it needs. A single client can touch tens of thousands of distinct shards in a stable window; those shards sit across hundreds of thousands of database hosts. The result is a dense many-to-many mesh of TLS connections: A typical client holds tens of thousands of outbound connections, and a typical database host can accept tens of thousands of inbound ones. ZippyDB 是一个极端的案例。在直接访问模型中，每个 ZippyDB 客户端都会连接到它所需的所有数据库主机。单个客户端在稳定窗口期内可能会触及数万个不同的分片；而这些分片分布在数十万台数据库主机上。结果形成了一个密集的 TLS 连接多对多网格：典型的客户端持有数万个出站连接，而典型的数据库主机则需要接受数万个入站连接。
Figure 1: Direct access produces an unbounded many-to-many mesh; ZGateway collapses client fan-out and database-host fan-in into bounded numbers. 图 1：直接访问产生了一个无边界的多对多网格；ZGateway 将客户端的扇出（fan-out）和数据库主机的扇入（fan-in）压缩为有限的数值。
That mesh is wasteful and fragile. Every open connection consumes memory, CPU, and a file descriptor on both ends, mostly while idle, and inbound connections grow with the client population, so every new client cohort makes every database host worse. And because each client manages its own pooling and failover, a sudden drop in connection reuse — a cohort restarts, a deploy rolls — hits the fleet with a storm of new connections; we have traced host crashes from file-descriptor exhaustion and OOMs to exactly this. It is hard to fix client-side because two systems move at once. Client fleets keep changing pooling policies and expanding while the database fleet consolidates on its own schedule. Coupling them directly feels like jumping between two moving cars. A proxy decouples them, collapsing the mesh into two bounded hops — a win on efficiency, performance, scalability, and above all reliability. 这种网格既浪费又脆弱。每个打开的连接在两端都会消耗内存、CPU 和文件描述符，且大部分时间处于空闲状态。入站连接随着客户端数量的增加而增长，因此每一个新的客户端群体都会加重数据库主机的负担。由于每个客户端都自行管理连接池和故障转移，连接重用率的突然下降（例如群体重启、部署滚动）会引发连接风暴冲击集群；我们曾追踪到主机因文件描述符耗尽和内存溢出（OOM）而崩溃，正是源于此。在客户端侧修复很难，因为两个系统在同时变动。客户端集群不断更改连接池策略并进行扩展，而数据库集群则按自己的节奏进行整合。将它们直接耦合就像在两辆行驶的汽车之间跳跃。代理将它们解耦，将网格压缩为两跳——这在效率、性能、可扩展性，尤其是可靠性方面都是一种胜利。
That last one matters most. Under direct access a reconnection storm becomes a catastrophe. In one incident a routing bug led every client to open a connection per shard, hosts breached their file-descriptor limit, and the fleet fell into a reboot loop. With ZGateway in the path that storm is contained at the proxy tier — a fleet we control, observe, and can harden centrally. Connection management does not disappear behind a proxy; it moves to the one place we can solve it. 最后一点最为重要。在直接访问模式下，重连风暴可能演变成灾难。在一次事故中，一个路由错误导致每个客户端为每个分片打开一个连接，主机突破了文件描述符限制，导致集群陷入重启循环。有了 ZGateway，风暴被限制在代理层——这是一个我们可以控制、观察并能集中加固的集群。连接管理并没有在代理背后消失，而是转移到了我们能够解决它的唯一地方。
Direct access was the right design for the initial part of ZippyDB’s life. Fan-in scales with the client population, so the resource use grows with adoption: at a few thousand clients the mesh is an inefficiency, at scale it is a reliability limit. The means arrived on the same schedule — ServiceRouter feature improvements, Thrift overload protection, a thin client, and other features are what make a shared tier feasible, and building ZGateway earlier would have meant building each of those first. 直接访问是 ZippyDB 初期阶段的正确设计。扇入（fan-in）随客户端数量扩展，因此资源使用量随采用率增长：在几千个客户端时，网格只是效率低下，但在大规模时，它就成了可靠性的瓶颈。实现这一目标的手段是同步到位的——ServiceRouter 功能改进、Thrift 过载保护、轻量级客户端以及其他特性使得共享层成为可能，如果过早构建 ZGateway，意味着必须先构建所有这些基础功能。
What ZGateway Is
ZGateway 是什么
ZGateway is a stateless proxy tier between ZippyDB clients and the database (ZServer) fleet. It can handle greater than 1 billion operations per second and carries about 40% of all ZippyDB traffic, projected to grow past 60%, while adding only about 6% computational overhead to an average use case. It runs as regional tiers discovered through ServiceRouter, Meta’s hyperscale service mesh solution, keeping every client near its gateway, in two flavors sharing one pipeline: a pure proxy and a read-through cache. It runs our thick C++ client as its engine, one internal client per use case. ZGateway is a ZippyDB client run as a managed service, which made moving capability here natural. ZGateway 是位于 ZippyDB 客户端和数据库（ZServer）集群之间的无状态代理层。它每秒可处理超过 10 亿次操作，承载了约 40% 的 ZippyDB 总流量，预计将增长至 60% 以上，同时对平均用例仅增加约 6% 的计算开销。它作为通过 Meta 超大规模服务网格解决方案 ServiceRouter 发现的区域层运行，使每个客户端都能靠近其网关。它有两种共享同一流水线的模式：纯代理和直读缓存。它以我们厚重的 C++ 客户端作为引擎，每个用例对应一个内部客户端。ZGateway 本质上是以托管服务形式运行的 ZippyDB 客户端，这使得将功能迁移到此处变得顺理成章。
Figure 2: The ZGateway request path, from a client’s sticky regional connection to the ZServer replicas. A client sends a request over its sticky connection to a regional ZGateway host, which terminates TLS, authorizes it against the use case’s ACLs, and applies per-tenant admission control, validation, and shaping. ZGateway resolves the shard, checks the local cache on a caching tier, and batches and coalesces misses and writes with other in-flight requests for that shard before sending them to the cor… 图 2：ZGateway 请求路径，从客户端的粘性区域连接到 ZServer 副本。客户端通过其粘性连接向区域 ZGateway 主机发送请求，该主机终止 TLS，根据用例的 ACL 进行授权，并应用租户级准入控制、验证和流量整形。ZGateway 解析分片，检查缓存层上的本地缓存，并在将未命中请求和写入请求发送到…之前，将其与其他针对该分片的进行中请求进行批处理和合并。