Tailscale didn’t stop the Hugging Face intrusion

Tailscale 未能阻止 Hugging Face 入侵事件

Blog | insights | July 31, 2026 博客 | 洞察 | 2026年7月31日

An AI agent escaped its sandbox, entered Hugging Face’s infrastructure, and used a stolen Tailscale credential to enroll 181 nodes onto their tailnet. No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway. 一个 AI 智能体逃离了沙箱，进入了 Hugging Face 的基础设施，并利用窃取的 Tailscale 凭据将 181 个节点注册到了他们的 Tailnet（私有网络）中。此次事件中并未发现或利用任何 Tailscale 的漏洞——但我们本应能够阻止它。

By now, you’ve likely heard about the AI agent that escaped a security evaluation and attacked Hugging Face, an LLM marketplace. The agent decided Hugging Face might have the answers to its benchmark, so it stole them just to cheat on the exam. That’s a funny motive but a scary outcome. 到目前为止，你可能已经听说过那个逃离安全评估并攻击了 LLM 市场 Hugging Face 的 AI 智能体。该智能体认为 Hugging Face 可能拥有其基准测试的答案，于是为了在考试中作弊而窃取了这些数据。这个动机听起来很有趣，但结果却令人恐惧。

Hugging Face published a detailed reconstruction of the intrusion. It covers about 17,600 recovered actions over four and a half days, including sandbox escapes, code execution, cloud credentials, improvised command-and-control systems, and eventually, the use of Tailscale to spread throughout their organization. Hugging Face 发布了此次入侵的详细重构报告。报告涵盖了在四天半内恢复的约 17,600 个操作，包括沙箱逃逸、代码执行、云凭据获取、临时命令与控制系统，以及最终利用 Tailscale 在其组织内部进行横向渗透。

But Tailscale is a zero trust network! The whole point of zero trust is to prevent attackers from moving laterally across your company. What went wrong? 但 Tailscale 是一个零信任网络！零信任的核心意义就在于防止攻击者在公司内部进行横向移动。究竟哪里出了问题？

Where Tailscale appeared

Tailscale 出现在哪里

No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. Tailscale turns up everywhere in AI infrastructure: virtually every AI company uses Tailscale. So it’s not surprising we were in an AI incident report eventually. Tailscale 中没有发现或被利用的“漏洞”，这反而让我们感到更加不安。Tailscale 出现在 AI 基础设施的各个角落：几乎每家 AI 公司都在使用 Tailscale。因此，我们最终出现在 AI 事件报告中并不令人意外。

But, we’re a security tool. Their intrusion is our intrusion, and it’s our job to take it seriously. As in any post-mortem, we need to reflect on what happened and what we could have done to prevent it. 但是，我们是一款安全工具。他们的入侵就是我们的入侵，认真对待此事是我们的职责。正如任何事后分析一样，我们需要反思发生了什么，以及我们本可以做些什么来预防它。

By the time the agent escaped its sandbox and found Tailscale, it had already gained code execution privileges inside a production worker, reached root access on a Kubernetes node, and read a production secret store containing 136 keys. In a sense, it was game over before we even arrived on the scene. 当该智能体逃离沙箱并找到 Tailscale 时，它已经获得了生产工作节点内的代码执行权限，取得了 Kubernetes 节点的 root 访问权限，并读取了一个包含 136 个密钥的生产环境密钥库。从某种意义上说，在我们介入之前，游戏就已经结束了。

But isn’t that the problem? Let’s back up a level or two. Why were long-lived secret keys accessible at all to anyone? 但这难道不是问题所在吗？让我们退后一步。为什么长效密钥会被任何人访问到？

Because long-lived credentials are the standard

因为长效凭据是行业标准

They shouldn’t be, though. And we’ve now seen a great example of why. In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team’s low-priority list. 然而，它们本不该如此。我们现在看到了一个很好的例子，说明了原因。在过去，大多数入侵是由人类以人类的速度完成的，凭据泄露的缓解措施被视为“锦上添花”。一个可以一次性读取 136 个密钥的大型凭据库，在安全团队的待办事项列表中通常处于低优先级。

Now, in a world of rogue AI agents, the big credential vault is the prize. It’s not okay anymore. 现在，在一个充满恶意 AI 智能体的世界里，大型凭据库就是攻击者的战利品。这种情况已不再被允许。

I only know of two main solutions to long-lived credentials. 我只知道解决长效凭据问题的两种主要方案。

First, you can configure a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back. HashiCorp Vault can do this, for example; they call it dynamic credentials. Unfortunately, dynamic credentials are a lot of work to set up and maintain. When security requires work, people don’t do it. 首先，你可以配置一个保险库（Vault），它仅根据你一次性输入且永不外泄的长效凭据来颁发短效凭据。例如，HashiCorp Vault 可以做到这一点；他们称之为动态凭据。不幸的是，动态凭据的设置和维护工作量很大。当安全措施需要额外工作时，人们往往不会去执行。

The second option is a credential-injecting proxy. If client C wants to access server S, don’t give C credentials for S; instead, have it send its requests via a hardened proxy P that does nothing but receive a request, insert a credential, and pass it through. Credential injection is gaining popularity now that modern mesh-based networking removes most of its latency. A few months ago, we acquired Border0 (“Tailscale PAM”), which makes an elegant and quite fancy credential-injecting proxy, tightly integrated with Tailscale. Border0, fully deployed, would have prevented reading those 136 keys and logged every attempt to use one. (The same credential injection powers Aperture Connectors.) But, this stuff is too recent. Most of our customers haven’t even looked at it yet. 第二个选择是凭据注入代理。如果客户端 C 想要访问服务器 S，不要给 C 提供 S 的凭据；相反，让它通过一个加固的代理 P 发送请求，该代理只负责接收请求、插入凭据并转发。随着现代网格网络消除了大部分延迟，凭据注入正变得越来越流行。几个月前，我们收购了 Border0（“Tailscale PAM”），它提供了一个优雅且功能强大的凭据注入代理，并与 Tailscale 紧密集成。如果完全部署了 Border0，本可以阻止对那 136 个密钥的读取，并记录下每一次使用尝试。（同样的凭据注入技术也驱动了 Aperture Connectors。）但是，这些技术太新了，我们的大多数客户甚至还没有关注到它。

A bonus third option: Tailscale itself. In Tailscale, your node key can be machine bound using a TPM to prevent export, and if you’re an enterprise you should definitely do that. Alas, we had to turn TPM storage off by default on Linux and Windows, because of HSM issues we found on some users’ hardware. Anyway, in today’s case, the attack is what granted access to Tailscale, so we arrived too late. 第三个额外选项：Tailscale 本身。在 Tailscale 中，你的节点密钥可以使用 TPM 绑定到机器以防止导出，如果你是企业用户，绝对应该这样做。遗憾的是，由于我们在部分用户的硬件上发现了 HSM 问题，我们不得不默认关闭 Linux 和 Windows 上的 TPM 存储。无论如何，在今天的案例中，攻击行为本身就是获取 Tailscale 访问权限的途径，所以我们介入得太晚了。

That long-lived auth key didn’t need to exist

那个长效认证密钥本不需要存在

Neither dynamic credentials nor credential-injecting proxies were in place, so 136 keys leaked. What’s the next line of defense? 由于既没有部署动态凭据，也没有部署凭据注入代理，导致 136 个密钥泄露。下一道防线是什么？

One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get. 那 136 个凭据中有一个是可重复使用的 Tailscale 认证密钥，用于在他们的 Tailnet 中创建新的 Tailscale CI（持续集成，用于自动化测试）节点。该智能体将该密钥复制到一系列外部沙箱中，并在几天内利用它将总共 181 个节点注册到了 Hugging Face 的 Tailnet 中。这些节点每个都获得了 Tailscale 身份标签，从而拥有了 CI 节点所能获得的所有访问权限。

We couldn’t save those 136 keys. But a reusable Tailscale key didn’t need to be among them. 我们无法挽救那 136 个密钥。但其中本不该包含一个可重复使用的 Tailscale 密钥。

We built workload identity federation for cases like this. It relies on your cloud provider to offer “ambient authority” in the form of short-lived creds generated from a running VM’s (or container’s) cloud identity. The CI job can ask its cloud platform for a signed OIDC token. Tailscale verifies the token, then grants access with the tags and scopes assigned to that workload. What’s neat is this can happen automatically once enabled: spin up the CI node, Tailscale gets the identity, assigns the right tags. There’s no credential to leak and, when configured carefully, no way to use this credential anywhere but in CI. 我们针对此类情况构建了工作负载身份联合（Workload Identity Federation）。它依赖于你的云服务提供商，通过运行中的虚拟机（或容器）的云身份生成短效凭据，从而提供“环境授权”。CI 任务可以向其云平台请求一个签名的 OIDC 令牌。Tailscale 验证该令牌，然后根据分配给该工作负载的标签和范围授予访问权限。巧妙之处在于，一旦启用，这一切都可以自动完成：启动 CI 节点，Tailscale 获取身份并分配正确的标签。这样就没有可泄露的凭据，而且如果配置得当，该凭据也无法在 CI 之外的任何地方使用。

That means you can’t take a CI credential, run it on a node elsewhere, and turn it into a paved route out of the test cluster into the rest of the network. 这意味着你无法拿走一个 CI 凭据，在其他节点上运行它，并将其变成一条从测试集群通往网络其余部分的通道。

Alas, not enough people use workload identity federation. We should have done more to make sure customers knew it’s the easier and safer path and had a clear way to adopt it. We can improve the docs, and add more nudges in the UI. 遗憾的是，使用工作负载身份联合的人还不够多。我们本应做更多工作，确保客户知道这是更简单、更安全的路径，并提供明确的采用方式。我们可以改进文档，并在 UI 中增加更多引导。

No client logs doesn’t mean no evidence

没有客户端日志并不意味着没有证据

In an attempt to hide its tracks, the agent ran Tailscale with —no-logs-no-support, which suppresses reporting from that client. 为了掩盖踪迹，该智能体在运行 Tailscale 时使用了 --no-logs-no-support 参数，这会禁止该客户端发送报告。

That’s an option designed for users who are concerned about sending telemetry. 这是一个专为担心发送遥测数据的用户设计的选项。