vercel-labs / agent-desktop
本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。
vercel-labs / agent-desktop
agent-desktop Computer use for AI agents: read and operate desktop apps through the OS accessibility tree. A CLI and C library for macOS, Windows and Linux. A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent. Every CLI call runs in its own process. Headless calls still prohibit foreground activation and synthesized input. Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider; it is separate from the native CLI. See privacy and data handling.
agent-desktop 是一款供 AI 智能体使用的计算机操作工具：通过操作系统辅助功能树（accessibility tree）读取并操作桌面应用程序。它为 macOS、Windows 和 Linux 提供了命令行界面（CLI）和 C 语言库。快照（snapshot）会返回一个 JSON，其中包含智能体可操作的每个元素的引用（ref）。智能体对引用执行操作后再次进行快照以进行验证，无需像素匹配；循环逻辑保留在智能体内部。每个 CLI 调用都在其独立的进程中运行。无头（headless）调用仍禁止前台激活和合成输入。快照、剪贴板读取、屏幕截图、命令输出和调试工件可能包含私有的桌面内容，请勿将其放入公共日志或上传。可选的 JEV 助手会将屏幕描述发送给配置的模型提供商；它与原生 CLI 是分开的。请参阅隐私和数据处理相关说明。
Installation Install on macOS, Windows or Linux with npm: npm install -g agent-desktop agent-desktop version Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable. Official Homebrew and Scoop packages are being submitted; they are not yet available in the default catalogs. The repository no longer maintains a custom Homebrew tap or Scoop bucket. macOS needs Accessibility permission: agent-desktop permissions —guide. Windows needs no grant for apps at your terminal’s integrity level. Linux needs the user’s graphical session with at-spi2-core. See installation and permissions for package-manager behavior, release archives, and building from source.
安装：通过 npm 在 macOS、Windows 或 Linux 上安装：npm install -g agent-desktop，随后运行 agent-desktop version。Bun、Yarn 和 pnpm 均可安装此包。它包含适用于 x64 和 ARM64 架构的原生二进制文件及平台助手。若需在没有 JavaScript 启动器的情况下直接执行，请从 GitHub Releases 下载对应的原生压缩包，并将助手文件保留在可执行文件旁边。官方 Homebrew 和 Scoop 包正在提交中，目前尚未在默认目录中提供。该仓库不再维护自定义的 Homebrew tap 或 Scoop bucket。macOS 需要辅助功能权限：运行 agent-desktop permissions --guide。Windows 在终端完整性级别下无需额外授权。Linux 需要用户的图形会话并安装 at-spi2-core。有关包管理器行为、发布存档和从源码构建的详细信息，请参阅安装与权限说明。
Quick start Observe, act on a ref, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example. Each command prints one JSON envelope; a trimmed snapshot follows the commands. agent-desktop snapshot —app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot —app Finder -i # observe again to verify { “version”: “2.6”, “ok”: true, “command”: “snapshot”, “data”: { “app”: “Finder”, “window”: { “id”: “w-204749”, “title”: “Downloads” }, “snapshot_id”: “s8f3k2p9”, “ref_count”: 59, “complete”: true, “tree”: { “role”: “window”, “name”: “Downloads”, “children”: [ { “role”: “treeitem”, “name”: “Documents”, “ref_id”: “@s8f3k2p9:e8”, “available_actions”: [“Click”] } ] } } } Refs belong to their snapshot. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET: take a new snapshot and use its refs. Several windows give AMBIGUOUS_TARGET; snapshot lists the candidate window ids in error.details. Pass one with —window-id. Dense apps: map first with snapshot —skeleton —app Slack -i —compact, then drill into one region with snapshot —root @s8f3k2p9:e3 -i —compact. Headless by default. Ref actions go through the accessibility API without moving the pointer or taking focus. The global —headed flag allows physical input. PowerShell. Quote every ref (‘@s8f3k2p9:e8’), because a bare @token is the splatting operator.
快速入门：观察、对引用执行操作、再次观察。可以使用 agent-desktop list-apps 列出的任何应用名称；Finder 是 macOS 的示例。每个命令打印一个 JSON 信封；精简后的快照紧随命令之后。引用属于其对应的快照。直接使用 @e8 会被拒绝并返回 INVALID_ARGS。当 UI 发生变化时，操作会返回 STALE_REF 或 AMBIGUOUS_TARGET：此时需获取新快照并使用其中的引用。多个窗口会导致 AMBIGUOUS_TARGET；快照会在 error.details 中列出候选窗口 ID，可通过 --window-id 指定一个。对于密集型应用：先使用 snapshot --skeleton --app Slack -i --compact 进行映射，然后使用 snapshot --root @s8f3k2p9:e3 -i --compact 深入特定区域。默认情况下为无头模式。引用操作通过辅助功能 API 执行，不会移动指针或获取焦点。全局 --headed 标志允许物理输入。在 PowerShell 中，请引用每个引用（如 '@s8f3k2p9:e8'），因为裸露的 @token 是展开运算符。
For AI agents The binary carries its own docs, matched to its version: agent-desktop skills get desktop # the loop, refs, verification and retry rules agent-desktop skills get platform # what differs on the OS you run on agent-desktop agent-desktop to see and operate desktop apps. Before the first use, run agent-desktop skills get desktop and agent-desktop skills get platform, and follow them. Every command prints one envelope with ok and either data or error. An error carries a code, a disposition and usually a suggestion; retry only when error.disposition.retry is safe. The contract, error codes and exit codes are in docs/json-output.md.
面向 AI 智能体：二进制文件自带与其版本匹配的文档：agent-desktop skills get desktop（循环、引用、验证和重试规则）、agent-desktop skills get platform（当前操作系统下的差异）以及 agent-desktop <command> --help（单个命令的标志）。通过 skills.sh 将技能安装到智能体框架中：npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop。或者将其添加到 AGENTS.md 中：使用 agent-desktop 查看和操作桌面应用。首次使用前，请运行 agent-desktop skills get desktop 和 agent-desktop skills get platform 并遵循说明。每个命令打印一个包含 ok 以及 data 或 error 的信封。错误包含代码、处置方式和建议；仅当 error.disposition.retry 为安全时才进行重试。契约、错误代码和退出代码详见 docs/json-output.md。
Commands It ships 60 command names, 56 operational commands. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button. Group Commands Observe snapshot, find, get, is, screenshot, list-surfaces Act on a ref click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to Keyboard and mouse press, hover, drag, mouse-move, mouse-click, mouse-wheel, key-down, key-up, mouse-down, mouse-up Apps and windows launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface Notifications list-notifications, dismiss-notification, dismiss-all-notifications, notification-action Clipboard clipboard-get, clipboard-set, clipboard-clear Wait wait Sessions session, trace, cursor-overlay System status, permissions, version, batch, skills
命令：该工具提供 60 个命令名称，其中 56 个为操作命令。其余四个（key-down, key-up, mouse-down, mouse-up）在守护进程能够释放按住的键或按钮之前拒绝运行。命令分组包括：观察（snapshot, find, get, is, screenshot, list-surfaces）、对引用操作（click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to）、键盘和鼠标（press, hover, drag, mouse-move, mouse-click, mouse-wheel, key-down, key-up, mouse-down, mouse-up）、应用和窗口（launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface）、通知（list-notifications, dismiss-notification, dismiss-all-notifications, notification-action）、剪贴板（clipboard-get, clipboard-set, clipboard-clear）、等待（wait）、会话（session, trace, cursor-overlay）以及系统（status, permissions, version, batch, skills）。
Platform support macOS Windows Linux Accessibility tree Yes Yes Yes (AT-SPI) Click / type / keyboard Yes Yes AT-SPI; X11 / GNOME keys Mouse input Yes Yes X11, —headed Screenshot Yes Yes X11 windows; Wayland portal Clipboard Yes Yes Yes; Wayland needs wl-clipboard App & window management Yes Yes Launch/list; X11 window control Notifications Yes Yes GNOME Shell Shell surfaces (open-system-surface) Dock, menu bar extras Yes GNOME tray, Action Center The Windows surface and launch notes and the Linux X11 and Wayland notes are in docs/install.md. agent-desktop skills get platform prints the full capability table for the OS you run on.
平台支持：macOS、Windows 和 Linux 在辅助功能树、点击/输入/键盘、鼠标输入、屏幕截图、剪贴板、应用与窗口管理、通知及 Shell 界面方面均有不同程度的支持。Windows 界面与启动说明以及 Linux X11 和 Wayland 的说明位于 docs/install.md 中。运行 agent-desktop skills get platform 可打印当前操作系统下的完整功能表。
Language bindings Each release also ships libagent_desktop_ffi for macOS, Linux and Windows, so a program can call the tool in process instead of starting the CLI for each command. agent-desktop skills get ffi covers the entry points, memory ownership and threading rules. import ctypes lib = ctypes.CDLL(”./lib/libagent_desktop_ffi.dylib”) lib.ad_adapter_create.restype = ctypes.c_void_p lib.ad_adapter_destroy
语言绑定：每个版本还附带适用于 macOS、Linux 和 Windows 的 libagent_desktop_ffi，因此程序可以在进程内调用该工具，而无需为每个命令启动 CLI。agent-desktop skills get ffi 涵盖了入口点、内存所有权和线程规则。示例代码展示了如何使用 ctypes 加载库并调用相关函数。