Syll: Open-Source Personal Automation with Cross-Surface Execution

Syll：支持跨界面执行的开源个人自动化系统

Personal AI agents must increasingly operate across APIs, shells, web surfaces, and desktop GUIs, yet many systems remain tuned to a single interface and offer limited support for user teaching and auditability.

个人 AI 智能体日益需要跨越 API、Shell 命令行、网页界面以及桌面图形用户界面（GUI）进行操作，然而许多现有系统仍局限于单一界面，且在用户教学和可审计性方面支持有限。

We present Syll, an open-source, self-hosted multimodal agent harness that unifies MCP/API tools, CLI execution, and visual GUI control in a modular runtime, enabling agents to coordinate computer use across heterogeneous interfaces while streamlining how users and agents exchange information.

我们推出了 Syll，这是一个开源、自托管的多模态智能体框架。它在一个模块化运行时中统一了 MCP/API 工具、CLI 执行和视觉 GUI 控制，使智能体能够跨异构界面协调计算机操作，同时简化了用户与智能体之间的信息交互流程。

At the core of Syll is a bidirectional user-agent interaction layer: users teach procedures through direct demonstration, which Syll compiles into reusable skills; agent execution is translated back into multimodal evidence — logs, keyframes, and approval checkpoints — for inspection and control.

Syll 的核心是一个双向的用户-智能体交互层：用户通过直接演示来教授操作流程，Syll 将其编译为可复用的技能；智能体的执行过程则被转化为多模态证据（包括日志、关键帧和审批检查点），以便用户进行检查和控制。

Syll further externalizes memory, skills, routines, and governance as editable local artifacts, supporting straightforward inspection, extension, and downstream development.

此外，Syll 将记忆、技能、例程和治理规则外部化为可编辑的本地工件（Artifacts），从而支持直接的检查、扩展和下游开发。

Our implementation has been validated on production desktop applications including Adobe Photoshop, Adobe Audition, Stardew Valley, macOS Finder and others. We report mechanism-oriented studies that validate multimodal routing, teachable GUI replay, and persistent local artifacts.

我们的实现已在 Adobe Photoshop、Adobe Audition、星露谷物语（Stardew Valley）、macOS Finder 等生产级桌面应用上得到了验证。我们通过面向机制的研究，验证了多模态路由、可教学的 GUI 重放以及持久化本地工件的有效性。

We hope Syll can serve as a practical open-source foundation for personal automation that users can teach, inspect, and continuously extend.

我们希望 Syll 能成为个人自动化的实用开源基石，让用户能够对其进行教学、检查并持续扩展。