AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery

AnovaX：一个具备大模型规划、类型化执行器与自适应恢复功能的本地多智能体语音助手

Abstract: Desktop voice assistants are still dominated by cloud pipelines that ship raw audio off the machine and expose a fixed set of skills. We describe AnovaX, a small local-first assistant that runs entirely on the user’s computer and treats the desktop itself as its action surface.

摘要： 桌面语音助手目前仍由云端流水线主导，这些流水线将原始音频传输至云端，并仅提供固定的技能集。我们介绍了 AnovaX，这是一个小型、以本地优先的助手，完全在用户计算机上运行，并将桌面本身视为其操作界面。

A single Python process wires together a wake-word gate, a speech pipeline, an LLM planner (Gemini) that emits a JSON plan of tool calls, a whitelist-and-denylist safety layer, a multi-agent orchestrator that translates each plan into typed child agents on a bounded thread pool, and an adaptive recovery loop that takes over whenever a core step fails.

一个单一的 Python 进程将以下组件串联在一起：唤醒词检测、语音处理流水线、负责生成工具调用 JSON 计划的大模型规划器（Gemini）、白名单与黑名单安全层、将每个计划转化为受限线程池中类型化子智能体的多智能体编排器，以及在核心步骤失败时接管任务的自适应恢复循环。

Every tool corresponds to a specialized agent class (AppAgent, TypingAgent, BrowserAgent and six others) with its own timeout, retry policy, and shared-resource locks. A recursive MetaAgent lets the planner delegate a sub-goal back to itself, capped at two levels of nesting. The recovery loop uses a compact ReAct-style prompt and hides Gemini’s latency behind speculative execution of read-only tools.

每个工具对应一个专门的智能体类（如 AppAgent、TypingAgent、BrowserAgent 等共八个），并拥有各自的超时设置、重试策略和共享资源锁。递归的 MetaAgent 允许规划器将子目标委派回自身，嵌套深度上限为两层。恢复循环使用紧凑的 ReAct 风格提示词，并通过对只读工具的推测性执行来隐藏 Gemini 的延迟。

A companion Flask server exposes a phone-friendly remote over the local WiFi, mirrors every agent lifecycle event to the phone in real time, and streams the laptop’s screen back over MJPEG so the user can watch remote commands land as they run.

配套的 Flask 服务器通过本地 WiFi 提供手机友好的远程控制界面，实时将每个智能体的生命周期事件镜像到手机上，并通过 MJPEG 流式传输笔记本电脑屏幕，以便用户在远程命令执行时实时观察其运行情况。

The point of the project is less to compete with Siri or Alexa than to show that a legible, few-thousand-line assistant is enough to open apps, type into them, run searches, coordinate concurrent actions, recover from single-step failures, and be driven entirely from a phone in another room — without the LLM ever touching the keyboard.

该项目的目的并非与 Siri 或 Alexa 竞争，而是为了证明一个代码清晰、仅数千行的助手就足以实现打开应用、输入文字、运行搜索、协调并发操作、从单步故障中恢复，并完全通过另一个房间的手机进行驱动——且无需大模型直接接触键盘。