I replaced the chat window for my local AI agent with a face

我把本地 AI 代理的聊天窗口换成了一张“脸”

I run a local LLM agent (Hermes) on my own machine. The problem was never the model — it was the interface. I had a Telegram tab open all day just to talk to it: type a command, wait, read a wall of text back, scroll. It felt like texting a very capable stranger. So I built Ghost Vessel — a monitor-resident, video-call-style avatar that fronts the agent. The name is the whole idea: the ghost is your agent, the vessel is the body it borrows. It’s not a waifu toy; it’s a real agent client that happens to have a face. Here’s what actually turned out to be interesting to build.

我在自己的机器上运行着一个本地 LLM 代理（Hermes）。问题从来不在模型本身，而在于交互界面。我整天开着一个 Telegram 标签页只为了和它对话：输入指令、等待、阅读大段文字、滚动屏幕。这感觉就像是在给一个能力超强的陌生人发短信。于是我构建了 Ghost Vessel——一个常驻屏幕、类似视频通话风格的虚拟形象，作为代理的前端。名字本身就概括了核心理念：Ghost（幽灵）是你的代理，Vessel（容器）是它借用的躯壳。它不是一个“二次元老婆”玩具，而是一个恰好长了张脸的真实代理客户端。以下是我在构建过程中发现的有趣之处。

The reply is a script, not a string

回复是脚本，而非字符串

The core idea is an output contract. Instead of treating the agent’s reply as text to print, I split every reply into three planes:

• dialogue → spoken via local TTS data
• code, logs, files → rendered as chat cards, never read aloud
• action → emotion beats that drive the avatar

核心理念是“输出契约”。我没有把代理的回复仅仅当作要打印的文本，而是将每一条回复拆分为三个层面：

• 对话 → 通过本地 TTS 数据朗读
• 代码、日志、文件 → 渲染为聊天卡片，不会被朗读出来
• 动作 → 驱动虚拟形象的“情感节拍”

Emotion beats are inline tags the model emits in-band with its answer:

• [working] — the avatar puts on glasses and takes notes while a task runs
• [confirm] deploy to prod? — pops a human-in-the-loop approve/cancel, and the agent blocks on your keypress
• [happy] / [concerned] / … — fine-grained facial expressions

“情感节拍”是模型在回答中嵌入的行内标签：

• [working] — 当任务运行时，虚拟形象会戴上眼镜并做笔记
• [confirm] 部署到生产环境吗？ — 弹出人工确认/取消窗口，代理会等待你的按键操作
• [happy] / [concerned] / … — 精细的面部表情

So “run the build, and if it passes, deploy” becomes a little performance: it looks busy while working, shows you the log as a card, then leans in and asks before the irreversible step. The text you’d have skim-read becomes something you glance at.

因此，“运行构建，如果通过则部署”变成了一场小小的表演：它在工作时看起来很忙碌，以卡片形式向你展示日志，然后在执行不可逆步骤前凑近询问。原本需要你快速扫视的文字，现在变成了只需瞥一眼就能理解的内容。

No runtime GPU for the avatar

虚拟形象无需运行时 GPU

The obvious way to animate a face is live inference. I didn’t want that — the GPU is busy running the actual model. Instead the avatar is ~30 pre-rendered clips, and the emotion beats just select and blend between them (blink-aligned seamless idle loops, a head-pose “settle gate” so an expression only reveals when the head is frontal). The avatar’s runtime cost is basically video playback. Your GPU stays 100% on your LLM.

让脸部动起来最直观的方法是实时推理。但我不想这样做——GPU 还要忙着运行模型本身。相反，虚拟形象由约 30 个预渲染的剪辑组成，情感节拍只需在这些剪辑之间进行选择和混合（例如对齐眨眼的无缝待机循环，以及一个头部姿态的“稳定门”，确保表情只在头部正对时显示）。虚拟形象的运行时开销基本上就是视频播放。你的 GPU 可以 100% 专注于 LLM。

The tradeoff: no real-time lip-sync. I decided a believable talking mouth loop + expressive face reads as “a person on a call” far more than perfect phoneme-matching does, and it costs nothing at runtime. That single decision collapsed most of the hard engineering.

权衡之处在于：没有实时唇形同步。我决定，一个可信的说话嘴型循环加上富有表现力的脸，比完美的音素匹配更能让人感觉像是在“视频通话”，而且运行时几乎零开销。这一个决定省去了大部分复杂的工程工作。

Hooking a real agent without the agent knowing

在代理不知情的情况下接入

The agent integration is the part I was most unsure would work. It turned out clean: the app registers as a connector. The agent’s gateway dials out to a local WebSocket the app hosts, and exchanges frames — so from the agent’s side, the avatar is just “another channel,” indistinguishable from Telegram.

代理集成是我最不确定能否成功的部分。结果证明非常简洁：应用注册为一个连接器。代理的网关拨入应用托管的本地 WebSocket 并交换帧——因此从代理的角度来看，这个虚拟形象只是“另一个频道”，与 Telegram 别无二致。

Inbound (you → agent): your typed text as a message event Outbound (agent → you): the reply, which the connector parses for emotion beats Adapters for Hermes and OpenClaw are included, plus a demo responder so it runs with zero setup. And the chat pane serves the agent’s live slash-command menu — type / and you get the same 52 commands you’d see in the messenger, passed straight through.

入站（你 → 代理）：你输入的文本作为消息事件 出站（代理 → 你）：回复内容，连接器会解析其中的情感节拍 内置了 Hermes 和 OpenClaw 的适配器，外加一个演示响应器，实现零配置运行。聊天面板还提供了代理的实时斜杠命令菜单——输入 / 即可获得你在即时通讯软件中看到的 52 个命令，直接透传。

The bits that make it feel alive

让它栩栩如生的细节

• Mood & affinity. A short-term mood decays toward a long-term relationship baseline. Scold it repeatedly and it settles into a subdued idle; praise it and it brightens. Persists across restarts.

• Voice both ways. Edge TTS out (swappable to local Qwen3-TTS / MeloTTS / Piper), Silero VAD + faster-whisper in.

• Monitor-resident. Two frameless, always-on-top windows (avatar + chat) you drag anywhere, built with Tauri v2 (Rust shell, Python servers, JS player).

• 情绪与亲密度。 短期情绪会向长期关系基准衰减。反复责骂它，它会进入低落的待机状态；赞美它，它会变得开朗。状态在重启后依然保留。

• 双向语音。 输出使用 Edge TTS（可切换为本地 Qwen3-TTS / MeloTTS / Piper），输入使用 Silero VAD + faster-whisper。

• 常驻屏幕。 两个无边框、始终置顶的窗口（虚拟形象 + 聊天），可拖动到任何位置，基于 Tauri v2 构建（Rust 外壳，Python 服务器，JS 播放器）。

Open-core

开放核心

The engine is MIT and usable on its own: clone it, drop a folder of clips named by emotion (happy.mp4, working.mp4, idle.mp4, …), point it at your agent. Avatars are pure-data bundles — no code runs when you install one — so you can build your own; the full reproducible method is in the repo.

引擎采用 MIT 协议，可独立使用：克隆它，放入一个按情绪命名的剪辑文件夹（happy.mp4, working.mp4, idle.mp4 等），然后指向你的代理即可。虚拟形象是纯数据包——安装时不会运行任何代码——所以你可以构建自己的形象；完整的复现方法都在仓库里。

Repo + 45s demo: https://github.com/ghdtjrtka/ghost-vessel Windows build: https://ghostvessel.space

What I’m curious about

我好奇的地方

The part I’d most like feedback on is the emotion-beat output format — the contract that turns LLM text into a UI performance. Has anyone else built output contracts to drive an interface from model output? What broke, and what did models reliably get right vs. wrong? I’d genuinely like to compare notes.

我最想获得反馈的部分是“情感节拍”输出格式——即那种将 LLM 文本转化为 UI 表演的契约。还有其他人构建过通过模型输出驱动界面的契约吗？哪些地方出过错，模型在哪些方面表现稳定，哪些方面又容易出错？我真心希望能交流一下心得。