I built a low-latency AI companion that plays Skyrim with me

我构建了一个能与我一起玩《上古卷轴5：天际》的低延迟 AI 同伴

The goal is simple: Let’s build a super-charged next-level gaming companion that actually feels good. There are already multiple frameworks that let LLMs control NPC dialogue. They are fantastic for role-playing and staying in character, but they have two recurring problems: weak world agency and latency. Good at talking but far less reliable at performing actions and terrible at complex instruction sets. 目标很简单：让我们构建一个超强、进阶级的游戏同伴，让它真正带来良好的体验。目前已经有多种框架可以让大语言模型（LLM）控制 NPC 对话。它们在角色扮演和保持人设方面表现出色，但有两个反复出现的问题：世界交互能力弱和延迟高。它们擅长聊天，但在执行动作时可靠性较差，且难以处理复杂的指令集。

You may have also noticed how popular AI NPC demos often cut between the player speaking and the AI replying, trying to mask latency. Can we do better? I wanted a companion that: 你可能也注意到，流行的 AI NPC 演示通常会在玩家说话和 AI 回复之间进行剪辑，试图掩盖延迟。我们能做得更好吗？我想要一个这样的同伴：

• Useful and instant. It should fight, fetch, loot, inspect, carry and give items etc., follow complex multi-step instructions without feeling buggy or experimental. This matters especially in VR, where navigating menus is cumbersome and immersion-breaking. It needs to be FAST fast, not just fast.

• 有用且即时。 它应该能战斗、拾取、搜刮、检查、搬运和给予物品等，并能执行复杂的多步骤指令，而不会让人觉得有 Bug 或处于实验阶段。这在 VR 中尤为重要，因为在 VR 中操作菜单既繁琐又会破坏沉浸感。它需要“极速”，而不仅仅是“快”。

• Alive and present. It should have a fun, endearing personality, not canned robotic pre-written responses. Remember shared experiences and change over time. The microphone stays active while a session is running: you do not summon Varkos through a dialogue menu, you talk to him. When immersion kicks in, it should feel like you are not playing alone.

• 鲜活且真实。 它应该拥有有趣、讨人喜欢的个性，而不是死板的机器人预设回复。它能记住共同经历并随时间演变。当游戏会话运行时，麦克风始终保持开启：你不需要通过对话菜单召唤 Varkos，你直接与他交谈。当沉浸感产生时，你会感觉自己不是在独自游戏。

• Local and private wherever practical. The elephant in the room is that cloud LLM calls can get quite pricy (especially with multi-thousand-token LLM calls) and the added latency can be an experience killer. And why turn a private single-player game into a metered and surveilled experience? Let’s try to give as much control to the user as possible (bonus: it’s a fun technical challenge). Basically: a single-player game where you are not playing alone.

• 尽可能本地化且私密。 显而易见的问题是，云端 LLM 调用可能会变得非常昂贵（尤其是数千 Token 的调用），且增加的延迟会毁掉体验。为什么要将一个私人的单机游戏变成一个被计量和监控的体验呢？让我们尽量把控制权交给用户（额外的好处是，这是一个有趣的技术挑战）。简而言之：一个你不再孤单的单机游戏。

Complex commands

复杂指令

Varkos can handle commands that extend beyond one immediate action. Plans can wait for events, preserve targets between steps, monitor progress and repair or stop when world state changes. Nothing is pre-scripted. Varkos 可以处理超出单一即时动作的指令。计划可以等待事件发生、在步骤之间保留目标、监控进度，并在世界状态改变时进行修复或停止。没有任何内容是预先编写脚本的。

Let’s see some examples: 让我们看一些例子：

Varkos receives a conditional instruction involving the next arrow. He registers the future trigger instead of acting immediately, waits for the correlated projectile impact and then continues the plan. Varkos 收到了一条涉及下一支箭的条件指令。他没有立即行动，而是注册了未来的触发器，等待相关的弹道命中，然后继续执行计划。

Long-form multi-step command: “I want you to wait here and I’m gonna go over there. Once you see the signal, the signal is going to be an arrow I fire up in the sky, I want you to pick up this potion and come and bring it to me. Okay?” 长篇多步骤指令： “我要你在这里等，我去那边。一旦你看到信号——信号就是我向天空射出的一支箭——我就要你去捡起这瓶药水，然后带给我。好吗？”

(A deferred command follows a real projectile event in Skyrim.) （在《天际》中，一个延迟指令跟随真实的弹道事件执行。）

Item search

物品搜索

Varkos can search the grounded world state for a requested item, identify where it is and respond using what is actually present in the game. Varkos 可以在真实的游戏世界状态中搜索请求的物品，识别其位置，并根据游戏中实际存在的内容进行回应。

“Do you see the ceremonial sword anywhere?” Varkos picks up a different sword and brings it to us. We tell him that’s not the one, then he offers to be on the lookout. “你看到仪式之剑了吗？”Varkos 捡起了一把不同的剑带给我们。我们告诉他那不是我们要找的，然后他主动提出会留意。

(Finding an item through game state rather than inventing an answer.) （通过游戏状态寻找物品，而不是编造答案。）

Hide-and-seek

捉迷藏

Hide-and-seek is not a single API call. It becomes a persistent goal with movement, waiting, monitoring and completion conditions. 捉迷藏不是一个单一的 API 调用。它变成了一个包含移动、等待、监控和完成条件的持久目标。

“Let’s play hide-and-seek again. You wait here and I’m gonna go hide, then count to ten and come and try to find me.” “我们再玩一次捉迷藏吧。你在这里等，我去躲起来，然后数到十，再来找我。”

(A game represented as a persistent plan rather than a line of dialogue.) （一个被表现为持久计划而非单行对话的游戏。）

Loot this chest and give me the potion

搜刮这个箱子并把药水给我

This combines a grounded container, a filtered loot step and an inventory transfer. Each physical result advances the next part of the plan. 这结合了具体的容器、过滤后的搜刮步骤和物品栏转移。每一个物理结果都会推进计划的下一部分。

(Loot, select and transfer while preserving the requested item.) （在保留所请求物品的同时进行搜刮、选择和转移。）

Pick up all the items

捡起所有物品

“Pick up all the items and give them to me” becomes a bounded collection plan over real references. Varkos gathers them, returns and transfers them without pretending that one magical action means “all.” “捡起所有物品并给我”变成了一个基于真实引用的有限收集计划。Varkos 收集它们、返回并转移，而不是假装一个魔法动作就代表了“所有”。

(A collection plan operating on grounded world objects.) （一个在具体世界对象上操作的收集计划。）

Combat

战斗

Varkos receives grounded events from the game, can warn the player through a fast reflex path and uses native body control to act. Instruction plans can strategize (e.g. attack this, then retreat, etc.), and his emotional state can affect how and if he chooses to fight. Varkos 从游戏中接收具体事件，可以通过快速反射路径警告玩家，并使用原生的身体控制来行动。指令计划可以进行策略规划（例如：攻击这个，然后撤退等），他的情绪状态会影响他如何选择战斗以及是否选择战斗。

(Combat footage 1: dungeon combat. Combat footage 2: perception, warning and physical action on the latency-sensitive path.) （战斗片段 1：地牢战斗。战斗片段 2：在延迟敏感路径上的感知、警告和物理动作。）

Personality evolution

个性演变

Varkos is fully customizable. He does not have to be a demon dog, and the runtime does not have to control only a single character. What systems are applied and what they do, is up to open configuration. Varkos 是完全可定制的。他不一定非要是只恶魔狗，运行时也不一定只能控制一个角色。应用什么系统以及它们做什么，取决于开放配置。

One part of my current build still fully depends on big model/cloud LLM calls: slow personality evolution. This work happens away from the real-time action path. As the player and Varkos travel together, important interactions become evidence for gradual changes to his personality. 我当前构建的一部分仍然完全依赖于大模型/云端 LLM 调用：缓慢的个性演变。这项工作发生在实时动作路径之外。随着玩家和 Varkos 一起旅行，重要的互动成为了他个性逐渐改变的证据。

My demo Varkos begins as a demon reincarnated as a dog. He considers his canine instincts humiliating, his dog body a prison, and is mistrustful, proud and sarcastic. Through shared experiences he can become more and more domesticated, grow attached to the player and starts enjoying being a dog. Eventually he starts bringing over toys because he wants to play, running off to chase things and seeking affirmation from the player. 我的演示版 Varkos 最初是一个转世为狗的恶魔。他认为自己的犬类本能是羞辱，狗的身体是监狱，他多疑、骄傲且尖酸刻薄。通过共同经历，他可以变得越来越温顺，对玩家产生依恋，并开始享受做一只狗。最终，他开始叼来玩具因为他想玩耍，跑去追逐事物，并寻求玩家的肯定。

Only the starting character traits are authored. The system changes both his explicit traits and his emotional homeostasis. How easily he becomes irritated, frightened, affectionate or playful, etc. He can overwrite parts of his vocabulary and code. Changes are versioned and reversible. I could make it more bounded, but I think there’s something fun about some open world clankiness, so how he evolves is up in the air. 只有初始的角色特征是预先设定的。该系统会改变他的显性特征和情绪稳态。他变得易怒、恐惧、深情或顽皮的程度等等。他可以覆盖部分词汇和代码。更改是版本化且可逆的。我可以让它更受限，但我认为开放世界中的一些“笨拙感”很有趣，所以他如何演变仍有不确定性。

(The demon slowly discovers that being a pup is not a bad life. And he has learnt to love cabbage…) （恶魔慢慢发现做一只小狗生活也不坏。而且他学会了爱吃卷心菜……）