Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness

表示形式影响检索：多模态智能体框架中技能发现与路由的案例研究

Abstract: A production agent harness must discover and rank, from a growing library of skills, the one most appropriate for a user’s task. At small scale this selection happens in context: the LLM planner chooses among skill representations exposed in its system prompt, without an explicit embedding-based retrieval step. We treat this in-context selection as the small-N counterpart to embedding-based skill retrieval at scale, and present a case study of how Tinycloud, a production multimodal video agent harness, represents its skills for the planner.

摘要： 生产级智能体框架必须从不断增长的技能库中发现并排序，以找到最适合用户任务的技能。在小规模场景下，这种选择是在上下文（in-context）中完成的：大语言模型（LLM）规划器在系统提示词所展示的技能表示中进行选择，而无需显式的基于嵌入（embedding-based）的检索步骤。我们将这种上下文选择视为大规模基于嵌入的技能检索在小样本（small-N）情况下的对应物，并以生产级多模态视频智能体框架 Tinycloud 为例，研究了它如何向规划器呈现其技能。

The harness ships skills under two recurring representations: tool-skills that wrap a single external API or system tool and serve as primitive vocabulary, and workflow-skills that orchestrate tool-skill calls plus a template render to produce one named deliverable. The harness exposes them via two surfaces in the system prompt: an inlined-body surface (full instructions, scripts, templates) for autoloaded skills, and a one-line listing for on-demand skills.

该框架通过两种常见的表示形式发布技能：一种是“工具技能”（tool-skills），它封装了单个外部 API 或系统工具，作为基础词汇；另一种是“工作流技能”（workflow-skills），它编排工具技能的调用并结合模板渲染，以生成特定的交付成果。该框架通过系统提示词中的两个界面展示这些技能：一种是用于自动加载技能的内联主体界面（包含完整的指令、脚本和模板），另一种是用于按需调用技能的单行列表。

A six-task selection ablation across three exposure regimes (all-on, default, all-off) shows that full autoload selects the gold skill on every task; all-off slows execution and produces hard discovery failures; and the production default misroutes one task because its lexical signal collides with an autoloaded tool-skill that pulls planner attention away from a listed workflow-skill.

通过在三种展示机制（全部开启、默认、全部关闭）下进行的六项任务选择消融实验表明：完全自动加载在每项任务中都能选出最优技能；全部关闭则会减慢执行速度并导致严重的发现失败；而生产环境的默认设置在处理一项任务时出现了路由错误，原因是其词汇信号与一个自动加载的工具技能发生了冲突，从而将规划器的注意力从列表中的工作流技能上引开了。

The headline finding is that in-prompt exposure of skills is not monotonically helpful: partial exposure can create lexical competition that suppresses correct selection. We connect this small-N observation to recent retrieval-based skill-routing work at large scale, and frame this contribution as a case study rather than a benchmark.

核心发现是：在提示词中展示技能并非总是越全面越好（并非单调递增的帮助）：部分展示可能会产生词汇竞争，从而抑制正确的选择。我们将这一小样本观察结果与近期大规模基于检索的技能路由工作联系起来，并将此贡献定位为案例研究，而非基准测试。