Vision-in-the-Loop: When the AI Rewrites Its Own Prompts from the Generated Frame

Vision-in-the-Loop：当 AI 根据生成的画面重写自己的提示词时

On the AI video ad platform I work on, every scene goes through the same painful loop: write a prompt, send it to an AI video model provider, wait two minutes, open the result, squint at the frame, and decide what went wrong. Camera too wide. Product missing from the hero shot. Color palette drifted warm when the brand brief says cool neutrals. Avatar looks like a different person than scene three. That loop was manual, slow, and expensive. Each regeneration burns GPU credits. Operators were becoming prompt engineers by accident — and still missing subtle failures until stitch time, when fixing scene four means re-rendering everything downstream.

在我工作的 AI 视频广告平台上，每一个场景都要经历同样痛苦的循环：编写提示词、发送给 AI 视频模型供应商、等待两分钟、打开结果、眯着眼查看画面，然后判断哪里出了问题。镜头太广了？产品在主镜头中消失了？品牌简报要求冷色调中性风，但色调却偏暖了？虚拟形象看起来和第三个场景里的人不是同一个？这个循环是手动的、缓慢的且昂贵的。每一次重新生成都会消耗 GPU 算力。操作员在无意中变成了提示词工程师，却依然会在剪辑阶段才发现细微的错误，而此时修复第四个场景意味着后续所有内容都要重新渲染。

The insight behind vision-in-the-loop prompt authoring is simple: the model that wrote the prompt can also look at its own output and rewrite the prompt with surgical fixes. Not a full replan — a per-scene correction grounded in the actual generated frame, not the operator’s memory of what they hoped would appear.

“Vision-in-the-loop”（视觉闭环）提示词创作背后的洞察很简单：编写提示词的模型同样可以审视自己的输出，并进行精准的修复重写。这并非完全重新规划，而是基于实际生成的画面进行逐场景修正，而不是依赖操作员对“预期画面”的记忆。

The manual loop we were trying to kill

我们试图终结的手动循环

Before this work shipped, the swipe iteration flow looked like this: 在这一功能上线之前，迭代流程是这样的：

• Plan — Claude generates a scene-by-scene script with visual prompts

• Generate — each scene renders independently through an AI video model provider

• Review — operator opens the portal, compares frames to the reference ad

• Rewrite — operator edits prompts in a text field, often guessing at what the model misread

• Regenerate — repeat until acceptable or budget exhausted

• 规划 — Claude 生成带有视觉提示词的逐场景脚本

• 生成 — 每个场景通过 AI 视频模型供应商独立渲染

• 审查 — 操作员打开门户网站，将画面与参考广告进行对比

• 重写 — 操作员在文本框中编辑提示词，通常是在猜测模型哪里读错了

• 重新生成 — 重复上述步骤，直到结果满意或预算耗尽

Steps three and four are where throughput dies. An experienced operator can spot “product not visible” in three seconds, but translating that into prompt language — “medium close-up, product centered in lower third, shallow depth of field” — takes another minute per scene. Multiply by twelve scenes and three swipe iterations, and a single ad creative consumes an hour of human attention that should be spent on brand strategy, not frame inspection. The generated frame is ground truth. The original prompt is a hypothesis. Vision-in-the-loop closes the gap between them automatically.

第三步和第四步是效率的杀手。一位经验丰富的操作员可以在三秒钟内发现“产品不可见”，但将其转化为提示词语言——“中景特写，产品位于下三分之一处居中，浅景深”——每个场景又需要一分钟。乘以十二个场景和三次迭代，一个广告创意就会消耗一小时的人力，而这些时间本应花在品牌策略上，而不是检查画面。生成的画面是“地面实况”（Ground Truth），而原始提示词只是一个假设。Vision-in-the-loop 自动弥合了两者之间的差距。

Vision-in-the-loop: iteration 3 architecture

Vision-in-the-loop：第三版迭代架构

The third swipe iteration introduced a per-scene feedback loop that runs immediately after the first frame of each scene is generated — before the operator ever opens the portal.

第三次迭代引入了一个逐场景反馈循环，它在每个场景的第一帧生成后立即运行——在操作员打开门户网站之前。

Stage	Input	Output
Frame capture	Generated clip (first keyframe extracted)	PNG at native resolution
Vision critique	Frame + original prompt + scene intent + brand constraints	Structured issue list (severity, category, evidence)
Prompt rewrite	Original prompt + issue list + reference frame (optional)	Revised prompt with targeted deltas only
Regenerate	Revised prompt + anchored persona image	New clip (single retry by default)

阶段	输入	输出
画面捕捉	生成的视频片段（提取第一关键帧）	原生分辨率的 PNG 图片
视觉评估	画面 + 原始提示词 + 场景意图 + 品牌约束	结构化问题列表（严重程度、类别、证据）
提示词重写	原始提示词 + 问题列表 + 参考帧（可选）	仅包含针对性修改的修订版提示词
重新生成	修订版提示词 + 固定的人物形象	新的视频片段（默认单次重试）

The vision model receives the generated frame alongside the scene’s intent metadata — shot type, product placement rules, palette constraints, and whether this is a person scene or product B-roll. It returns a structured critique, not free-form prose. That structure matters: downstream rewrite logic keys off issue categories like framing_too_wide, product_absent, palette_drift, and identity_mismatch.

视觉模型接收生成的画面以及场景的意图元数据——包括镜头类型、产品摆放规则、色调约束，以及这是人物场景还是产品 B-roll（辅助镜头）。它返回的是结构化的评估，而不是自由格式的文字。这种结构至关重要：下游的重写逻辑会根据问题类别（如 framing_too_wide 构图过广、product_absent 产品缺失、palette_drift 色调偏移、identity_mismatch 身份不符）进行处理。

(Code snippet omitted for brevity)

The rewrite step is constrained: it may not change locked elements (avatar reference, product SKU imagery, mandated taglines). It only adjusts the visual prompt fields that the vision critique flagged. This prevents the common failure mode where an aggressive rewrite “fixes” the framing but drops the brand voice or swaps the protagonist.

重写步骤受到约束：它不能更改已锁定的元素（如虚拟形象参考、产品 SKU 图像、强制要求的标语）。它只会调整视觉评估中标记出的提示词字段。这避免了常见的失败模式，即激进的重写虽然“修复”了构图，却丢失了品牌调性或更换了主角。

What the vision model actually catches

视觉模型实际捕捉到了什么

In production testing across roughly 200 scene generations, the vision critique surfaced issues that text-only plan QA missed entirely: 在约 200 个场景生成的生产测试中，视觉评估发现了纯文本规划 QA 完全忽略的问题：

• Framing drift — wide establishing shot when the plan called for medium close-up on the spokesperson

• Product absence — hero product not visible despite explicit product-lock instructions in the prompt

• Palette mismatch — warm golden-hour tones on a brand that specifies cool clinical whites

• Background clutter — competing visual elements that dilute the focal subject

• Motion mismatch — static hold when the reference scene had subtle camera push-in

• 构图偏移 — 计划要求对发言人进行中景特写，结果却生成了广角定场镜头

• 产品缺失 — 尽管提示词中有明确的产品锁定指令，但主打产品却不可见

• 色调不匹配 — 品牌要求冷色调的临床白，结果却生成了温暖的黄金时刻色调

• 背景杂乱 — 竞争性的视觉元素稀释了焦点主体

• 运动不匹配 — 参考场景有细微的镜头推近，结果却生成了静态画面

Each issue maps to a prompt delta. “Framing too wide” becomes tighter focal length language and explicit subject placement. “Product absent” triggers a product-lock reinforcement clause and shot-type downgrade to guarantee visibility. The rewrite is incremental — we append and refine, not replace wholesale.

每个问题都对应一个提示词的增量修改。“构图过广”会转化为更紧凑的焦距描述和明确的主体位置。“产品缺失”会触发产品锁定强化条款和镜头类型降级，以确保可见性。重写是增量式的——我们进行追加和精炼，而不是全盘替换。

Motion-transfer defaults for person scenes

人物场景的默认动作迁移

Vision-in-the-loop solves prompt accuracy. A parallel iteration — swipe iteration 2 — solved a different problem: person scenes that felt dead on arrival. Generic text-to-video generation treats every scene the same. For product B-roll, that works — a slow pan across a bottle on marble is fine with standard generation. For person scenes, the reference ad almost always has subtle body motion, micro-expressions, and natural idle movement that pure text-to-video renders as uncanny mannequin holds. The fix: make motion-transfer the default generation mode for any scene classified as a person scene. Motion-transfer takes a reference clip (from the source ad or a prior good take) and t…

Vision-in-the-loop 解决了提示词准确性的问题。另一个并行迭代——第二次迭代——解决了另一个问题：看起来毫无生气的人物场景。通用的文生视频模型对待每个场景的方式都是一样的。对于产品 B-roll 来说，这没问题——在标准生成下，缓慢扫过大理石上的瓶子效果很好。但对于人物场景，参考广告几乎总是有细微的身体动作、微表情和自然的闲置动作，而纯文生视频往往将其渲染成诡异的模特定格。解决方案是：将“动作迁移”设为所有被归类为人物场景的默认生成模式。动作迁移会获取一个参考片段（来自源广告或之前效果良好的拍摄），并……