Action Jets: Learning the Local Shape of the Action-conditioned Data Manifold

Computer Science > Computer Vision and Pattern Recognition arXiv:2610.19007 (cs) [Submitted on 13 Oct 2026] Title: Action Jets: Learning the Local Shape of the Action-conditioned Data Manifold Authors: Anton Arapin, Chahyon Ku, Bernadette Bucher.

计算机科学 > 计算机视觉与模式识别 arXiv:2610.19007 (cs) [提交于 2026 年 10 月 13 日] 标题：Action Jets：学习动作条件数据流形的局部形状 作者：Anton Arapin, Chahyon Ku, Bernadette Bucher。

Abstract: World models are a powerful means of reasoning and planning, but many of these action-conditioned predictive models are high dimensional and computationally expensive. These challenges limit the number of action candidates that can be evaluated by a planner. We show that this cost is avoidable for planners over short horizons, which only require local information with respect to the current state.

摘要：世界模型是推理和规划的有力手段，但许多此类动作条件预测模型维度高且计算成本昂贵。这些挑战限制了规划器可以评估的动作候选数量。我们证明，对于仅需要相对于当前状态的局部信息的短视界规划器而言，这种成本是可以避免的。

For a fixed state, the frames reachable by varying a low-dimensional action (a camera movement or an end-effector command) lie on a low-dimensional manifold. We estimate the local shape of that manifold directly via a network trained to estimate how each image region moves per unit of each action axis supervised by data collected from samples by the generator.

对于固定的状态，通过改变低维动作（如摄像机移动或末端执行器指令）所能达到的帧位于一个低维流形上。我们通过一个网络直接估计该流形的局部形状，该网络经过训练，在生成器样本数据的监督下，估计每个图像区域随每个动作轴单位的变化而如何移动。

A correction head fit to trajectory data estimates the shape of this manifold region up to a third-order Taylor polynomial. We call this network estimating the Taylor polynomial describing the manifold shape an Action Jet and compute its radius of validity in closed form.

一个拟合轨迹数据的校正头估计了该流形区域的形状，最高可达三阶泰勒多项式。我们将这个估计描述流形形状的泰勒多项式的网络称为“动作喷流”（Action Jet），并以闭式计算其有效半径。

By traversing this approximation of the local manifold shape, we only have to perform inference once per state for any number of candidate actions. Scoring sixteen candidates from a single extraction is over 300x faster than with either source generator, and this gap widens with the number of candidates a planner evaluates.

通过遍历这种局部流形形状的近似值，我们对于任意数量的候选动作，每个状态只需执行一次推理。从单次提取中对十六个候选动作进行评分，比使用任何源生成器快 300 倍以上，并且随着规划器评估候选动作数量的增加，这一差距会进一步扩大。

We measure against state-of-the-art action-conditioned generators, IRASim and WorldGym on BridgeData V2, a real 7-DoF manipulation benchmark, and FrameCrafter on DL3DV scenes, where we chart from the novel-view generators DepthSplat and LagerNVS.

我们针对最先进的动作条件生成器进行了评估，包括 BridgeData V2（一个真实的 7 自由度操作基准）上的 IRASim 和 WorldGym，以及 DL3DV 场景上的 FrameCrafter，并参考了新视角生成器 DepthSplat 和 LagerNVS 的图表。

Because only the tangent comes from the generator and every other term from the data, the Action Jet also predicts held-out frames more accurately than IRASim, the generator that supplied its tangent, at short horizons and at over 100x less compute.

由于只有切线来自生成器，而所有其他项均来自数据，因此在短视界内，Action Jet 预测留出帧的准确度也高于为其提供切线的生成器 IRASim，且计算量减少了 100 倍以上。