Memory Internalization via Test-Time Training for Long-Video Understanding

Title: Memory Internalization via Test-Time Training for Long-Video Understanding

标题：通过测试时训练实现长视频理解的记忆内化

Long-video understanding often requires answering many questions about a single video whose evidence is distributed across extended temporal contexts. Conventional video-language models repeatedly encode frames or retrieve visual evidence for every question. Parameterized video memory offers a write-once alternative, but existing methods rely primarily on synthesized text and fixed adaptation configurations, leaving open where video knowledge should be written and how visual evidence should shape the memory.

长视频理解通常需要针对单个视频回答许多问题，而这些问题的证据分布在扩展的时间上下文中。传统的视频语言模型会针对每个问题重复编码帧或检索视觉证据。参数化视频记忆提供了一种“一次写入”的替代方案，但现有方法主要依赖于合成文本和固定的适配配置，这使得视频知识应该写入何处以及视觉证据应如何塑造记忆等问题仍未得到解决。

We introduce Memory Internalization via Test-Time Training (MIT^3), a per-video adaptation framework that writes each long video into compact, video-specific adaptive parameters for video-free, text-only question answering. Under compact multi-scale supervision, causal tracing measures the causal memory utility of each model layer for video-grounded answering and selects video-specific adaptation locations.

我们引入了通过测试时训练实现记忆内化（MIT^3），这是一个针对每个视频的适配框架，它将每个长视频写入紧凑的、特定于视频的自适应参数中，以实现无需视频、仅基于文本的问答。在紧凑的多尺度监督下，因果追踪技术可以衡量每个模型层对于基于视频的回答的因果记忆效用，并选择特定于视频的适配位置。

Visual memory distillation then trains the text-only memory to reproduce the representation change induced by the relevant video clip, preserving the internal effect of visual evidence beyond textual supervision. Across LVBench, LongVideoBench, and Video-MME, MIT^3 achieves data-efficient video-free question answering with substantially fewer synthesized samples than prior parameterized video-memory methods, demonstrating that effective video internalization depends on both where knowledge is written and how visual evidence is distilled into the model.

随后，视觉记忆蒸馏训练纯文本记忆，以重现由相关视频片段引起的表征变化，从而在文本监督之外保留视觉证据的内部影响。在 LVBench、LongVideoBench 和 Video-MME 测试中，MIT^3 以远少于先前参数化视频记忆方法的合成样本量，实现了数据高效的无视频问答，这证明了有效的视频内化既取决于知识写入的位置，也取决于视觉证据如何被蒸馏到模型中。