TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation

Abstract: Large-scale text corpora have become a quiet bottleneck in modern NLP, not just in storage, but in the accumulated cost of training, fine-tuning, and continual learning.

摘要： 大规模文本语料库已成为现代自然语言处理（NLP）中一个隐蔽的瓶颈，这不仅体现在存储方面，还体现在训练、微调和持续学习所累积的成本上。

We propose a text dataset distillation framework that reduces corpora to as little as 0.1% of their original size while preserving downstream task fidelity.

我们提出了一种文本数据集蒸馏框架，该框架能将语料库缩减至原始规模的 0.1%，同时保持下游任务的保真度。

We approach distillation through the lens of influence functions, which quantify each sample’s contribution to the downstream objective, a natural and principled basis for selection.

我们通过影响函数（influence functions）的视角进行蒸馏，该方法量化了每个样本对下游目标的贡献，为样本选择提供了一个自然且有原则的基础。

We introduce Trajectory-Aware Knowledge Estimation (TAKE), which convolves the knowledge-based influence along the training trajectory into a single per-sample knowledge score, capturing informative samples.

我们引入了轨迹感知知识估计（TAKE），它将训练轨迹上的基于知识的影响卷积为一个单一的样本知识评分，从而捕捉到信息量大的样本。

These scores serve as sample weights within a discrete Optimal Transport objective, guiding prototype selection from a synthetically generated candidate pool.

这些评分在离散最优传输（Optimal Transport）目标中作为样本权重，指导从合成生成的候选池中进行原型选择。

We evaluate TAKE on downstream accuracy across text classification and natural language inference tasks at extreme compression (0.1% or 20 samples/class), showing that data efficiency is achievable without sacrificing task fidelity.

我们在极端压缩率（0.1% 或每类 20 个样本）下，对文本分类和自然语言推理任务的下游准确率进行了评估，结果表明在不牺牲任务保真度的前提下，实现数据高效性是可行的。

The approach is theoretically grounded, with broader implications for coreset construction and data-centric AI. We release our source code at this https URL.

该方法具有坚实的理论基础，对核心集（coreset）构建和以数据为中心的 AI 具有更广泛的意义。我们已在链接处发布了源代码。