Training Variable Long Sequences with Data-Centric Parallel

使用数据中心并行（Data-Centric Parallel）训练可变长序列

Abstract: Training deep learning models on variable long sequences poses significant computational challenges. Existing methods force a difficult trade-off between efficiency and ease-of-use. Simple approaches use static configurations that cause workload imbalance low efficiency, while complex methods introduces significant complexity and code change for new models.

摘要： 在可变长序列上训练深度学习模型带来了巨大的计算挑战。现有的方法在效率和易用性之间被迫进行艰难的权衡。简单的方法使用静态配置，导致工作负载不平衡和效率低下；而复杂的方法则为新模型引入了显著的复杂性和代码变更。

To break this trade-off, we introduce Data-Centric Parallel (DCP). Its core principle is to let the data itself drive the runtime. It achieves this by dynamically adjusting direct runtime settings (e.g., parallel size, gradient accumulation, recomputation) based on each batch’s sequence length.

为了打破这种权衡，我们引入了数据中心并行（Data-Centric Parallel, DCP）。其核心原则是让数据本身驱动运行时。它通过根据每个批次的序列长度动态调整直接运行时设置（例如并行大小、梯度累积、重计算）来实现这一点。

Empirical results demonstrate that our method achieves up to a 2.88$\times$ speedup on 32 H200 GPUs. Designed for generalization, it can be integrated into any model with 10 lines of code. We anticipate this simple yet effective approach will serve as a robust baseline and facilitate future advancements in distributed training for variable long sequences.

实证结果表明，我们的方法在 32 块 H200 GPU 上实现了高达 2.88 倍的加速。该方法专为通用性而设计，仅需 10 行代码即可集成到任何模型中。我们预计这种简单而有效的方法将成为一个稳健的基准，并促进可变长序列分布式训练的未来发展。