OpenESM: Energy-Steered Models Are Scalable Foundation Learners

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.33005 (cs) [Submitted on 7 Oct 2026] Title: OpenESM: Energy-Steered Models Are Scalable Foundation Learners Authors: Yuan Pu, Yudong Lu, Ting Wang, Zhicong Hong, Qinuo Li, Xuansheng Wu, Na Zou, Xia Hu, Guanchu Wang.

计算机科学 > 计算与语言 arXiv:2610.33005 (cs) [提交于 2026 年 10 月 7 日] 标题：OpenESM：能量引导模型是可扩展的基础学习器作者：Yuan Pu, Yudong Lu, Ting Wang, Zhicong Hong, Qinuo Li, Xuansheng Wu, Na Zou, Xia Hu, Guanchu Wang。

Abstract: Energy-based models have shown promising properties for domain-specific tasks but remain challenging in modeling general-purpose scenarios. To bridge this gap, we design energy-steered models (ESMs) for foundational language modeling. ESMs integrate energy steering and token generation within a single architecture to leverage the energy-guided search to refine token generation.

摘要：基于能量的模型在特定领域任务中表现出了良好的特性，但在通用场景建模方面仍然具有挑战性。为了弥补这一差距，我们设计了用于基础语言建模的能量引导模型（ESMs）。ESMs 将能量引导和标记生成集成在单一架构中，利用能量引导搜索来优化标记生成。

Specifically, ESMs first optimize a latent state over the energy landscape, then leverages the refined latent state for token generation. By performing such energy-guided pre-training on large-scale corpora, ESMs learn generally effective energy landscape where favorable tokens correspond to lower energy values.

具体而言，ESMs 首先在能量图景上优化潜在状态，然后利用优化后的潜在状态进行标记生成。通过在大规模语料库上进行这种能量引导的预训练，ESMs 学习到了普遍有效的能量图景，其中理想的标记对应于较低的能量值。

To explore the scaling behavior of ESMs, we pre-train ESMs on three large-scale corpora, then evaluate across 26 downstream tasks spanning a broad range of domains. Our results demonstrate that ESMs have promising scaling behaviors with increasing model size and training data, and outperform GPTs on most downstream tasks.

为了探索 ESMs 的扩展行为，我们在三个大规模语料库上对 ESMs 进行了预训练，然后在涵盖广泛领域的 26 个下游任务中进行了评估。我们的结果表明，随着模型规模和训练数据的增加，ESMs 具有良好的扩展行为，并且在大多数下游任务中优于 GPT 模型。

These findings highlight ESMs as a promising architecture for large-scale pre-training and general purpose problem solving. We fully release the model weights and training infrastructure to enable the community to completely reproduce and extend our results.

这些发现突显了 ESMs 作为大规模预训练和通用问题解决的有前途的架构。我们完全发布了模型权重和训练基础设施，以使社区能够完全复现并扩展我们的结果。