Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models

Computer Science > Artificial Intelligence arXiv:2610.26904 (cs) [Submitted on 16 Oct 2026] Title: Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models Authors: Koki Takeshita, Kenji Uebayashi, Hikaru Tomonari, Masaya Tsunokake, Koichiro Yawata, Tatsuya Sasaki, Yusuke Ohtsubo, Kota Dohi, Yasuhiro Sogawa.

计算机科学 > 人工智能 arXiv:2610.26904 (cs) [提交于 2026 年 10 月 16 日] 标题：通过大语言模型进行多目标强化学习的帕累托最优 RTL 代码生成 作者：Koki Takeshita, Kenji Uebayashi, Hikaru Tomonari, Masaya Tsunokake, Koichiro Yawata, Tatsuya Sasaki, Yusuke Ohtsubo, Kota Dohi, Yasuhiro Sogawa。

Abstract: Recent studies have proposed reinforcement learning with verifiable rewards (RLVR) methods that enable LLMs to generate high-quality RTL circuits from natural-language specifications. Existing approaches optimize circuit PPA (Power, Performance, Area) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto-optimal solutions across different trade-offs such as low power, low latency, and small area.

摘要：近期的研究提出了带有可验证奖励的强化学习（RLVR）方法，使大语言模型（LLM）能够根据自然语言规范生成高质量的 RTL 电路。现有的方法通过将多个目标聚合成单一标量奖励来优化电路的 PPA（功耗、性能、面积），这使得在低功耗、低延迟和小面积等不同权衡之间同时生成多样化的帕累托最优解变得困难。

However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high-quality candidate set is beneficial in that designers can choose a preferred trade-off a posteriori. In this work, we are the first to propose a learning method for LLM-based circuit generation that optimizes, not the quality of a single circuit, but the quality and diversity of the entire circuit set obtained through multiple generations in the PPA objective space.

然而，在实际设计中，每个目标的相对重要性会根据产品需求和设计约束而变化；因此，获得一个多样化且高质量的候选集是有益的，因为设计师可以在事后选择首选的权衡方案。在这项工作中，我们首次提出了一种基于 LLM 的电路生成学习方法，它优化的不是单个电路的质量，而是通过在 PPA 目标空间中多次生成所获得的整个电路集的质量和多样性。

The proposed method consists of (1) a generation-based training pipeline that reformulates coverage improvement of the entire Pareto solution set into a sequential circuit generation problem, and (2) a reward function based on Hypervolume Contribution (HVC) that measures how much a generated circuit contributes to the Pareto front.

所提出的方法包括：(1) 一个基于生成的训练流水线，它将整个帕累托解集的覆盖率提升重新表述为一个顺序电路生成问题；(2) 一个基于超体积贡献（HVC）的奖励函数，用于衡量生成的电路对帕累托前沿的贡献程度。

Evaluation on the RTLLM v2 benchmark shows that the proposed method achieves +13.3 percentage points in relative Hypervolume over a baseline without PPA optimization and +14.4 percentage points over a single-objective reinforcement learning method, while also demonstrating superiority in circuit diversity.

在 RTLLM v2 基准测试上的评估表明，该方法在相对超体积方面比没有 PPA 优化的基线提高了 13.3 个百分点，比单目标强化学习方法提高了 14.4 个百分点，同时在电路多样性方面也表现出优越性。