Function-Level Execution Feedback for Code Preference Optimization

基于函数级执行反馈的代码偏好优化

Abstract: Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize.

摘要： 过程监督（Process supervision）已在数学推理领域取得了显著进展，因为数学中的中间步骤可以自然地表达为思维链。然而，在代码生成领域，过程监督的研究尚不充分，原因在于目前缺乏关于“步骤”的标准定义。监督可以针对代码行、推理轨迹或程序状态，这使得确定标注和优化的对象变得模糊不清。

We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program.

我们提出了 STEP-KTODER，这是一个用于代码偏好优化的框架。该框架将“步骤”定义为分解后的多函数程序中的模块级函数，并通过自动生成的单元测试分配二元正确性标签。我们的方法为分步 KTO（Stepwise KTO）提供了一种针对代码的实例化方案，将函数级的过程监督与针对完整程序的输出级反馈相结合。

We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: this https URL.

我们在 HumanEval(+)、MBPP(+)、BigCodeBench 和 LiveCodeBench 上进行了评估，结果表明 STEP-KTODER 的表现优于仅基于输出的 KTO 和 DPO 方法。进一步分析显示，基于执行的标签至关重要：使用大模型作为裁判（LLM-as-a-judge）的标注会系统性地高估函数失败率，从而破坏正向步骤标签，并降低下游偏好优化的效果。代码已开源，详见：此链接。