Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
评估用于计算与实验数学的 SageMath 增强型大语言模型智能体
Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored. We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath, together with Context7 for the up-to-date documentation.
摘要： 近期人工智能在数学领域的发展主要集中于自动形式化和定理证明，而计算机代数系统（CAS）在智能体大语言模型工作流中的作用尚未得到充分探索。我们提出了一种 ReAct 风格的智能体架构，将大语言模型的推理能力与来自 SageMath 的可验证反馈相结合，并辅以 Context7 提供最新的文档支持。
We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark in a setting that emulates a computational-mathematics research loop. We also propose a refinement to the RealMath benchmark by introducing a multi-step post-processing procedure and a multi-stage validation pipeline, both of which improve the quality and reliability of the extracted problem set.
我们在模拟计算数学研究循环的环境中，评估了该智能体架构在解决 RealMath 基准测试中研究级数学问题时的表现，并涵盖了多个前沿模型。我们还通过引入多步后处理程序和多阶段验证流水线，对 RealMath 基准测试进行了改进，这两项措施均提升了所提取问题集的质量与可靠性。
Our experiments reveal substantial performance gains from SageMath access across all evaluated models on +9.7pp on average, the gains range from 1.5pp to 27.8pp and narrow the gap between open-weight and closed models. Qwen3.7-Max benefits from SageMath the most, while GPT-5.5 achieves the highest solve rate of 75.2% and the lowest token usage among tool-enabled configurations.
实验结果表明，在所有评估模型中，接入 SageMath 后性能均有显著提升，平均提升幅度达 9.7 个百分点，增幅范围在 1.5 至 27.8 个百分点之间，这缩小了开源权重模型与闭源模型之间的差距。其中，Qwen 3.7-Max 从 SageMath 中获益最多，而 GPT-5.5 在所有启用工具的配置中实现了 75.2% 的最高求解率，且代币消耗量最低。
Our findings suggest that CAS-augmented agents represent a promising direction for assisting mathematicians in computational exploration, and we believe that this work is a step towards automated conjecture discovery. The project repository is available online.
我们的研究结果表明，CAS 增强型智能体是辅助数学家进行计算探索的一个极具前景的方向，我们相信这项工作是迈向自动化猜想发现的一步。项目代码库已在线发布。