PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems

PhysMent：一种用于大语言模型物理推理的交互式方法

Large language models (LLMs) perform strongly on static science benchmarks, yet their ability to reason about the physical world through active experimentation remains poorly understood. 大语言模型（LLMs）在静态科学基准测试中表现强劲，但它们通过主动实验来推理物理世界的能力仍未得到充分理解。

We introduce PhysMent, a benchmark that evaluates LLM physical reasoning via iterative, tool-mediated interaction with a MuJoCo physics simulator. 我们引入了 PhysMent，这是一个通过与 MuJoCo 物理模拟器进行迭代式、工具辅助交互来评估大语言模型物理推理能力的基准测试。

Unlike static benchmarks that supply all quantities upfront, PhysMent requires models to discover information by applying forces, querying object states, advancing time, and modifying scene geometry before answering. 与那些预先提供所有数值的静态基准测试不同，PhysMent 要求模型在回答问题前，通过施加力、查询对象状态、推进时间以及修改场景几何结构来主动发现信息。

The benchmark comprises 105 scenes of classical mechanics, organized across four difficulty regimes (Easy/Hard and Single/Multi), three scene modalities (standard, object creation, hidden objects), and a scene-manipulation category, evaluated with a six-dimensional scoring framework. 该基准测试包含 105 个经典力学场景，分为四个难度等级（简单/困难和单体/多体）、三种场景模态（标准、对象创建、隐藏对象）以及一个场景操作类别，并采用六维评分框架进行评估。

Results show that current models perform reasonably well on qualitative single-concept tasks (up to 80% accuracy) but degrade substantially on quantitative tasks that demand precise, multi-step experimental procedures: most models fall below 30% on the hardest single-concept category, where the bottleneck is procedural (adaptive multi-step tool use) rather than conceptual load. 结果显示，当前模型在定性的单概念任务上表现尚可（准确率高达 80%），但在需要精确、多步实验过程的定量任务上表现大幅下降：在最困难的单概念类别中，大多数模型的准确率低于 30%，其瓶颈在于程序化操作（自适应多步工具使用）而非概念理解负荷。

Across the seven models, accuracy ranges from 25% to 67%, with failures due to premature answer submission, inefficient exploration, and inconsistent grounding in simulator feedback rather than conceptual gaps. 在七个模型中，准确率范围从 25% 到 67% 不等，失败原因主要归结为过早提交答案、探索效率低下以及对模拟器反馈的理解不一致，而非概念性知识的缺失。