Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
Backtrader-Bench：利用自生成多项选择题评估算法交易中的大模型智能体
Abstract: Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution.
摘要： 在算法交易领域评估大模型（LLM）编程智能体具有一定难度，因为静态基准测试存在数据污染的风险，且数值回测结果需要通过实际代码执行来获取真实值（Ground Truth）。
We present Backtrader-Bench, a framework with two complementary pipelines. A deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations across five trading strategies, 33 templates, and three difficulty tiers, with an independent checker that re-derives every answer.
我们提出了 Backtrader-Bench，这是一个包含两个互补流水线的框架。其中，确定性多项选择题（MCQ）流水线通过五种交易策略、33 个模板和三个难度等级的回测配置生成问题，并配有一个能够重新推导每个答案的独立检查器。
A generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them to MCQs, and discards any that a no-tool solver can answer without code execution.
另一个生成器-求解器过滤流水线则用于自动挖掘难度更高的问题：生成器编写经由可执行代码验证的问题，将其转换为多项选择题，并剔除那些无需代码执行即可由无工具求解器回答的问题。
We evaluate 11 models without tools (10 runs each) and four with-tools configurations on a 30-question curated set. Tool-augmented agents reach 90.0% accuracy in a single pass (GPT-5.5 and Opus 4.7), outperforming the best no-tools baselines (73.0%, averaged over 10 runs) by 17 percentage points.
我们在一个包含 30 个问题的精选集上评估了 11 个无工具模型（每个模型运行 10 次）和 4 种带工具配置。配备工具的智能体在单次运行中达到了 90.0% 的准确率（GPT-5.5 和 Opus 4.7），比表现最好的无工具基准模型（平均准确率为 73.0%，基于 10 次运行）高出 17 个百分点。
On 38 separately mined questions, no-tools accuracy drops further, with half the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to produce a training corpus for reinforcement learning, with the ultimate goal of building a specialized agent for quantitative trading workflows.
在另外 38 个挖掘出的问题上，无工具模型的准确率进一步下降，半数模型的表现跌至随机猜测水平（约 25%）。除了评估功能外，该可扩展的 MCQ 基础设施还旨在为强化学习生成训练语料库，其最终目标是构建一个专门用于量化交易工作流的智能体。