Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Dream-RSI：通过演化世界实现递归自我改进

Abstract: Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts.

摘要： 递归自我改进对于自主 AI 智能体而言正变得日益重要，其进展取决于能否在复杂领域中发现高价值的解决方案。这一过程的驱动力在于有效的探索，然而，管理和改进探索策略仍然是一个主要的瓶颈。当前的系统面临着一个根本性的困境：当搜索空间扩大时，固定策略无法适应；而在线策略优化则需要在长周期运行中，在延迟且昂贵的反馈下导航庞大的元搜索空间。

We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations.

我们引入了 \textsc{Dream-RSI}，这是一个用于可扩展且递归自我改进探索的框架。一个轻量级的编排层使探索过程变得显式且可编程，同时保持底层编码智能体不变。我们的核心见解是，积累的发现历史可以作为已实现搜索空间的重放模拟器。通过在由历史发现树构建的重放模拟器中进行“梦境（dreaming）”，\textsc{Dream-RSI} 能够获得即时、低成本的离线策略反馈，从而在无需调用重复且昂贵的在线评估的情况下，评估并优化探索策略。

The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.

改进后的策略随后被重新部署到在线环境中以推动进一步的发现，在一个自我改进的循环中不断扩展模拟器池。在算法工程、数学优化和 GPU 内核工程等多个领域中，\textsc{Dream-RSI} 在实现具有竞争力或更优发现质量的同时，在多种设置下大幅降低了发现成本。