Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Prediction

行为诱导的镜像近端时序差分学习：实现更快的离策略预测

Abstract: Gradient temporal-difference methods provide stable off-policy prediction with linear function approximation, but their practical performance is strongly affected by the geometry induced by the auxiliary-variable metric. Existing Mirror-Prox TD methods typically use the feature covariance metric, whereas hybrid TD methods suggest that behavior-policy transition information can provide a more informative update geometry.

摘要： 梯度时序差分（Gradient temporal-difference）方法在利用线性函数近似进行离策略预测时表现稳定，但其实际性能深受辅助变量度量所诱导的几何结构影响。现有的镜像近端（Mirror-Prox）TD方法通常使用特征协方差度量，而混合TD方法则表明，行为策略的转移信息能够提供更具信息量的更新几何结构。

This paper proposes a behavior-induced Mirror-Prox temporal-difference method, called STHTD-MP, which replaces the covariance metric in the primal-dual saddle-point formulation with the symmetric part of the behavior-policy Bellman matrix. The method keeps a single learning rate for the primal and auxiliary variables and applies a Mirror-Prox prediction-correction step to the resulting hybrid saddle-point operator.

本文提出了一种名为 STHTD-MP 的行为诱导镜像近端时序差分方法。该方法在原始-对偶鞍点公式中，将协方差度量替换为行为策略贝尔曼矩阵的对称部分。该方法为原始变量和辅助变量保持单一的学习率，并对所得的混合鞍点算子应用镜像近端预测-校正步骤。

We provide a formal convergence analysis for fixed-policy linear prediction under standard stochastic approximation assumptions: the behavior-induced metric is positive definite, the joint mean system is Hurwitz, boundedness follows from a Lyapunov argument, and the stochastic recursion converges by the ODE method.

我们在标准的随机近似假设下，为固定策略的线性预测提供了正式的收敛性分析：行为诱导度量是正定的，联合均值系统是 Hurwitz 稳定的，有界性通过 Lyapunov 参数论证得出，且随机递归通过常微分方程（ODE）方法收敛。

We further derive projected-oracle ergodic gap bounds and an exact mean-operator comparison with GTD2-MP based on the spectral radius of the deterministic Mirror-Prox error matrix. The analysis shows that STHTD-MP can have a smaller mean contraction factor than GTD2-MP when the behavior-induced metric improves the saddle-point geometry.

我们进一步推导了投影预言机遍历间隙界（projected-oracle ergodic gap bounds），并基于确定性镜像近端误差矩阵的谱半径，与 GTD2-MP 进行了精确的均值算子比较。分析表明，当行为诱导度量改善了鞍点几何结构时，STHTD-MP 可以比 GTD2-MP 具有更小的均值收缩因子。

Exact numerical mean-operator analysis on two-state, Random Walk, and Boyan Chain benchmarks supports this condition, while Baird’s counterexample is identified as a singular boundary case where the strict assumptions fail.

在两状态、随机游走（Random Walk）和 Boyan 链基准测试上的精确数值均值算子分析支持了这一结论，同时指出 Baird 反例是一个严格假设失效的奇异边界情况。