RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation

RouteRec：推荐智能体选择与聚合的严格评估

Abstract: Recommender systems increasingly face a choice among heterogeneous agents — collaborative filters, sequential models, content-based retrievers, and LLM-based rerankers — yet no single agent is uniformly best. We study this choice as task-aware agent ranking under cost constraints using RouteRec, a framework that compares request-level hard selection with item-level learned aggregation over four traditional recommender agents and one LLM reranker agent.

摘要： 推荐系统正日益面临在异构智能体（如协同过滤、序列模型、基于内容的检索器以及基于大语言模型的重排序器）之间进行选择的挑战，然而目前尚无单一智能体在所有场景下表现最优。我们通过 RouteRec 框架，将这一选择问题研究为成本约束下的任务感知智能体排序。该框架对比了请求级别的“硬选择”（hard selection）与项目级别的“学习聚合”（learned aggregation），涵盖了四个传统推荐智能体和一个大语言模型（LLM）重排序智能体。

On MovieLens-1M, the full quality oracle has substantial headroom (HR@10 = 0.584), confirming that useful cross-agent signal exists. Under a leakage-free 5-fold out-of-fold protocol, however, hard selection remains below BM25 (0.223 vs. 0.254), and selective LLM escalation does not improve it.

在 MovieLens-1M 数据集上，全质量预言机（full quality oracle）显示出巨大的提升空间（HR@10 = 0.584），证实了跨智能体之间存在有价值的信号。然而，在无泄漏的 5 折交叉验证协议下，硬选择的表现仍低于 BM25（0.223 对比 0.254），且选择性的 LLM 升级策略并未带来改善。

The same protocol yields a different outcome for learned aggregation: its cheap-only variant matches BM25 in HR and has a higher NDCG point estimate (0.123 vs. 0.114), while gated all-agent aggregation reaches HR@10 = 0.295 with 70.2% LLM calls. The resulting lesson is not that routing is solved, but that request-level selection of one complete agent list is too coarse for this sparse fixed-candidate setting; item-level aggregation is the more promising action space.

同样的协议对于“学习聚合”则得出了不同的结论：其仅使用低成本模型的变体在 HR 指标上与 BM25 持平，并具有更高的 NDCG 点估计值（0.123 对比 0.114）；而门控全智能体聚合在调用 70.2% LLM 的情况下，HR@10 达到了 0.295。由此得出的结论并非路由问题已得到解决，而是对于这种稀疏的固定候选集设置而言，请求级别的单一智能体列表选择过于粗糙；项目级别的聚合才是更具前景的行动空间。