Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries

Computer Science > Computation and Language arXiv:2611.00420 (cs) [Submitted on 30 Oct 2026] Title: Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries Authors: Shariar Kabir, Hong Kiat Tan, Swastik Agrawal, Sai V R Chereddy, Sriram Balasubramanian.

计算机科学 > 计算与语言 arXiv:2611.00420 (cs) [提交于 2026 年 10 月 30 日] 标题：Circuit Oracle：通过自然语言查询自动化归因图分析 作者：Shariar Kabir, Hong Kiat Tan, Swastik Agrawal, Sai V R Chereddy, Sriram Balasubramanian。

Abstract: Attribution graphs, an emerging tool in mechanistic interpretability, use transcoders to decompose language model computations into sparse interpretable features connected by causal edges. However, turning a graph into a safety-relevant insight requires hours of manual analysis by experts. We introduce Circuit Oracle, an off-the-shelf LLM equipped with a multi-agent harness and task-specific authored skills, which answers natural-language questions about a target model by reading the attribution graph through tool calls.

摘要：归因图是机械可解释性领域的一种新兴工具，它使用转码器将语言模型计算分解为由因果边连接的稀疏可解释特征。然而，将图表转化为与安全相关的见解需要专家进行数小时的手动分析。我们引入了 Circuit Oracle，这是一个配备了多智能体框架和特定任务技能的现成大语言模型，它通过工具调用读取归因图，从而回答有关目标模型的自然语言问题。

We evaluate it on three safety-relevant proxy tasks: detecting spurious features in probe circuits, eliciting hidden knowledge from taboo-finetuned models, and suppression jailbreaking via feature-level interventions. Beyond task scores, the oracle produces pruned subgraphs and context-specific feature explanations that we assess qualitatively. Results are mixed. On spurious-feature detection, Circuit Oracle reaches 78.2 $\pm$ 2.4% accuracy, outperforming single-layer feature-ranking baselines.

我们在三个与安全相关的代理任务上对其进行了评估：检测探测电路中的虚假特征、从禁忌微调模型中引出隐藏知识，以及通过特征级干预抑制越狱。除了任务得分外，该预言机还生成了我们进行定性评估的修剪子图和特定上下文的特征解释。结果喜忧参半。在虚假特征检测方面，Circuit Oracle 的准确率达到了 78.2 $\pm$ 2.4%，优于单层特征排序基线。

On secret elicitation, trained self-explainers like Activation Oracles remain the stronger baseline. On refusal suppression over 50 prompts, its single committed intervention is slightly below a diff-in-means baseline, and only exceeds it when allowed the best of five attempts. Results are stable across five runs and across model choices for the agents. We also release the pruned subgraphs and context-specific feature readings that revise existing global labels, and invite future work to develop evaluations that can measure their usefulness directly.

在秘密引出方面，像 Activation Oracles 这样经过训练的自解释器仍然是更强的基线。在针对 50 个提示的拒绝抑制任务中，其单次干预效果略低于均值差异基线，只有在允许五次尝试中取最佳结果时才能超过该基线。结果在五次运行和智能体的不同模型选择中保持稳定。我们还发布了修剪后的子图和特定上下文的特征解读，这些内容修正了现有的全局标签，并邀请未来的研究开发能够直接衡量其效用的评估方法。