Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

选择审核的位置与方式：过滤器部署与回复重写中的端到端权衡

Abstract: Content-moderation classifiers are usually evaluated in isolation, but deployment requires choosing where to intervene and what follows a flag. We evaluate these choices using two end-to-end customer-outcome metrics rather than component accuracy: Usefulness, the fraction of turns with a shown, non-harmful, relevant response, and Harmful Exposure, the fraction with a shown harmful response. Latency and error rates are diagnostics.

摘要： 内容审核分类器通常是独立评估的，但在实际部署中，需要选择干预的位置以及触发警报后的后续处理方式。我们使用两个端到端的客户结果指标而非组件准确率来评估这些选择：即“有用性”（Usefulness，指显示出无害且相关回复的对话轮次占比）和“有害暴露”（Harmful Exposure，指显示出有害回复的对话轮次占比）。延迟和错误率则作为诊断指标。

We compare Input only, Response only, and Input + response hard blocking on a human-labelled product benchmark and public ToxicChat evaluation. At the evaluated operating points, Response only achieves the highest filter-only Usefulness in both settings, while Input + response achieves lower Harmful Exposure. Replacing Response only blocking with Response + rewrite recovers most blocked traffic and yields the same observed Harmful Exposure count as Response only blocking for the selected configuration; this equality is not an equivalence result.

我们在人工标注的产品基准测试和公开的 ToxicChat 评估中，对比了“仅输入端”、“仅回复端”以及“输入+回复端”的硬拦截策略。在评估的操作点上，“仅回复端”在两种环境下均实现了最高的过滤器有用性，而“输入+回复端”则实现了更低的有害暴露率。将“仅回复端”拦截替换为“回复+重写”策略，可以恢复大部分被拦截的流量，且在选定配置下，其观察到的有害暴露计数与“仅回复端”拦截相同；但这种数值上的相等并不等同于效果上的等价。

Probe routing substantially reduces conditional route-and-generation time relative to LLM routing at comparable measured outcomes. A focused output review shows how rewrites balance filter passage with usefulness by generalizing triggering language while retaining benign intent and safe redirection; some sensitive-domain outputs nevertheless omit potentially safety-relevant support information. These results support comparing moderation configurations under deployment-specific safety and latency constraints rather than applying a universal placement rule.

在测量结果相当的情况下，探测路由（Probe routing）相比大模型（LLM）路由显著减少了条件路由和生成的耗时。针对输出内容的重点审查显示，重写机制通过泛化触发词汇，同时保留良性意图和安全重定向，从而在通过过滤器与保持有用性之间取得了平衡；不过，某些敏感领域的输出仍会遗漏潜在的安全相关支持信息。这些结果支持在部署特定的安全和延迟约束下比较审核配置，而不是盲目应用通用的部署规则。