Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

Oyster-II：用于大语言模型建设性安全对齐的强化学习方法

Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. 大语言模型（LLMs）在各种应用中展现出了卓越的能力，然而，如何同时确保其安全性、有用性和可信度仍然是一个持续存在的挑战。

Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. 传统的以拒绝为导向的对齐策略虽然减轻了有害内容的生成，但往往无法满足用户的合法需求，经常隐瞒那些本可以安全且建设性地解决敏感查询背后意图的信息。

Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. 基于 Oyster-I 开创的建设性安全范式（该范式超越了“一刀切”的拒绝，转向深思熟虑的、以响应为导向的安全对齐），我们指出了其基于监督微调（SFT）方案的两个关键局限性：一是对分布外场景的安全泛化能力不足；二是出现了一种我们称之为“安全思维链（CoT）过度泛化”的现象，即安全导向的推理模式被过度应用于良性查询，从而降低了有用性和用户体验。

To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning approach. 为了解决这些局限性，我们提出了 Oyster-II，这是一个基于强化学习（RL）的建设性安全对齐框架，它采用了“零样本强化学习（Zero-RL）”范式，并结合了多阶段强化学习方法。

Across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B. 在广泛的基准测试中，Oyster-II 在安全维度上全面超越了 Qwen3-14B 及其前身 Oyster-I，实现了与 Qwen3-Max 和 Qwen3.5-397B 相当的跨规模性能。