GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization

GRASP：通过组相对策略优化增强语言模型匿名化器

Large language models can infer sensitive personal attributes, such as age, location, and occupation, from ordinary text, turning everyday writing into a privacy risk. Adversarial anonymization defends against this by rewriting a text with a capable language model that also plays the attacker, but it needs a powerful model at inference time and thus sends private text to a third party, the very exposure anonymization should prevent.

大型语言模型可以从普通文本中推断出年龄、地点和职业等敏感个人属性，从而使日常写作面临隐私风险。对抗性匿名化通过使用一个既能重写文本又能扮演攻击者的强大语言模型来防御这种风险，但它在推理时需要一个强大的模型，因此必须将私密文本发送给第三方，而这正是匿名化本应防止的泄露行为。

Recent work distills this behavior into a small on-device model using supervised fine-tuning and direct preference optimization (DPO), but DPO only imitates the teacher’s offline choices and never directly optimizes the privacy—utility objective we care about.

最近的研究通过监督微调和直接偏好优化（DPO）将这种行为蒸馏到一个小型设备端模型中，但 DPO 只是模仿教师模型的离线选择，从未直接优化我们所关注的“隐私-效用”目标。

We introduce GRASP (Group-Relative Anonymization via Self-refinement Policy-optimization), which reinforces the local anonymizer online with Group Relative Policy Optimization. A single small model acts as anonymizer, adversary, and utility judge, trained against a self-generated reward that hides attributes while preserving meaning, with a design that guards against reward hacking.

我们引入了 GRASP（通过自我优化策略进行组相对匿名化），它利用组相对策略优化（Group Relative Policy Optimization）在在线环境下增强本地匿名化器。该方案使用单一小型模型同时充当匿名化器、攻击者和效用评估者，通过自我生成的奖励进行训练，在隐藏属性的同时保留语义，其设计还能有效防止奖励欺骗（reward hacking）。

Trained on Llama-3.1-8B, GRASP improves the privacy—utility trade-off over the DPO-distilled baseline, consistently across three independent LLM judges. Against adversarial anonymization driven by frontier models such as Gemini 2.5 Flash and Claude, it achieves a comparable or better overall trade-off while removing substantially more private information, and it runs entirely on-device at roughly 1% of the GPT-4o teacher’s cost.

基于 Llama-3.1-8B 训练的 GRASP 在三个独立的 LLM 评估器中，均表现出比 DPO 蒸馏基线更好的“隐私-效用”权衡。与由 Gemini 2.5 Flash 和 Claude 等前沿模型驱动的对抗性匿名化相比，它在实现相当或更优的整体权衡的同时，去除了更多的隐私信息，并且完全在设备端运行，成本仅为 GPT-4o 教师模型的 1% 左右。