Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification
Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification
文档敏感度分类中经典模型与 Transformer 模型基准测试
Abstract: Automatic sensitivity classification of organizational documents is a critical yet underserved problem, where the consequences of misclassification range from regulatory violations to security breaches. While AI-based approaches offer a scalable alternative to manual review, their reliability depends fundamentally on the integrity of training data.
摘要： 组织文档的自动敏感度分类是一个关键但尚未得到充分研究的问题，分类错误的后果从违反监管规定到安全漏洞不等。虽然基于人工智能的方法为人工审核提供了一种可扩展的替代方案，但其可靠性从根本上取决于训练数据的完整性。
A pervasive but underreported problem in this domain is label leakage: residual classification markers embedded within document bodies that allow models to exploit surface shortcuts rather than learning genuine content-based sensitivity signals, producing performance estimates that are inflated and unreliable.
在该领域中，一个普遍存在但鲜有报道的问题是“标签泄露”（label leakage）：即嵌入在文档正文中的残留分类标记，这使得模型能够利用表层捷径，而非学习真正基于内容的敏感度信号，从而导致性能评估结果虚高且不可靠。
This paper addresses this problem by introducing Strategic 16K, a carefully constructed, leakage-controlled corpus of 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD), and presents a systematic benchmark evaluating six model architectures spanning classical machine learning and transformer-based approaches.
本文通过引入“Strategic 16K”来解决这一问题。这是一个经过精心构建、受控泄露的语料库，包含来自维基解密美国外交公共图书馆（PlusD）的 16,000 份外交电报，并提出了一个系统性的基准测试，评估了涵盖经典机器学习和基于 Transformer 方法的六种模型架构。
We document an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies. On the clean benchmark, BERT achieves the strongest performance (Accuracy = 89.14%, F1 = 89.33%), followed by ELECTRA (Accuracy = 88.57%, F1 = 88.90%).
我们记录了一套扩展的泄露清除协议，用于识别并消除嵌入在文档正文中的三类残留分类标记。在清洗后的基准测试中，BERT 表现最强（准确率 = 89.14%，F1 分数 = 89.33%），其次是 ELECTRA（准确率 = 88.57%，F1 分数 = 88.90%）。
Among classical models, TF-IDF with Logistic Regression achieves the strongest performance at significantly lower computational cost. These results constitute the first fully reproducible sensitivity classification benchmark constructed under explicit leakage-controlled conditions from WikiLeaks PlusD.
在经典模型中，TF-IDF 结合逻辑回归（Logistic Regression）以显著更低的计算成本实现了最强的性能。这些结果构成了首个在明确的泄露受控条件下，基于维基解密 PlusD 构建的、完全可复现的敏感度分类基准。