Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models

Fairness benchmarks for large language models (LLMs) are overwhelmingly English-born and American in their social categories; translated versions carry those categories into societies where they do not apply, and almost none fix their hypotheses before the models are run.

大型语言模型（LLM）的公平性基准测试绝大多数源自英语国家，且其社会分类具有浓厚的美国色彩；翻译版本将这些分类带入并不适用的社会中，且几乎没有基准测试会在模型运行前确定其假设。

We present Qistas (Arabic qistas, “the just balance”), a protocol for evaluating how LLMs treat the social, confessional, and regional constituencies of a society, portable across countries and instantiated first for Iraq.

我们提出了 Qistas（阿拉伯语意为“公正的平衡”），这是一种用于评估大语言模型如何对待社会、宗教和区域群体的协议，该协议可在不同国家间移植，并首先在伊拉克进行了实例化应用。

In its pilot form it measures only what needs no verdict on contested truth: procedural symmetry across matched counterfactual pairs, differential refusal, stereotype reliance under ambiguity, and terminological framing of documented events.

在其试点形式中，它仅测量那些无需对争议性事实进行裁决的内容：匹配反事实对之间的程序对称性、差异化拒绝、模糊情况下的刻板印象依赖，以及对已记录事件的术语框架。

Every measurement is paired by construction and analysed with paired machinery (sign-flip permutation tests, exact McNemar tests, template-clustered bootstrap intervals, Holm-corrected families), with a power analysis that states, before any data exist, the smallest effect each axis can detect after the design effect of template clustering.

每一项测量在构建时均采用配对方式，并使用配对机制（符号翻转置换检验、精确 McNemar 检验、模板聚类自助法区间、Holm 校正族）进行分析；此外还包含功效分析，在任何数据存在之前，说明在考虑模板聚类的设计效应后，每个轴所能检测到的最小效应。

We release the authoring apparatus (a pruned 3x5 taxonomy with four cells closed by design, per-type checklists, a cumulative rulings log, a structural audit, and a human review notebook) and report its first field test: a semantic review of a 58-item seed bank in which 24 of 25 template groups failed at least one check.

我们发布了编写工具（包含一个经过精简的 3x5 分类法，其中四个单元格按设计关闭、每种类型的检查清单、累积裁决日志、结构审计和人工审查笔记本），并报告了其首次实地测试结果：对一个包含 58 个项目的种子库进行了语义审查，其中 25 个模板组中有 24 个至少有一项检查未通过。

The defect patterns it exposed (evaluation criteria leaking into prompts, token-level swaps that break Arabic agreement, epistemically unbalanced pairs, single-theme refusal probes, unwritten dialect twins counted as coverage) are, we argue, general hazards of counterfactual fairness benchmarks.

我们认为，该测试暴露出的缺陷模式（评估标准泄露到提示词中、破坏阿拉伯语一致性的词元级替换、认知上不平衡的配对、单一主题的拒绝探测、被计入覆盖范围的未书写方言对）是反事实公平性基准测试中普遍存在的风险。

This is a protocol paper, not a results paper: no model has been evaluated, and the accompanying pre-registration commits the hypotheses, metrics, exclusion rules, and analysis plan that the pilot will follow.

这是一篇协议论文，而非结果论文：目前尚未对任何模型进行评估，随附的预注册文件明确了试点将遵循的假设、指标、排除规则和分析计划。