NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation

NL2SHACL-Bench：用于自然语言转 SHACL 的基准测试套件

SHACL is a core technology for validating the conformance of RDF knowledge graphs (KGs). Yet, authoring SHACL shapes requires technical expertise that most domain experts lack. Translating natural language requirements into SHACL (NL2SHACL) would lower this barrier. SHACL 是验证 RDF 知识图谱（KG）一致性的核心技术。然而，编写 SHACL 形状（shapes）需要大多数领域专家所不具备的专业技术知识。将自然语言需求翻译为 SHACL（即 NL2SHACL）可以降低这一门槛。

However, there is no dedicated benchmark for NL2SHACL, and evaluating generated shapes requires methods beyond string comparison, as semantically equivalent shapes can differ in serialisation and structure. To tackle these challenges, we present NL2SHACL-Bench, a benchmark suite for natural language to SHACL translation. 然而，目前尚无专门针对 NL2SHACL 的基准测试，且评估生成的形状需要超越字符串比较的方法，因为语义等价的形状在序列化和结构上可能存在差异。为了应对这些挑战，我们提出了 NL2SHACL-Bench，这是一个用于自然语言转 SHACL 翻译的基准测试套件。

Using NL2SHACL-Bench, we evaluate four state-of-the-art large language models (LLMs) for this task. Our results show that current LLMs are highly capable of generating syntactically valid SHACL, but still struggle to produce semantically equivalent constraints for complex logical and structural patterns. This indicates that NL2SHACL-Bench provides a meaningful basis for measuring advances in the NL2SHACL state of the art. 利用 NL2SHACL-Bench，我们评估了四种最先进的大型语言模型（LLM）在该任务上的表现。结果表明，当前的 LLM 在生成语法正确的 SHACL 方面能力很强，但在处理复杂的逻辑和结构模式时，仍难以生成语义等价的约束。这表明 NL2SHACL-Bench 为衡量 NL2SHACL 领域的技术进展提供了一个有意义的基准。