RFCLLM: Evaluating LLMs’ Reasoning Ability of Network Protocol State Machines

RFCLLM：评估大语言模型对网络协议状态机的推理能力

Mapping textual specifications into formal representations is essential for ensuring the correctness of protocol designs and implementations. LLM-generated mappings, used for networking security or testing, are assumed to capture a perfect understanding of the specification, which may not hold in practice. 将文本规范映射为形式化表示对于确保协议设计和实现的正确性至关重要。用于网络安全或测试的大语言模型（LLM）生成的映射通常被假设为能够完美理解规范，但在实际应用中情况可能并非如此。

The goal of this paper is to assess the extent to which LLMs can interpret the specification correctly. We examine the degree to which an LLM’s implicit representation of a finite-state transition system—defined via natural language descriptions—aligns with a manually generated ground-truth model. 本文旨在评估大语言模型正确解读规范的程度。我们研究了大语言模型通过自然语言描述所定义的有限状态转换系统的隐式表示，与人工生成的基准模型（Ground-truth model）之间的一致性程度。

We designed 4 tasks and 1482 task queries for 16 protocols. We evaluated different judge biases, observed the inherent difficulty gaps between tasks, looked into the effect of 4 context types, and the influence of protocol characteristics. 我们针对 16 种协议设计了 4 项任务和 1482 个任务查询。我们评估了不同的评判偏差，观察了任务之间固有的难度差距，并探讨了 4 种上下文类型的影响以及协议特征的作用。

Our work contributes to a step toward verifying whether LLMs can really be trusted in FSM (Finite State Machine) reasoning of protocol specifications. 我们的工作迈出了验证大语言模型在协议规范的有限状态机（FSM）推理方面是否真正值得信赖的一步。