Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs
Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs
大语言模型真的能理解题目难度吗？对使用大语言模型进行自动出题的启示
Abstract: The estimation of item difficulty plays a key role in both formative assessment and large-scale high-stakes summative assessments. This study explores how large language models (LLMs) perform in predicting item difficulty levels using items from a large-scale Reading and Writing test. 摘要： 题目难度评估在形成性评价和大规模高风险总结性评价中均发挥着关键作用。本研究探讨了大语言模型（LLMs）在使用大规模阅读与写作测试题目进行难度等级预测时的表现。
The study investigated various prompting strategies and parameter settings across multiple LLMs. LLM performance was compared with encoder-only language models and feature-based supervised machine learning models. 本研究调查了多个大语言模型在不同提示策略和参数设置下的表现，并将大语言模型的性能与仅编码器（encoder-only）语言模型以及基于特征的监督机器学习模型进行了对比。
Zero-shot GPT-4.1 with a temperature of 0 yielded the highest item difficulty level prediction accuracy, with a quadratic weighted kappa (QWK) of 0.578. However, LLMs’ prediction accuracy was lower than that of ConvBERT (QWK = 0.625), which outperformed the best feature-based supervised machine learning model. 在温度参数为 0 的零样本（Zero-shot）GPT-4.1 模型中，题目难度等级预测的准确率最高，其二次加权 Kappa 系数（QWK）为 0.578。然而，大语言模型的预测准确率仍低于 ConvBERT（QWK = 0.625），后者表现优于表现最好的基于特征的监督机器学习模型。
Further analysis showed that all LLMs struggled to label hard items; in particular, the current advanced GPT-5.4 tended to underestimate item difficulty levels. Dimension reduction of embeddings showed that item embeddings from different difficulty levels were mixed together, indicating that semantic information from items alone is likely insufficient for item difficulty level prediction. 进一步分析表明，所有大语言模型在标注高难度题目时都表现吃力；特别是当前先进的 GPT-5.4 模型，往往倾向于低估题目难度。对嵌入向量进行降维分析显示，不同难度等级的题目嵌入向量混杂在一起，这表明仅凭题目的语义信息可能不足以进行难度等级预测。
The findings suggest that if LLMs cannot understand item difficulty levels as evidenced by empirical data and tend to treat most items as easy when their own capabilities increase, caution should be exercised when using LLMs to generate items with targeted difficulty levels. 研究结果表明，如果实证数据证明大语言模型无法理解题目难度，且随着模型自身能力的提升反而倾向于将大多数题目视为简单题目，那么在利用大语言模型生成具有特定难度要求的题目时，应当保持谨慎。