A Systematic Approach for Large Language Models Debugging

大语言模型调试的系统化方法

Large language models (LLMs) have become central to modern AI workflows, powering applications from open-ended text generation to complex agent-based reasoning. However, debugging these models remains a persistent challenge due to their opaque and probabilistic nature and the difficulty of diagnosing errors across diverse tasks and settings.

大语言模型（LLM）已成为现代人工智能工作流的核心，为从开放式文本生成到复杂的基于智能体的推理等各类应用提供支持。然而，由于这些模型具有不透明性和概率性，且在不同任务和环境下诊断错误存在困难，调试这些模型仍然是一个持续存在的挑战。

This paper introduces a systematic approach for LLM debugging that treats models as observable systems, providing structured, model-agnostic methods from issue detection to model refinement. By unifying evaluation, interpretability, and error-analysis practices, our approach enables practitioners to iteratively diagnose model weaknesses, refine prompts and model parameters, and adapt data for fine-tuning or assessment, while remaining effective in contexts where standardized benchmarks and evaluation criteria are lacking.

本文介绍了一种用于大语言模型调试的系统化方法，将模型视为可观测系统，提供了从问题检测到模型优化的结构化、与模型无关的方法。通过整合评估、可解释性和错误分析实践，我们的方法使从业者能够迭代地诊断模型弱点、优化提示词和模型参数，并调整用于微调或评估的数据，同时在缺乏标准化基准和评估标准的场景下依然保持有效。

We argue that such a structured methodology not only accelerates troubleshooting but also fosters reproducibility, transparency, and scalability in the deployment of LLM-based systems.

我们认为，这种结构化的方法论不仅能加速故障排除，还能促进基于大语言模型系统的部署过程中的可重复性、透明度和可扩展性。