Token Signatures of Code: Comparing Coding Behaviors Across Large Language Models
Token Signatures of Code: Comparing Coding Behaviors Across Large Language Models
代码的 Token 特征：比较大语言模型间的编码行为
Abstract: The evaluation of large language models (LLMs) on coding tasks has primarily focused on performance metrics such as pass@k. As LLMs continue to advance, many models now meet baseline performance requirements, reducing the discriminative power of performance-based evaluation alone. Yet a key question remains largely unexplored: how do LLMs differ in their coding behavior?
摘要： 目前对大语言模型（LLM）在编码任务上的评估主要集中在 pass@k 等性能指标上。随着大语言模型的不断进步，许多模型已经达到了基准性能要求，这使得仅依靠性能评估的区分能力有所下降。然而，一个关键问题仍未得到充分探讨：大语言模型在编码行为上究竟有何不同？
We propose CLIC (Code Learning for Identification and Comparison), a visual analytics approach that characterizes LLM coding behavior through token-frequency analysis. CLIC represents each code sample as a feature vector of token frequencies and trains an interpretable decision tree to separate two LLMs’ code sets.
我们提出了 CLIC（代码识别与比较学习），这是一种通过 Token 频率分析来刻画大语言模型编码行为的可视化分析方法。CLIC 将每个代码样本表示为 Token 频率的特征向量，并训练一个可解释的决策树来区分两个大语言模型的代码集。
Beyond classification accuracy, we define two new metrics: robustness, which measures whether the two LLMs remain distinguishable as their most-discriminative tokens are progressively removed, and concentration, which measures whether the difference is driven by a few dominant tokens or spread across many.
除了分类准确率之外，我们还定义了两个新指标：鲁棒性（Robustness），用于衡量在逐步移除最具区分度的 Token 后，两个大语言模型是否仍可被区分；以及集中度（Concentration），用于衡量这种差异是由少数主导 Token 驱动的，还是分散在许多 Token 中。
Interpreting numerous pairwise comparisons (across LLM pairs, tasks, and tokenization levels) and tracing the full analytical chain form an inherently multi-scale, hypothesis-driven exploration task. We therefore develop an interactive visual analytics system to navigate the comparison landscape, identify pairs of interest, and drill down into discriminative tokens and their code contexts.
解读大量的成对比较（跨越不同的模型对、任务和 Token 化级别）并追踪完整的分析链，构成了一项本质上多尺度、以假设为驱动的探索任务。因此，我们开发了一个交互式可视化分析系统，用于导航比较全景、识别感兴趣的模型对，并深入挖掘具有区分度的 Token 及其代码上下文。
Case studies comparing 10 LLMs across 22 Kaggle ML tasks reveal actionable insights for LLM selection and prompt engineering.
通过对 22 个 Kaggle 机器学习任务中 10 个大语言模型的案例研究，我们揭示了对于大语言模型选择和提示工程（Prompt Engineering）具有指导意义的见解。