Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models

模态成熟度指数（MMI）：评估全能模型多模态能力的基准

Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. 前沿语言模型正越来越多地被宣传为能够跨模态感知和响应的“全能系统”（omni systems）。然而，现有的评估框架几乎只关注双模态理解，通常是文本加上另一种模态。

We propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. 我们提出了模态成熟度指数（Modality Maturity Index, MMI），这是一个旨在评估大型语言模型在五种模态（文本、图像、音频、视频和文档）以及输入和输出中最多三种模态组合下的多模态能力的基准。

MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. MMI 包含 893 个问题，每个问题都经过精心设计，要求模型展示其对多种输入模态的理解，并生成包含各种输出格式的响应。这些问题被设计为自包含的，并对准确响应所需的正确模态或模态组合有明确的预期。

Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model’s MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. 每个 MMI 提示词都附有人工编写的评分标准，针对响应中预期的每种输出模态；模型的 MMI 值表示每个提示词中各模态得分的平均值。由于低分可能反映了未能生成某种模态（缺乏存在感）或未能生成正确内容，我们还引入了补充性的模态存在分数（Modality Presence Score, MPS），即针对每个提示词在预期输出模态上的 F1 分数。

Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. 通过将 MMI 应用于五个前沿多模态模型，我们发现 MPS 分数仅在 15.6（Claude Opus 4.6）到 34.9（GPT-5.4）之间。鉴于返回的模态数量过少，甚至难以进行评分，我们在等待模型改进的同时，将 MPS 作为我们的主要结果进行报告。

To assess the viability of judging output correctness with LLM judges and rubrics, we run a separate experiment with custom generation tools. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments. 为了评估使用 LLM 裁判和评分标准来判断输出正确性的可行性，我们使用自定义生成工具进行了单独的实验。在生成的资产上，我们发现应用评分标准的 LLM 裁判与不看评分标准的盲测人类标注员（直接对输出进行评分，从未见过评分标准）在 70.8% 的判断上达成了一致。