NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction
NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction
NVIDIA Kumo Tabular 为表格预测树立了准确性与效率的新标杆
Highlights (TL;DR) 亮点（摘要）
NVIDIA Kumo Tabular, part of the NVIDIA Kumo Structured model collection, is an open foundation model for tabular data now available on Hugging Face. Given a table of labeled rows, it predicts the labels of new rows in a single forward pass, with no training, no tuning, and no feature engineering, for both classification and regression. It was pretrained only on artificial data, comes in three sizes (28M to 215M parameters), runs through our open-source library, and is released under the OpenMDW-1.1 license for commercial use. It ranks first on the four benchmarks TabArena, BeyondArena, TALENT and ScoringBench. NVIDIA Kumo Tabular 是 NVIDIA Kumo 结构化模型系列的一部分，是一个现已在 Hugging Face 上发布的表格数据开源基础模型。给定包含标注行的表格，它可以在单次前向传播中预测新行的标签，无需训练、无需调优、无需特征工程，适用于分类和回归任务。该模型仅在人工数据上进行预训练，提供三种尺寸（2800 万至 2.15 亿参数），通过我们的开源库运行，并以 OpenMDW-1.1 许可证发布以供商业使用。它在 TabArena、BeyondArena、TALENT 和 ScoringBench 四项基准测试中均排名第一。
The Shift to Tabular Foundation Models 向表格基础模型的转变
Tabular data is the backbone of enterprise machine learning. Customer records, transactions, sensor logs, claims, and orders all live in tables, and predicting churn, default, demand, or price from them is among the most common machine learning tasks in industry. For two decades, this work has been done with gradient-boosted trees, and it has worked well. But the lifecycle around those models has barely changed. Every new question means collecting labels, engineering features, searching hyperparameters, validating, and deploying a model that knows nothing about tables in general and learns each task from scratch. 表格数据是企业机器学习的基石。客户记录、交易、传感器日志、索赔和订单都以表格形式存在，从中预测流失率、违约、需求或价格是工业界最常见的机器学习任务之一。二十年来，这项工作一直由梯度提升树（Gradient-boosted trees）完成，且效果良好。但这些模型的生命周期几乎没有改变。每一个新问题都意味着需要收集标签、进行特征工程、搜索超参数、验证并部署一个对表格一无所知且必须从零开始学习每个任务的模型。
Large Language Models showed a different way of working with new tasks. Given a few examples in the prompt, a pretrained model solves the task without updating a single weight. This is in-context learning, and it applies to tables just as well as to text: a model pretrained on millions of tables can read a labeled table as its context and predict the labels of new rows directly. Today, we are releasing NVIDIA Kumo Tabular (GitHub, HuggingFace), an open foundation model for tabular classification and regression. Given a table with labeled rows and the rows you want predictions for, Kumo Tabular returns class probabilities or numeric predictions in a single forward pass. 大型语言模型展示了一种处理新任务的不同方式。在提示词中给出几个示例，预训练模型无需更新任何权重即可解决任务。这就是上下文学习（In-context learning），它不仅适用于文本，也同样适用于表格：一个在数百万张表格上预训练的模型可以读取带标签的表格作为上下文，并直接预测新行的标签。今天，我们发布了 NVIDIA Kumo Tabular (GitHub, HuggingFace)，这是一个用于表格分类和回归的开源基础模型。给定一个包含标注行的表格以及需要预测的行，Kumo Tabular 可以在单次前向传播中返回类别概率或数值预测。
How Kumo Tabular Works Kumo Tabular 的工作原理
Kumo Tabular is a Transformer built around the structure of a table, utilizing column, row and in-context attention as introduced in TabICL and TabPFN. To predict a label it has to do three things: (1) understand what each value means within its column, (2) understand how the columns of a row interact, and (3) relate the context rows with existing labels to the query rows with unknown labels. Kumo Tabular 是一个围绕表格结构构建的 Transformer，利用了 TabICL 和 TabPFN 中引入的列注意力、行注意力和上下文注意力机制。为了预测标签，它必须完成三件事：(1) 理解每个值在其列中的含义；(2) 理解一行中各列如何相互作用；(3) 将带有现有标签的上下文行与标签未知的查询行关联起来。
Cell Embedding: A group of cells becomes a token. Numerical and categorical values pass through Fourier features, sines and cosines of learned frequencies, with separate weights for each type. Missing values need no imputation and are treated specially. Finally, every token in the context receives a label embedding. 单元格嵌入 (Cell Embedding)： 一组单元格被转化为一个 token。数值型和类别型值通过傅里叶特征（学习频率的正弦和余弦函数）进行处理，并为每种类型分配独立的权重。缺失值无需填充，而是进行特殊处理。最后，上下文中的每个 token 都会接收一个标签嵌入。
Row Embedding: We then turn each row into an embedding by alternating two kinds of attention multiple times. Column attention looks down a single column and learns what a value means in the distribution of its column, e.g., whether a 42 is typical or extreme, via induced self-attention. Its cost therefore grows linearly with the number of rows. Row attention looks across the tokens of a single row and learns how features interact, with rotary positions to tell columns apart. Four learnable [CLS] tokens join each row and act as the final readout of a row. After this row compression, the cost of the final stage no longer depends on the number of columns. 行嵌入 (Row Embedding)： 我们通过多次交替使用两种注意力机制将每一行转化为嵌入。列注意力机制纵向观察单列，通过诱导自注意力（induced self-attention）学习某个值在其列分布中的含义，例如 42 是典型值还是极端值。因此，其计算成本随行数线性增长。行注意力机制横向观察单行中的 token，并学习特征如何相互作用，同时使用旋转位置编码（rotary positions）来区分各列。四个可学习的 [CLS] token 加入每一行，作为该行的最终读出结果。经过这种行压缩后，最后阶段的成本不再取决于列数。
In-context Learning: A final Transformer operates on the row embeddings. Context rows attend to each other, while query rows attend to context rows only. Each prediction therefore depends only on the context and on the row itself, not on which other rows are scored alongside it. Because the context never looks at the queries, its keys and values are computed once and can be reused for follow-up predictions. Query rows utilize Test-GQA, which shrinks the cache that every prediction reads. A head turns each query row into class probabilities for classification and 999 quantiles for regression, from which a point prediction and an uncertainty estimate follow. 上下文学习 (In-context Learning)： 最终的 Transformer 对行嵌入进行操作。上下文行之间相互关注，而查询行仅关注上下文行。因此，每个预测仅取决于上下文和行本身，而不取决于同时进行评分的其他行。由于上下文从不查看查询，其键（keys）和值（values）只需计算一次，即可重复用于后续预测。查询行利用 Test-GQA，这缩小了每个预测所读取的缓存。一个头部（head）将每个查询行转化为分类任务的类别概率或回归任务的 999 个分位数，从而得出点预测和不确定性估计。
Length-aware Attention Temperature: Softmax attention spreads out as the number of keys grows. Attention that is sharp over a few hundred rows can dissolve over tens of thousands, which is exactly the situation when a table at inference is much larger than a typical training table. Kumo Tabular therefore scales every query by a temperature that grows with the logarithm of the number of keys, with a coefficient learned separately for each attention head. The result is attention that stays sharp as tables grow longer or wider. 长度感知注意力温度 (Length-aware Attention Temperature)： 随着键（keys）数量的增加，Softmax 注意力会变得分散。在几百行数据上表现敏锐的注意力机制在处理数万行数据时可能会失效，而这正是推理时的表格远大于典型训练表格时的情况。因此，Kumo Tabular 通过一个随键数量对数增长的温度系数来缩放每个查询，该系数为每个注意力头单独学习。其结果是，即使表格变得更长或更宽，注意力机制依然保持敏锐。
How Kumo Tabular was Built Kumo Tabular 的构建方式
Kumo Tabular is pretrained entirely on artificial tables. Each training table is sampled from a Structural Causal Model (SCM) in the six steps shown below: We first draw a configuration for the whole table, from its size and task to its mechanisms and missingness. A random causal graph then links hidden variables, evaluated from root to leaf via randomly drawn functions at every node (e.g., linear maps, small neural networks, trees or Gaussian processes). Some nodes become numerical or categorical columns, one becomes the target, and t… Kumo Tabular 完全在人工表格上进行预训练。每个训练表格均通过结构因果模型 (SCM) 按以下六个步骤采样得出：我们首先为整个表格绘制配置，从其大小和任务到其机制和缺失情况。随后，一个随机因果图连接隐藏变量，并通过每个节点上随机抽取的函数（例如线性映射、小型神经网络、树或高斯过程）从根到叶进行评估。一些节点成为数值型或类别型列，其中一个成为目标变量，且 t…