Welcome Inkling by Thinking Machines

查看原文 / View Original 编辑 / Edit

Welcome Inkling by Thinking Machines

欢迎使用 Thinking Machines 推出的 Inkling

Inkling is a large (1T params!) open model to natively accept image, text, and audio inputs. TLDR; Inkling by Thinking Machines is out on Hugging Face. Inkling is a huge multimodal LLM that understands all modalities (image, audio, text), has agentic capabilities, and supports 1M context. It comes in full BF16 and a well-calibrated NVFP4 variant, and includes speculative MTP layers for faster inference. There’s day-0 support in transformers, SGLang, and llama.cpp.

Inkling 是一个大型（1 万亿参数！）开源模型，能够原生接收图像、文本和音频输入。简而言之，Thinking Machines 推出的 Inkling 现已在 Hugging Face 上发布。Inkling 是一个庞大的多模态大语言模型（LLM），能够理解所有模态（图像、音频、文本），具备智能体（Agentic）能力，并支持 100 万上下文窗口。它提供完整的 BF16 版本和经过良好校准的 NVFP4 变体，并包含用于加速推理的推测性 MTP 层。目前 transformers、SGLang 和 llama.cpp 已实现首日支持。

What makes Inkling special? Inkling is the first large open model with ~1T parameters and 1M context window to natively receive image, text, and audio inputs, trained on 45 trillion tokens of text, images, audio and video. It’s focused on reasoning across modalities such as audio, images, and text; and is intended for domain adaptation via fine-tuning. We’ve tinkered with this model to build some demos and explore the architecture, and we think it’s great for building a new wave of multimodal reasoning apps.

Inkling 有何特别之处？Inkling 是首个拥有约 1 万亿参数和 100 万上下文窗口的大型开源模型，能够原生接收图像、文本和音频输入，并基于 45 万亿个文本、图像、音频和视频 Token 进行训练。它专注于跨模态（如音频、图像和文本）的推理，旨在通过微调实现领域适配。我们已经尝试使用该模型构建了一些演示并探索了其架构，我们认为它非常适合构建新一代的多模态推理应用。

Overall Capabilities and Architecture 总体能力与架构

Inkling is a decoder-only multimodal Mixture-of-Experts model with 975B total and 41B active parameters. There are a lot of things going on, so let’s break each part down: Inkling 是一个仅解码器（Decoder-only）的多模态混合专家（MoE）模型，总参数量为 9750 亿，激活参数量为 410 亿。由于涉及的技术点较多，我们逐一拆解：

  • Decoder-only: This means that the architecture supports causal autoregressive generation, like in most state-of-the-art LLMs.

  • 仅解码器架构： 这意味着该架构支持因果自回归生成，与大多数最先进的 LLM 一致。

  • Multimodal: The model can ingest text, audio, and images.

  • 多模态： 该模型可以摄取文本、音频和图像。

  • Mixture of Experts (MoE): The feed forward networks inside each layer are sparse, achieving faster inference because only 41B parameters are active at any given time. The model has 256 experts, as we’ll see later.

  • 混合专家模型 (MoE)： 每个层内的前馈网络是稀疏的，由于在任何给定时间仅激活 410 亿个参数，从而实现了更快的推理速度。正如稍后将看到的，该模型拥有 256 个专家。

Relative attention: Instead of RoPE, which is the usual method to inject positional information in transformers models, Inkling uses relative attention to encode position information. Each attention layer learns position directly in the attention logits. Aside from key-query-values, there’s a fourth projection producing a per-token, per-head relative feature R. This projection tensor is then tweaked with distance information (distance between the key and the query vector) and propagated into the attention module.

相对注意力机制：Inkling 没有使用 Transformer 模型中常见的 RoPE（旋转位置编码）来注入位置信息，而是使用相对注意力机制来编码位置信息。每个注意力层直接在注意力 Logits 中学习位置。除了键（Key）、查询（Query）和值（Value）之外，还有一个第四投影，用于生成每个 Token、每个头部的相对特征 R。该投影张量随后会根据距离信息（键向量与查询向量之间的距离）进行调整，并传播到注意力模块中。

Hybrid attention: The decoder layers alternate between global attention (attending to the full context length at once) and sliding window attention (attending to a fixed context window in a sliding fashion). The architecture has a pattern of 5:1 sliding window to global attention layers. This hybrid attention scheme provides efficiency in computation. The final layer uses global attention to help build feature-rich representations.

混合注意力机制：解码器层在全局注意力（一次性关注整个上下文长度）和滑动窗口注意力（以滑动方式关注固定的上下文窗口）之间交替。该架构采用了 5:1 的滑动窗口与全局注意力层比例。这种混合注意力方案提高了计算效率。最后一层使用全局注意力，有助于构建特征丰富的表示。

Short convolution: The model uses a distinctive short 1D convolution, or SConv over the hidden states. SConv reads the current token and the previous W-1 hidden states, with W being the sliding window size. The intuition here is that SConv helps with local attention while freeing the attention and MoE modules from local representations.

短卷积：该模型在隐藏状态上使用了一种独特的短一维卷积（SConv）。SConv 读取当前 Token 和前 W-1 个隐藏状态（W 为滑动窗口大小）。其直觉在于，SConv 有助于处理局部注意力，同时将注意力模块和 MoE 模块从局部表示的任务中解放出来。

MoE with shared experts sink: In Inkling, the router scores both routed experts and shared experts. Top-k selection is performed over 6 experts, plus 2 shared experts always active.

带有共享专家池的 MoE：在 Inkling 中，路由器会对路由专家和共享专家进行评分。系统会在 6 个专家中执行 Top-k 选择，外加 2 个始终处于激活状态的共享专家。

Vision understanding: The model includes a simple hierarchical MLP patchifier consisting of several linear layers. Each layer merges pixels progressively, until the final layer produces one embedding per patch。 视觉理解：该模型包含一个简单的分层 MLP 分块器（Patchifier），由多个线性层组成。每一层逐步合并像素，直到最后一层为每个分块生成一个嵌入（Embedding）。

Audio understanding: The architecture employs a discretized mel spectrogram, where each of the audio chunks (of 100 ms) are converted to the mel scale and then classified into the exact mel spectrogram bin. 音频理解：该架构采用离散化的梅尔频谱图（Mel Spectrogram），其中每个音频块（100 毫秒）被转换为梅尔刻度，然后分类到精确的梅尔频谱图箱中。

Inference Support 推理支持

Inkling comes with day-0 transformers support and is supported in major inference engines like SGLang and vLLM. This model is huge. The bf16 checkpoint requires 2 TB of VRAM, while the nvfp4 version requires 600 GB of VRAM. You can try the model through serverless inference routers like Inference Providers, or use ggml quants for local deployment with llama.cpp. Inkling 提供首日 transformers 支持，并受 SGLang 和 vLLM 等主流推理引擎支持。该模型非常庞大。bf16 检查点需要 2 TB 的显存，而 nvfp4 版本需要 600 GB 的显存。你可以通过 Inference Providers 等无服务器推理路由器尝试该模型，或者使用 ggml 量化版本通过 llama.cpp 进行本地部署。

Transformers Transformers

The easiest way to infer with transformers directly is to use the any-to-any pipeline. You can use either the 16 bit “thinkingmachines/Inkling” on Hopper or later GPUs, or the quantized NVFP4 checkpoint “thinkingmachines/Inkling-NVFP4” on Blackwell Nvidia GPUs. Make sure to have the latest version of transformers (5.14.0 was released today) (pip install -U transformers). 直接使用 transformers 进行推理的最简单方法是使用 any-to-any 流水线。你可以在 Hopper 或更新的 GPU 上使用 16 位的 “thinkingmachines/Inkling”，或者在 Blackwell Nvidia GPU 上使用量化的 NVFP4 检查点 “thinkingmachines/Inkling-NVFP4”。请确保安装了最新版本的 transformers（今天发布的 5.14.0 版本）（pip install -U transformers）。

← 返回新闻列表 / Back to news list