Transformers now runs llama.cpp quants

Transformers 现在支持运行 llama.cpp 量化模型

We’re adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop’s memory through the familiar transformers APIs. Pick a GGUF from the Hub, load it with from_pretrained, and start generating on your own machine. 我们正在为 transformers 添加高效运行 GGUF 模型的功能，这样你就可以通过熟悉的 transformers API 使用适合你笔记本电脑内存大小的检查点（checkpoints）。只需从 Hugging Face Hub 上挑选一个 GGUF 模型，使用 from_pretrained 加载，即可在自己的机器上开始生成。

Running AI models on your laptop has become much easier, and llama.cpp has been a big part of that. Its inference engine powers local AI tools such as Ollama, LM Studio, and Jan. Alongside projects like MLX, it has helped make local inference a practical option for everyday use. 在笔记本电脑上运行 AI 模型变得容易多了，而 llama.cpp 在其中发挥了重要作用。它的推理引擎为 Ollama、LM Studio 和 Jan 等本地 AI 工具提供了动力。与 MLX 等项目一起，它使本地推理成为了日常使用的实用选择。

GGUF, developed by the llama.cpp team, is a widely used format for local inference. The team also shares quantized checkpoints under ggml-org on the Hub. Publishers such as Unsloth, LM Studio Community, and bartowski also provide ready-to-use GGUF checkpoints in a range of quantizations, so users can pick the version that fits their machine. GGUF models have been downloaded millions of times. We want to make it easier to run these models locally with transformers, too. GGUF 由 llama.cpp 团队开发，是一种广泛用于本地推理的格式。该团队还在 Hub 的 ggml-org 下共享量化检查点。Unsloth、LM Studio Community 和 bartowski 等发布者也提供了各种量化版本的现成 GGUF 检查点，方便用户选择适合自己机器的版本。GGUF 模型已被下载了数百万次。我们也希望通过 transformers 让在本地运行这些模型变得更加简单。

Compatibility is only useful if the model is pleasant to run. To bring performance close to llama.cpp, we’re reusing its underlying ggml kernels through the kernels library, and reducing overhead in generate. Our initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture. 兼容性只有在模型运行体验良好时才有意义。为了使性能接近 llama.cpp，我们通过 kernels 库复用了其底层的 ggml 内核，并减少了生成过程中的开销。我们最初的重点是 Apple Silicon 上的本地推理，并从 Qwen3.5 架构开始。

What is the GGUF file format?

什么是 GGUF 文件格式？

GGUF packages model weights and metadata, including tokenizer information and an optional chat template, in one file. It supports different quantization levels, letting you trade some precision for a smaller memory footprint. Variants such as Q4_K_M mix tensor precisions, using mostly 4-bit weights while keeping sensitive tensors at higher precision. GGUF 将模型权重和元数据（包括分词器信息和可选的聊天模板）打包在一个文件中。它支持不同的量化级别，让你能够以牺牲少量精度为代价换取更小的内存占用。诸如 Q4_K_M 之类的变体混合了张量精度，主要使用 4-bit 权重，同时保持敏感张量处于较高精度。

We suggest starting with Q4_K_M, then trying Q5_K_M or Q6_K if you have more memory available. More aggressive quantization can help larger models fit, but the quality tradeoff depends on the model and the task. Evaluate it on the work you actually want the model to do. 我们建议从 Q4_K_M 开始，如果你有更多可用内存，可以尝试 Q5_K_M 或 Q6_K。更激进的量化可以帮助更大的模型适配内存，但质量折损取决于模型和任务本身。请在你实际需要模型完成的工作上进行评估。

Load GGUF with transformers

使用 transformers 加载 GGUF

To get started, you need: 要开始使用，你需要：

• An Apple Silicon Mac. (一台 Apple Silicon Mac。)
• A PyTorch version supported by the published ggml-quantization kernel builds, usually the two latest PyTorch releases. (已发布的 ggml-quantization 内核构建所支持的 PyTorch 版本，通常是最近的两个 PyTorch 发行版。)
• The latest version of transformers (main for now, until the next release) and a compatible version of kernels. (最新版本的 transformers（目前为 main 分支，直到下一次发布）以及兼容版本的 kernels。)

pip install -U "git+https://github.com/huggingface/transformers.git" kernels

To load a GGUF model, pass its Hub model_id and filename as gguf_file to from_pretrained. No extra configuration is needed: when the weights stay packed on Metal, transformers automatically loads the compatible ggml/Metal layer kernels and uses ggml-org/ggml-attn as the attention implementation. 要加载 GGUF 模型，请将其 Hub 的 model_id 和文件名作为 gguf_file 传递给 from_pretrained。无需额外配置：当权重保持在 Metal 上打包时，transformers 会自动加载兼容的 ggml/Metal 层内核，并使用 ggml-org/ggml-attn 作为注意力实现。

Serve GGUF with your preferred interface

使用你偏好的界面服务 GGUF

You can also use the same checkpoint with transformers serve, which exposes an OpenAI-compatible API: 你也可以使用相同的检查点通过 transformers serve 来运行，它会暴露一个兼容 OpenAI 的 API：

pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels
transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

The model argument uses <model_id>:<filename>.gguf: before the colon is the Hub repository, and after it is the file to load. This selects a specific quantization from a repository that may contain several. 模型参数使用 <model_id>:<filename>.gguf 格式：冒号前是 Hub 仓库，冒号后是要加载的文件。这可以从可能包含多个量化版本的仓库中选择特定的一个。