Run Qwen 3.8 27B Locally: Real GGUF Sizes, the KV Cache Trick, and the Template Trap
Run Qwen 3.8 27B Locally: Real GGUF Sizes, the KV Cache Trick, and the Template Trap
在本地运行 Qwen 3.8 27B：真实的 GGUF 大小、KV 缓存技巧与模板陷阱
Qwen 3.8 arrived as two different releases with two different licences, and only one of them is something you can put on a card you own. The 2.4 trillion parameter A95B opened up on 12 August under Alibaba’s own qwen3.8-max terms. The one that matters for local work is Qwen 3.8 27B, whose safetensors went up on 13 August at 08:23 UTC with an Apache 2.0 LICENSE file following the next morning. Both dates are off the Hugging Face commit log, not a launch post. Qwen 3.8 以两个不同的版本和两种不同的许可证发布，其中只有一个是你可以在自己的显卡上运行的。2.4 万亿参数的 A95B 于 8 月 12 日在阿里巴巴自有的 qwen3.8-max 条款下开放。对于本地工作而言，重要的是 Qwen 3.8 27B，其 safetensors 文件于 8 月 13 日 08:23 UTC 上传，Apache 2.0 许可证文件则于次日上午发布。这两个日期均来自 Hugging Face 的提交日志，而非发布公告。
Here is the practical picture: what it needs, why its long context is unusually cheap, and the one setting that makes people think they downloaded a broken quant. The shape of the model decides everything: 27B dense parameters across 64 layers, hidden size 5120. The interesting part is in config.json, where layer_types reads 48 linear attention layers and 16 full attention layers, alternating three to one (full_attention_interval: 4). Only those 16 layers keep a KV cache. The rest of the shape: 24 attention heads with head_dim 256 and 4 KV heads, a 248,320 token vocabulary, and max_position_embeddings of 262,144. It is a native vision language model, so images and video go in without a wrapper, and the ggml-org pack also ships a multi token prediction head as a separate file. 以下是实际情况：它需要什么、为什么它的长上下文成本异常低，以及那个让人们误以为下载了损坏量化版本的设置。模型的架构决定了一切：27B 稠密参数分布在 64 层中，隐藏层大小为 5120。有趣的部分在于 config.json，其中 layer_types 显示为 48 层线性注意力层和 16 层全注意力层，以三比一的比例交替（full_attention_interval: 4）。只有这 16 层保留 KV 缓存。其余架构包括：24 个注意力头（head_dim 为 256）和 4 个 KV 头，词表大小为 248,320，最大位置编码（max_position_embeddings）为 262,144。它是一个原生的视觉语言模型，因此图像和视频无需封装即可输入，ggml-org 压缩包还提供了一个多 token 预测头作为单独文件。
The numbers: Sizes below are the file sizes Hugging Face reports for unsloth/Qwen3.8-27B-GGUF, read on 14 August 2026. Packs differ by a few hundred megabytes, so check the repo you actually pull from. lmstudio-community has Q4_K_M at 16.8 GB and ggml-org at 19.0 GB for the same nominal quant. 数据：以下大小为 Hugging Face 上 unsloth/Qwen3.8-27B-GGUF 的文件大小，读取于 2026 年 8 月 14 日。不同压缩包之间会有几百兆字节的差异，因此请检查你实际拉取的仓库。对于相同的标称量化版本，lmstudio-community 的 Q4_K_M 为 16.8 GB，而 ggml-org 为 19.0 GB。
| Quant | Size on disk | Realistic home |
|---|---|---|
| UD-IQ2_XXS | 9.0 GB | 12 GB cards, visible quality cost |
| UD-Q2_K_XL | 10.7 GB | 12 GB cards, almost no context left |
| UD-Q3_K_XL | 13.4 GB | 16 GB cards |
| Q3_K_M | 13.8 GB | 16 GB cards |
| IQ4_XS | 15.7 GB | largest quant that stays whole on 16 GB |
| Q4_K_M (sweet spot) | 17.1 GB | 24 GB cards |
| Q5_K_M | 19.8 GB | 24 GB, less context headroom |
| Q6_K | 22.9 GB | 24 GB barely, or 32 GB |
| Q8_0 | 29.0 GB | 32 GB or a two card split |
| BF16 (from ggml-org) | 53.8 GB | server cards, or CPU and patience |
| mmproj | 0.9 GB | the vision encoder, separate file |
| 量化版本 | 磁盘大小 | 实际家用建议 |
|---|---|---|
| UD-IQ2_XXS | 9.0 GB | 12 GB 显卡，画质有明显损失 |
| UD-Q2_K_XL | 10.7 GB | 12 GB 显卡，几乎没有上下文空间 |
| UD-Q3_K_XL | 13.4 GB | 16 GB 显卡 |
| Q3_K_M | 13.8 GB | 16 GB 显卡 |
| IQ4_XS | 15.7 GB | 16 GB 显卡上能完整运行的最大量化 |
| Q4_K_M (最佳平衡) | 17.1 GB | 24 GB 显卡 |
| Q5_K_M | 19.8 GB | 24 GB 显卡，上下文余量较少 |
| Q6_K | 22.9 GB | 24 GB 显卡勉强运行，或 32 GB |
| Q8_0 | 29.0 GB | 32 GB 显卡或双卡拆分 |
| BF16 (来自 ggml-org) | 53.8 GB | 服务器显卡，或 CPU 加耐心 |
| mmproj | 0.9 GB | 视觉编码器，独立文件 |
Machine classes, honestly: 24 GB GPU: Q4_K_M whole, with real context headroom. This is the card the model was sized for. 16 GB GPU: IQ4_XS fits whole; Q4_K_M works with a few layers offloaded and costs you speed. 12 GB GPU: only the 2-bit quants, and you will feel it. A 3060 runs it, slowly. Apple Silicon: 32 GB unified memory is comfortable at Q4_K_M, 24 GB works if nothing else is open. 老实说，机器配置建议如下：24 GB 显存 GPU：可完整运行 Q4_K_M，且有充足的上下文余量。这是该模型设计的基准显卡。16 GB 显存 GPU：可完整运行 IQ4_XS；Q4_K_M 需要卸载部分层到内存，会牺牲速度。12 GB 显存 GPU：只能运行 2-bit 量化，且会有明显卡顿。3060 可以运行，但很慢。Apple Silicon：32 GB 统一内存运行 Q4_K_M 很舒适，24 GB 在没有其他程序运行时也可以运行。
Where the hybrid layout pays off: Every full attention layer stores a KV cache that grows with the context. At fp16 one token costs 2 x 4 heads x 256 dim x 2 bytes = 4 KB per layer. A conventional 64 layer model pays that on all 64 layers, which is 256 KB per token. Qwen 3.8 27B pays it on 16 layers, so 64 KB per token. 混合架构的优势：每个全注意力层都会存储随上下文增长的 KV 缓存。在 fp16 下，一个 token 的成本为 2 x 4 个头 x 256 维度 x 2 字节 = 每层 4 KB。传统的 64 层模型在所有 64 层上都要消耗，即每个 token 256 KB。Qwen 3.8 27B 仅在 16 层上消耗，因此每个 token 仅需 64 KB。
| Context | KV cache | Q4_K_M total |
|---|---|---|
| 8K | 0.5 GB | 17.6 GB |
| 32K | 2.0 GB | 19.1 GB |
| 128K | 8.0 GB | 25.1 GB |
| 262K | 16.4 GB | 33.5 GB |
| 上下文 | KV 缓存 | Q4_K_M 总计 |
|---|---|---|
| 8K | 0.5 GB | 17.6 GB |
| 32K | 2.0 GB | 19.1 GB |
| 128K | 8.0 GB | 25.1 GB |
| 262K | 16.4 GB | 33.5 GB |
That is the difference between a long window on the spec sheet and one you actually turn on. A 128K session that would need 32 GB of cache on a dense model needs 8 GB here. With a current llama.cpp: 这就是规格表上的长窗口与你实际能开启的长窗口之间的区别。在稠密模型上需要 32 GB 缓存的 128K 会话，在这里只需要 8 GB。使用当前的 llama.cpp：
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
--jinja \
-ngl 99 \
-c 32768The template trap: That —jinja flag is not optional, and it is the single biggest source of “this quant is broken” reports. Qwen 3.8 ships its own chat template. Load the model without it and the model has no reliable marker for where your turn ends and its answer begins. The two failure modes both look like a bad conversion: either it rambles past the stop token, or it answers in a clipped voice and loses the conversation between turns. There is a second, sharper version of this. The official template wraps each assistant turn in a think block even when the reasoning is empty, then opens another one when generation starts. Across several turns those nest and the history gets truncated. Several GGUF packs already ship a corrected chat_template.jinja. If you converted the weights yourself, swap the template before you blame the quantization. 模板陷阱：—jinja 标志不是可选的，它是“这个量化版本坏了”这类反馈的最大来源。Qwen 3.8 自带聊天模板。如果不加载它，模型就无法准确识别你的对话在哪里结束、它的回答从哪里开始。两种失败模式看起来都像是转换错误：要么它在停止符之后继续胡言乱语，要么它回答得断断续续并丢失对话历史。还有第二个更严重的问题：官方模板会将每个助手回复包裹在 think 块中（即使推理内容为空），然后在生成开始时又打开一个新的。经过几轮对话后，这些嵌套会导致历史记录被截断。一些 GGUF 压缩包已经提供了修正后的 chat_template.jinja。如果你是自己转换的权重，请先更换模板，再质疑量化质量。
Vision needs the second file: The vision encoder is not inside the language GGUF. Download the mmproj file, about 0.9 GB, and load it alongside: 视觉功能需要第二个文件：视觉编码器不在语言 GGUF 文件中。下载约 0.9 GB 的 mmproj 文件，并同时加载：
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
--mmproj mmproj-F16.gguf \
--jinja -ngl 99Skip it and you have a strong text model that will politely tell you it cannot see the image you just pasted. 如果跳过这一步，你将得到一个强大的文本模型，它会礼貌地告诉你它无法看到你刚刚粘贴的图像。
Thinking is a dial, not a switch: The model reasons before answering by default. Lower reasoning_effort from the default to medium or low for quick answers, or pass chat_template_kwargs with enable_thinking: false to skip the reasoning pass entirely. Locally the cost of thinking is not money, it is your own time watching tokens arrive, which is the more annoying currency. 思考是一个刻度盘，而非开关：模型默认会在回答前进行推理。若需快速回答，可将 reasoning_effort 从默认值调低至 medium 或 low，或者通过 chat_template_kwargs 传入 enable_thinking: false 来完全跳过推理过程。在本地，思考的成本不是金钱，而是你等待 token 生成的时间，这才是更令人恼火的“货币”。
The no-terminal route: If you would rather click than type flags, Locally Uncensored (open source, AGPL) wraps this: install, open the Model Manager, paste a Qwen 3.8 27B GGUF repo, pick the quant that fits your card, chat. It carries the llama.cpp engine, handles the offload split and the template, and keeps everything on your machine with no account and no telemetry. 非命令行方案：如果你更喜欢点击而非输入标志，Locally Uncensored（开源，AGPL 协议）封装了这些功能：安装后打开模型管理器，粘贴 Qwen 3.8 27B GGUF 仓库地址，选择适合你显卡的量化版本，即可开始聊天。它内置了 llama.cpp 引擎，处理了卸载拆分和模板问题，并将所有内容保留在你的机器上，无需账户，无遥测。
And the 2.4T reality check: The big Qwen 3.8, Qwen/Qwen3.8-2.4T-A95B, has open weights but not an open licence. Hugging Face reports license: other with license_name: qwen3.8-max, so read the terms before you build a product on it, and do not repeat the “Apache” line that has been going around. At 2.4 trillion total parameters with 95 billion active it is a data center model regardless. The split that works: run the 27B at home for anything private, offline, or repetitive, and reach the A95B through a hosted API when a task genuinely needs that scale. It is on DeepInfra, and both it and the 27B class of models sit in LU Labs Cloud if you want them next to each other in one picker. 关于 2.4T 的现实核查：大型 Qwen 3.8（Qwen/Qwen3.8-2.4T-A95B）拥有开放权重，但没有开放许可证。Hugging Face 显示其许可证为 other，名称为 qwen3.8-max，因此在基于它构建产品前请阅读条款，不要重复流传的“Apache”说法。它拥有 2.4 万亿总参数和 950 亿激活参数，无论如何都是一个数据中心级模型。可行的方案是：在本地运行 27B 版本处理私密、离线或重复性任务，当任务确实需要该规模时，通过托管 API 调用 A95B。它已在 DeepInfra 上线，如果你希望在同一个选择器中同时使用它和 27B 模型，它们也都位于 LU Labs Cloud 中。
FAQ: Is 27B enough? For local work it is the interesting size: big enough for real coding and agent loops, small enough to sit on one consumer card. The hybrid attention means long context does not price you out. 常见问题：27B 够用吗？对于本地工作，这是一个有趣的尺寸：足够进行实际编程和智能体循环，又足够小以至于能放在一张消费级显卡上。混合注意力机制意味着长上下文不会让你因成本过高而望而却步。