Qwen3 4B vs 0.6B on a 6 GB GPU: Speed, Memory, and Answer Quality
Qwen3 4B vs 0.6B on a 6 GB GPU: Speed, Memory, and Answer Quality
For my personal retrieval-augmented generation (RAG) study, I needed a local model to answer questions from documents. My laptop has an RTX 3060 Laptop GPU with 6 GiB of VRAM, so memory use mattered alongside speed. I compared qwen3:4b and qwen3:0.6b using Ollama. The smaller model generated tokens roughly three times faster. The larger model passed all six document checks, while the smaller one passed four. That gave me a practical starting point for the lab. Here is what I measured and what the failures showed.
为了进行个人检索增强生成（RAG）研究，我需要一个本地模型来回答文档中的问题。我的笔记本电脑配备了拥有 6 GiB 显存的 RTX 3060 移动版 GPU，因此内存占用和速度同样重要。我使用 Ollama 对比了 qwen3:4b 和 qwen3:0.6b。较小的模型生成 token 的速度大约快三倍。较大的模型通过了全部六项文档检查，而较小的模型通过了四项。这为我的实验提供了一个实用的起点。以下是我测量的数据以及失败案例所揭示的内容。
The setup: The experiment ran on October 20, 2026, with a Ryzen 7 6800H, approximately 15 GiB of usable system RAM, and Ollama 0.40.6. Both models used Q4_K_M quantization, a 4,096-token context, temperature 0, seed 42, and thinking enabled. I loaded one model at a time and warmed it up before measuring three short prompts and three document-heavy prompts. The short workload requested a long list of document-organization tips. The document workload supplied 60 synthetic workspace records. Both workloads reached a 256-token output cap, so they measured generation throughput rather than completed answers.
实验设置：实验于 2026 年 10 月 20 日进行，硬件环境为 Ryzen 7 6800H，约 15 GiB 可用系统内存，软件环境为 Ollama 0.40.6。两个模型均使用 Q4_K_M 量化、4,096 token 上下文、温度 0、随机种子 42，并启用了思考模式。我每次加载一个模型，并在测量三个短提示词和三个文档密集型提示词之前对其进行预热。短工作负载要求列出长串文档整理建议。文档工作负载提供了 60 条合成的工作区记录。两个工作负载均达到 256 token 的输出上限，因此测量的是生成吞吐量而非完整答案。
Separately, six synthetic document questions tested lookup, exceptions, policy updates, multiple conditions, arithmetic, and missing information. Each question ran once per model with a 2,048-token output budget, including reasoning. Scoring compared parsed JSON against the expected answer; Markdown code fences were accepted. No schema-constrained decoding was used.
此外，六个合成文档问题测试了查找、异常、策略更新、多重条件、算术和缺失信息。每个问题在每个模型上运行一次，输出预算为 2,048 token（包含推理过程）。评分是将解析后的 JSON 与预期答案进行对比；接受 Markdown 代码块。未使用模式约束解码。
The measured tradeoff:
| Measurement | Qwen3 4B | Qwen3 0.6B |
|---|---|---|
| Short-prompt generation | 93.4 tokens/s | 302.1 tokens/s |
| Document-prompt generation | 83.9 tokens/s | 234.8 tokens/s |
| Peak sampled whole-GPU memory | 3.12 GiB | 1.04 GiB |
| Document checks passed | 6/6 | 4/6 |
测量权衡：
| 测量指标 | Qwen3 4B | Qwen3 0.6B |
|---|---|---|
| 短提示词生成速度 | 93.4 tokens/s | 302.1 tokens/s |
| 文档提示词生成速度 | 83.9 tokens/s | 234.8 tokens/s |
| 采样全 GPU 峰值内存 | 3.12 GiB | 1.04 GiB |
| 通过的文档检查数 | 6/6 | 4/6 |
Generation rates are medians of three warm requests per workload and include both reasoning and final-answer tokens. They do not describe how quickly a complete answer appears. Both models reported their full model allocation in VRAM at the tested context. The memory figures above are sampled whole-GPU peaks, including other GPU activity; they are not model file sizes. The full report contains every measured request and document answer.
生成速率是每个工作负载三次预热请求的中位数，包含推理和最终答案的 token。它们并不代表完整答案出现的快慢。两个模型在测试的上下文环境下均报告了其在显存中的完整模型分配情况。上述内存数据是采样得到的全 GPU 峰值，包含其他 GPU 活动；它们并非模型文件大小。完整报告包含了每一次测量的请求和文档答案。
What the smaller model got wrong: The first failure involved combining requirements. A pilot required an EU region, enabled backups, and at least 100 active users. Only Aster satisfied all three conditions. Expected answer, also returned by 4B: {“workspaces”: [“Aster”]}. The 0.6B model returned: {“workspaces”: [“Aster”, “Cedar”, “Dune”]}. Cedar had backups disabled. Dune had only 60 active users. Both names were present in the supplied documents, but neither met every requirement. This was an answer-correctness failure.
较小模型的错误之处：第一次失败涉及组合条件。一个试点项目要求位于欧盟地区、启用备份且至少有 100 名活跃用户。只有 Aster 满足所有三个条件。预期答案（4B 模型也给出了该答案）为：{“workspaces”: [“Aster”]}。0.6B 模型返回了：{“workspaces”: [“Aster”, “Cedar”, “Dune”]}。Cedar 未启用备份，Dune 只有 60 名活跃用户。这两个名称都出现在提供的文档中，但都不满足所有要求。这是一个答案正确性方面的失败。
The second failure involved missing information. The documents mentioned phone support but specified no guaranteed response time. The requested answer was: {“hours”: null, “sources”: []}. The 4B model returned that structure. The 0.6B model returned: {“hours”: [null, null], “sources”: [null, null]}. This was an output-format failure: arrays appeared where a scalar null and an empty source array were required. It did not invent a response-time guarantee. That distinction matters when diagnosing what an application needs to validate.
第二次失败涉及缺失信息。文档提到了电话支持，但未指定保证响应时间。要求的答案是：{“hours”: null, “sources”: []}。4B 模型返回了该结构。0.6B 模型返回了：{“hours”: [null, null], “sources”: [null, null]}。这是一个输出格式失败：在需要标量 null 和空源数组的地方出现了数组。它没有编造响应时间保证。在诊断应用程序需要验证的内容时，这种区别很重要。
A thinking-mode mismatch changed the protocol: Preliminary attempts tried to establish a shared non-thinking baseline. The installed 4B tag continued emitting reasoning, exhausting the short output budget before some answers completed. Its metadata advertised only thinking enabled, while the installed 0.6B tag advertised both settings. I excluded those attempts from the final comparison. The recorded run explicitly enabled thinking for both models and increased the document-check budget. All six document answers completed for each model without truncation. This observation applies to the installed tags and runtime used here. It does not establish that every Qwen3 4B release behaves the same way. The repository preserves the excluded results and their explanations.
思考模式的不匹配改变了协议：初步尝试试图建立一个共享的非思考基准。安装的 4B 标签持续输出推理过程，在部分答案完成前就耗尽了短输出预算。其元数据仅显示启用了思考，而安装的 0.6B 标签则显示两种设置均已启用。我将这些尝试排除在最终对比之外。记录的运行过程明确为两个模型都启用了思考，并增加了文档检查预算。每个模型的全部六个文档答案均在未截断的情况下完成。此观察结果仅适用于此处使用的安装标签和运行时环境。它不能证明每个 Qwen3 4B 版本表现都相同。存储库中保留了被排除的结果及其解释。
Run the comparison yourself: The repository includes a reusable runner using Python’s standard library. It requires Linux, Python 3.10 or newer, local Ollama, and nvidia-smi for GPU telemetry. Clone it and download the models: git clone https://github.com/rafaself/lab-ollama-local-benchmarks.git cd lab-ollama-local-benchmarks ollama pull qwen3:4b ollama pull qwen3:0.6b ollama ps
自行运行对比：该存储库包含一个使用 Python 标准库的可重用运行器。它需要 Linux、Python 3.10 或更高版本、本地 Ollama 以及用于 GPU 遥测的 nvidia-smi。克隆它并下载模型： git clone https://github.com/rafaself/lab-ollama-local-benchmarks.git cd lab-ollama-local-benchmarks ollama pull qwen3:4b ollama pull qwen3:0.6b ollama ps
With Ollama serving locally and no models loaded, choose an unused results directory:
python3 benchmarks/compare_models.py
—output runs/local-qwen3-comparison-001/results.json
The runner tests models sequentially and unloads them afterward. To check the published evidence without running inference:
python3 benchmarks/validate_run.py runs/2026-10-20-qwen3-4b-vs-0.6b
Model tags can change. Compare installed digests with the recorded configuration when attempting to reproduce this baseline.
在 Ollama 本地运行且未加载模型的情况下，选择一个未使用的结果目录：
python3 benchmarks/compare_models.py
—output runs/local-qwen3-comparison-001/results.json
运行器会按顺序测试模型并在之后卸载它们。若要在不运行推理的情况下查看已发布的证据：
python3 benchmarks/validate_run.py runs/2026-10-20-qwen3-4b-vs-0.6b
模型标签可能会更改。在尝试复现此基准时，请对比已安装的摘要与记录的配置。
What I would use next: This was an exploratory experiment on one laptop. Six questions cannot establish general accuracy. Model order was fixed, normal prompt caching remained enabled, and temperature, power policy, and desktop activity were uncontrolled. Embeddings and retrieval were not evaluated: every document question included its passages directly. The methodology documents these limits. For this lab, 4B is my main answer-generation choice: it fit entirely in GPU memory and passed these checks. I would use 0.6B for quick iteration where outputs are easy to inspect. A useful next experiment would expand the document questions and alternate model order across independent sessions before making broader claims.
我接下来会怎么做：这是一次在单台笔记本电脑上的探索性实验。六个问题无法确立通用准确性。模型顺序是固定的，正常的提示词缓存保持启用，且温度、电源策略和桌面活动未受控制。未评估嵌入和检索：每个文档问题都直接包含了其段落。方法论记录了这些局限性。对于本实验室而言，4B 是我主要的答案生成选择：它完全适合 GPU 内存并通过了这些检查。我会将 0.6B 用于输出易于检查的快速迭代。一个有用的后续实验将是扩展文档问题，并在独立会话中交替模型顺序，然后再做出更广泛的结论。