PrismML-Eng / Bonsai-demo

查看原文 / View Original 编辑 / Edit

PrismML-Eng / Bonsai-demo

Bonsai Demo Website | GitHub | Discord HF Collections: Bonsai 27B · Bonsai (1-bit) · Ternary-Bonsai Whitepapers: Bonsai 27B · 1-bit Bonsai 8B · Ternary-Bonsai 8B

Bonsai 演示网站 | GitHub | Discord HuggingFace 合集：Bonsai 27B · Bonsai (1-bit) · Ternary-Bonsai 白皮书：Bonsai 27B · 1-bit Bonsai 8B · Ternary-Bonsai 8B

Using this demo repository you can run Bonsai (1-bit) and Ternary-Bonsai language models locally on Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), or CPU. 通过此演示仓库，你可以在 Mac (Metal)、Linux/Windows (CUDA, Vulkan, ROCm) 或 CPU 上本地运行 Bonsai (1-bit) 和 Ternary-Bonsai 语言模型。

🌱 New: Bonsai 27B The family’s newest and largest generation, and its first vision-language models (Bonsai 27B collection): 🌱 新增：Bonsai 27B 这是该系列最新、最大的版本，也是其首个视觉语言模型（Bonsai 27B 合集）：

  • Vision: send photos, screenshots, and PDFs; ask about them (see VISION.md).

  • 视觉： 发送照片、截图和 PDF；并针对它们进行提问（详见 VISION.md）。

  • Agentic tool calling: native OpenAI-style tool_calls with full round-trips, plus MCP servers in both demo UIs (see TOOLS.md).

  • 智能体工具调用： 原生 OpenAI 风格的 tool_calls，支持完整的往返交互，并在两个演示 UI 中集成了 MCP 服务器（详见 TOOLS.md）。

  • Thinking: a reasoning model; pick the reasoning effort per chat in the UI or budget it per request.

  • 思考： 一个推理模型；你可以在 UI 中为每次对话选择推理强度，或为每个请求设定预算。

  • Long context: 256k+ token conversations.

  • 长上下文： 支持 256k+ token 的对话。

  • Tiny footprint: the 1-bit Bonsai-27B packs to ~1.125 bits per weight: it fits on a modern iPhone without memory offloading.

  • 极小占用： 1-bit 的 Bonsai-27B 压缩至每个权重约 1.125 bit：无需内存卸载即可在现代 iPhone 上运行。

Ternary-Bonsai-27B (~1.7 bits per weight, packed into 2-bit for fast accelerated kernels) is the higher-quality option and this demo’s default. Ternary-Bonsai-27B（每个权重约 1.7 bit，打包为 2-bit 以实现快速加速内核）是更高质量的选择，也是本演示的默认模型。

Quick Start below gets you there in two commands: ./setup.sh downloads Ternary-Bonsai-27B by default, then ./scripts/start_llama_server.sh gives you chat, vision, and tools at http://localhost:8080. 下方的快速入门只需两条命令即可完成：./setup.sh 默认下载 Ternary-Bonsai-27B，随后运行 ./scripts/start_llama_server.sh，即可在 http://localhost:8080 使用聊天、视觉和工具功能。

Quick Start

快速入门

Setting things up with an AI coding agent? Point it at AGENTS.md, a guide written for agents (hardware-specific knobs, defaults, and what to ask the user). 正在使用 AI 编程智能体进行配置？请将其指向 AGENTS.md，这是一份专为智能体编写的指南（包含硬件特定参数、默认设置以及如何向用户提问）。

macOS / Linux

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
# (Optional) Choose a model size: 27B (default), 8B, 4B, or 1.7B
export BONSAI_MODEL=27B
# Set your HuggingFace token (only required for 27B while its repos are private)
export BONSAI_TOKEN="hf_your_token_here"
# One command does everything: installs deps, downloads models + binaries
./setup.sh

Windows (PowerShell)

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
# (Optional) Choose a model size: 27B (default), 8B, 4B, or 1.7B
$env:BONSAI_MODEL = "27B"
# Set your HuggingFace token (only required for 27B while its repos are private)
$env:BONSAI_TOKEN = "hf_your_token_here"
# Run setup
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1

Switching families and sizes

切换系列与尺寸

You can switch between the Ternary (default) and 1-bit families, and different model sizes instantly: 你可以即时在 Ternary（默认）和 1-bit 系列以及不同模型尺寸之间切换：

# run Ternary-Bonsai 4B
BONSAI_FAMILY=ternary BONSAI_MODEL=4B ./scripts/download_models.sh
BONSAI_FAMILY=ternary BONSAI_MODEL=4B ./scripts/run_llama.sh -p "Hello!"

# for Windows:
$env:BONSAI_FAMILY="ternary"; $env:BONSAI_MODEL="4B"
.\setup.ps1
.\scripts\run_llama.ps1 -p "Hello!"

Speed Benchmarks

速度基准测试

See community-benchmarks/ for results on different hardware and templates to submit your own. 查看 community-benchmarks/ 获取不同硬件上的测试结果，并使用模板提交你自己的测试数据。

Models

模型

Two model families are available, each in sizes 27B, 8B, 4B, and 1.7B. The 27B models are vision-language models: they accept images as well as text; all 27B repos are gathered in the Bonsai 27B HF collection. 目前提供两个模型系列，每个系列均有 27B、8B、4B 和 1.7B 四种尺寸。27B 模型为视觉语言模型：它们既接受文本也接受图像；所有 27B 仓库均汇总在 Bonsai 27B HF 合集中。

Both formats are landing in mainline llama.cpp: Q1_0 (1-bit) is fully merged upstream, and Q2_0 (ternary) now runs on mainline CPU and Metal, with Vulkan in review. 两种格式均正在进入 llama.cpp 主线：Q1_0 (1-bit) 已完全合并至上游，Q2_0 (ternary) 目前可在主线 CPU 和 Metal 上运行，Vulkan 支持正在审核中。

Bonsai (1-bit)

Available in GGUF (llama.cpp) and MLX 1-bit formats. 提供 GGUF (llama.cpp) 和 MLX 1-bit 格式。

ModelFormatHuggingFace Repo
Bonsai-27BGGUFprism-ml/Bonsai-27B-gguf
Bonsai-27BMLXprism-ml/Bonsai-27B-mlx-1bit
Bonsai-8BGGUFprism-ml/Bonsai-8B-gguf
Bonsai-8BMLXprism-ml/Bonsai-8B-mlx-1bit
Bonsai-4BGGUFprism-ml/Bonsai-4B-gguf
Bonsai-4BMLXprism-ml/Bonsai-4B-mlx-1bit
Bonsai-1.7BGGUFprism-ml/Bonsai-1.7B-gguf
Bonsai-1.7BMLXprism-ml/Bonsai-1.7B-mlx-1bit

Set BONSAI_MODEL to choose which size to download and run (default: 27B). 设置 BONSAI_MODEL 以选择要下载和运行的尺寸（默认：27B）。

Ternary-Bonsai

Available in GGUF (llama.cpp) and MLX 2-bit formats. This is the default family. Set BONSAI_FAMILY=bonsai to use the 1-bit Bonsai family instead. 提供 GGUF (llama.cpp) 和 MLX 2-bit 格式。这是默认系列。设置 BONSAI_FAMILY=bonsai 可切换至 1-bit Bonsai 系列。

ModelFormatHuggingFace Repo
Ternary-Bonsai-27BGGUFprism-ml/Ternary-Bonsai-27B-gguf
Ternary-Bonsai-27BMLX (2-bit)prism-ml/Ternary-Bonsai-27B-mlx-2bit
Ternary-Bonsai-8BGGUFprism-ml/Ternary-Bonsai-8B-gguf
Ternary-Bonsai-8BMLX (2-bit)prism-ml/Ternary-Bonsai-8B-mlx-2bit
Ternary-Bonsai-4BGGUFprism-ml/Ternary-Bonsai-4B-gguf
Ternary-Bonsai-4BMLX (2-bit)prism-ml/Ternary-Bonsai-4B-mlx-2bit
Ternary-Bonsai-1.7BGGUFprism-ml/Ternary-Bonsai-1.7B-gguf
Ternary-Bonsai-1.7BMLX (2-bit)prism-ml/Ternary-Bonsai-1.7B-mlx-2bit

Environment variables

环境变量

Both variables are optional. If you set neither, the default is Ternary-Bonsai-27B: that’s what plain ./setup.sh downloads and runs. They’re read by setup.sh, setup.ps1, download_models.sh, and every run_* / start_* script (Linux, macOS, and Windows). 两个变量均为可选。如果不设置，默认值为 Ternary-Bonsai-27B：这也是直接运行 ./setup.sh 时下载和运行的内容。它们会被 setup.sh、setup.ps1、download_models.sh 以及所有 run_* / start_* 脚本（Linux、macOS 和 Windows）读取。

VariableDefaultValid valuesPurpose
BONSAI_FAMILYternaryternary, bonsai, all模型系列。ternary = Ternary-Bonsai; bonsai = 1-bit Bonsai. all 展开为两个系列（仅限 setup/download）。
BONSAI_MODEL27B27B, 8B, 4B, 1.7B, all模型尺寸。all 展开为全部四种尺寸（仅限 setup/download）。
BONSAI_TOKEN—HF read-only token仅在 27B 模型仓库私有时需要（发布后已移除）。

all is only valid for setup.sh / setup.ps1 / download_models.sh — the run/server scripts need a concrete family/size. all 仅对 setup.sh / setup.ps1 / download_models.sh 有效——运行/服务器脚本需要具体的系列/尺寸。

Combine them freely: 自由组合示例：

./setup.sh # Ternary-Bonsai-27B (default)
BONSAI_MODEL=1.7B ./setup.sh # Ternary-Bonsai-1.7B
BONSAI_FAMILY=bonsai ./setup.sh # Bonsai-27B (1-bit)
BONSAI_FAMILY=bonsai BONSAI_MODEL=4B ./setup.sh # Bonsai-4B
BONSAI_MODEL=all ./setup.sh # All 4 Ternary-Bonsai sizes
BONSAI_FAMILY=all BONSAI_MODEL=all ./setup.sh # Full matrix (8 downloads)

Upstream Status for Binary

二进制上游状态

Q1_0 is supported out of the box in upstream llama.cpp across many backends: CPU (generic, NEON, and optimized x86), Metal, CUDA, and Vulkan. Q1_0 在上游 llama.cpp 的多个后端中均可直接支持：CPU（通用、NEON 和优化后的 x86）、Metal、CUDA 和 Vulkan。

Runtime Status

运行时状态

  • llama.cpp (CPU, Metal, CUDA, Vulkan): ✅ Merged upstream, works out of the box
  • llama.cpp (CPU, Metal, CUDA, Vulkan): ✅ 已合并至上游，可直接使用
  • MLX (1-bit): ⏳ Pending upstream: mlx#3161; until it merges, use PrismML-Eng/mlx (branch prism, built automatically by setup.sh)
  • MLX (1-bit): ⏳ 等待上游合并：mlx#3161；在合并前，请使用 PrismML-Eng/mlx（分支 prism，由 setup.sh 自动构建）

Upstream Status for Ternary

Ternary 上游状态

Ternary support is in the middle of migrating in Ternary 支持目前正处于迁移过程中

← 返回新闻列表 / Back to news list