FlashML-org / FreeToken

About FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include: Fast Edge-Native Runtime: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format. Semantic-Aware Caching: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid redundant context recomputation. Elastic Memory Management: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading. Broad MoE & Ecosystem Support: Supports frontier open-weight MoE models (e.g., DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) across various parameter scales and quantization formats (e.g., MXFP4, NVFP4, FP8, BF16), with Anthropic/OpenAI-compatible APIs for seamless integration with real-world coding and tool-calling agents (e.g., Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness). Diverse Consumer Hardware: Scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.

关于 FreeToken 是一款边缘原生混合专家（MoE）服务引擎，旨在个人和消费级硬件上运行前沿规模的开源权重模型。它将异构的边缘资源（GPU、CPU、主机内存和互连）视为一个统一的、弹性的推理平台。其核心功能包括：快速边缘原生运行时：提供高效的 MoE 服务，具备带宽自适应 CPU-GPU 协同执行（$q^\star$ 策略）、全层双缓冲预填充流式传输、全局 LRU 专家缓存、图兼容执行以及 FTW 快速权重格式。语义感知缓存：针对循环状态和 KV 缓存提供语义锚点检查点，允许智能体上下文编辑（如工具调用、思维块）以避免冗余的上下文重新计算。弹性内存管理：支持在专家缓存和 KV 内存之间进行动态的运行时显存重分配，无需重启引擎或重新加载权重。广泛的 MoE 与生态支持：支持各种参数规模和量化格式（如 MXFP4、NVFP4、FP8、BF16）的前沿开源权重 MoE 模型（如 DeepSeek-V4-Flash、Qwen3.6-35B-A3B、GLM-5.2），并提供与 Anthropic/OpenAI 兼容的 API，以便与实际的编码和工具调用智能体（如 Codex、Claude Code、OpenCode、OpenClaw、DeepSeek Harness）无缝集成。多样的消费级硬件：可扩展至消费级笔记本电脑、游戏台式机和工作站 GPU，并原生支持 NVIDIA RTX 30、RTX 40 和 RTX 50 系列 GPU。

Getting Started Desktop app Download FreeToken for Windows or Linux at flashml.ai. It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine. CLI Install FreeToken with uv (recommended) or pip: uv pip install “freetoken[accel]” Or build from source: git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken uv venv && source .venv/bin/activate uv pip install -e ”.[accel]” For More details: Install FreeToken Quick start Supported models CLI reference Repairing old FTW checkpoints Diagnosing a checkpoint that fails to load

入门指南桌面应用：在 flashml.ai 下载适用于 Windows 或 Linux 的 FreeToken。它会为你配置引擎，并提供一个用于运行模型、聊天和调整引擎的图形用户界面（GUI）。命令行界面（CLI）：使用 uv（推荐）或 pip 安装 FreeToken：uv pip install “freetoken[accel]” 或从源码构建：git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken uv venv && source .venv/bin/activate uv pip install -e ”.[accel]” 更多详情：安装 FreeToken、快速入门、支持的模型、CLI 参考、修复旧的 FTW 检查点、诊断无法加载的检查点。

Citation If you use FreeToken for your research, please cite our paper: @article{yang2026freetoken, title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution}, author={Yang, Shuo and Fan, Xiaoze and Pan, Melissa and Xi, Haocheng and Wang, Zhe and Sun, Shanlin and Keutzer, Kurt and Han, Song and Zaharia, Matei and Xu, Chenfeng and Stoica, Ion}, journal={arXiv preprint arXiv:2608.16157}, year={2026} } Acknowledgment FreeToken was deeply inspired by mini-sglang, and learned the design and reused code from the following projects: SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp. License Apache License 2.0.

引用：如果你在研究中使用 FreeToken，请引用我们的论文：@article{yang2026freetoken, title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution}, author={Yang, Shuo and Fan, Xiaoze and Pan, Melissa and Xi, Haocheng and Wang, Zhe and Sun, Shanlin and Keutzer, Kurt and Han, Song and Zaharia, Matei and Xu, Chenfeng and Stoica, Ion}, journal={arXiv preprint arXiv:2608.16157}, year={2026} } 致谢：FreeToken 深受 mini-sglang 的启发，并学习了以下项目的设计并复用了其代码：SGLang、vLLM、FlashInfer、flash-linear-attention、LightLLM 和 llama.cpp。许可证：Apache License 2.0。