DeepInfer: High-performance inference framework for DeepSeek models

DeepInfer is the high-performance inference framework that powers DeepSeek’s large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

DeepInfer 是支持 DeepSeek 大规模生产服务的高性能推理框架。此次开源版本提供了在大规模服务中所需的核心离线推理功能。

Required for a complete inference service (not included in this release):

构建完整推理服务所需条件（本版本未包含）：

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

在构建 DeepInfer 之前，请创建并激活一个空的 Python 3.12 环境。

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

默认检查点 deepseek-ai/DeepSeek-V4.1-Flash-1017 会在首次运行时下载到 Hugging Face 缓存中；若要从本地副本运行，请设置 DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017。

See examples for details.

详情请参阅示例。

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

测试基于 4 张 SM100 GPU 进行预填充（prefill），64 张 GPU 进行解码（decode）——均在单个纵向扩展域内。如需复现基准测试，请参阅 run_sweep.sh。

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

DeepInfer 基于 Apache License 2.0 协议发布。详情请参阅 LICENSE 文件。

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

DeepInfer 最初主要基于 vLLM 构建，并参考了 SGLang 和 TensorRT-LLM 的设计。我们感谢这些项目的开发者。

DeepInfer: High-performance, large-scale inference framework for DeepSeek models

DeepInfer：面向 DeepSeek 模型的高性能、大规模推理框架