lyogavin / airllm

查看原文 / View Original 编辑 / Edit

lyogavin / airllm

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB, and DeepSeek-V3 (671B) on ~12GB.

AirLLM 显著降低了推理时的内存占用，使得 70B 参数的大语言模型能够在单张 4GB 显存的 GPU 上运行，且无需进行量化、蒸馏或剪枝。你甚至可以在 8GB 显存上运行 405B 的 Llama 3.1，在约 12GB 显存上运行 DeepSeek-V3 (671B)。

Updates

更新日志

[2026/06] v3.0: FP8 model support + the latest models. Run DeepSeek-V3 (671B) on ~12GB and Qwen3-235B on ~3GB, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single AutoModel.

[2026/06] v3.0：支持 FP8 模型及最新模型。可在约 12GB 显存上运行 DeepSeek-V3 (671B)，在约 3GB 显存上运行 Qwen3-235B，此外还支持 Qwen3、Llama 3.x/4、DeepSeek V2/V3、Phi-4、Gemma 等——所有这些均可通过单一的 AutoModel 实现。

Quickstart

快速开始

  1. Install package: First, install the airllm pip package. pip install airllm

  2. 安装包：首先，安装 airllm 的 pip 包。 pip install airllm

  3. Inference: Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model.

  4. 推理：接着，初始化 AirLLMLlama2，传入所使用模型的 HuggingFace 仓库 ID 或本地路径，即可像使用常规 Transformer 模型一样进行推理。

from airllm import AutoModel
MAX_LENGTH = 128 
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B") 
# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B, runs in ~12GB

input_text = ['What is the capital of United States?',]
input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=False)
generation_output = model.generate(input_tokens['input_ids'].cuda(), max_new_tokens=20, use_cache=True, return_dict_in_generate=True)
output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory.

注意：在推理过程中，原始模型会首先被分解并按层保存。请确保 HuggingFace 缓存目录中有足够的磁盘空间。

Model Compression - 3x Inference Speed Up!

模型压缩 - 推理速度提升 3 倍！

We just added model compression based on block-wise quantization-based model compression. Which can further speed up the inference speed for up to 3x, with almost ignorable accuracy loss!

我们刚刚增加了基于分块量化（block-wise quantization）的模型压缩功能。这可以在几乎不损失精度的情况下，将推理速度进一步提升至原来的 3 倍！

How to enable model compression speed up: 如何开启模型压缩加速：

Step 1. make sure you have bitsandbytes installed: pip install -U bitsandbytes Step 2. make sure airllm version later than 2.0.0: pip install -U airllm Step 3. when initialize the model, passing the argument compression (‘4bit’ or ‘8bit’):

第一步：确保已安装 bitsandbytes：pip install -U bitsandbytes 第二步：确保 airllm 版本高于 2.0.0：pip install -U airllm 第三步：在初始化模型时，传入 compression 参数（‘4bit’ 或 ‘8bit’）：

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", compression='4bit')

Configurations

配置项

When initialize the model, we support the following configurations: 在初始化模型时，我们支持以下配置：

  • compression: supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression.

  • profiling_mode: supported options: True to output time consumptions or by default False.

  • layer_shards_saving_path: optionally another path to save the splitted model.

  • hf_token: huggingface token can be provided here if downloading gated models.

  • prefetching: prefetching to overlap the model loading and compute. By default, turned on.

  • delete_original: if you don’t have too much disk space, you can set delete_original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space.

  • compression：支持 4bit、8bit 分块量化，默认为 None（不压缩）。

  • profiling_mode：支持 True（输出耗时统计），默认为 False。

  • layer_shards_saving_path：可选，用于保存分解后模型的路径。

  • hf_token：如果下载受限模型（gated models），可在此提供 HuggingFace Token。

  • prefetching：预取功能，用于重叠模型加载与计算过程，默认开启。

  • delete_original：如果磁盘空间不足，可将此项设为 True，删除原始下载的 HuggingFace 模型，仅保留转换后的模型，以节省一半磁盘空间。

MacOS

苹果系统 (MacOS)

Just install airllm and run the code the same as on linux. Only Apple silicon is supported. Make sure you installed mlx and torch.

只需安装 airllm，运行代码的方式与 Linux 相同。仅支持 Apple Silicon 芯片。请确保已安装 mlx 和 torch。

← 返回新闻列表 / Back to news list