higgsfield-ai / higgsfield

Higgsfield: Multi-node training without the tears

Higgsfield 是一个开源、容错且高度可扩展的 GPU 编排与机器学习框架，专为训练参数规模从数十亿到数万亿的模型（如大语言模型 LLM）而设计。

Higgsfield serves as a GPU workload manager and machine learning framework with five primary functions: Higgsfield 作为一个 GPU 工作负载管理器和机器学习框架，具备五大核心功能：

Allocating exclusive and non-exclusive access to compute resources (nodes) to users for their training tasks.
为用户分配计算资源（节点）的独占或非独占访问权限，以执行训练任务。
Supporting ZeRO-3 deepspeed API and fully sharded data parallel API of PyTorch, enabling efficient sharding for trillion-parameter models.
支持 DeepSpeed 的 ZeRO-3 API 和 PyTorch 的完全分片数据并行（FSDP）API，实现万亿参数模型的高效分片。
Offering a framework for initiating, executing, and monitoring the training of large neural networks on allocated nodes.
提供一个在已分配节点上启动、执行和监控大型神经网络训练的框架。
Managing resource contention by maintaining a queue for running experiments.
通过维护运行实验的队列来管理资源竞争。
Facilitating continuous integration of machine learning development through seamless integration with GitHub and GitHub Actions.
通过与 GitHub 和 GitHub Actions 的无缝集成，促进机器学习开发的持续集成。

Higgsfield streamlines the process of training massive models and empowers developers with a versatile and robust toolset. Higgsfield 简化了大规模模型的训练流程，并为开发者提供了功能多样且强大的工具集。

Install

安装

$ pip install higgsfield==0.0.3

Train example

训练示例

That’s all you have to do in order to train LLaMa in a distributed setting: 在分布式环境下训练 LLaMa，你只需要做这些：

from higgsfield.llama import Llama70b
from higgsfield.loaders import LlamaLoader
from higgsfield.experiment import experiment
import torch.optim as optim
from alpaca import get_alpaca_data

@experiment("alpaca")
def train(params):
    model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")
    optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)
    dataset = get_alpaca_data(split="train")
    train_loader = LlamaLoader(dataset, max_words=2048)
    
    for batch in train_loader:
        optimizer.zero_grad()
        loss = model(batch)
        loss.backward()
        optimizer.step()
        model.push_to_hub('alpaca-70b')

How it’s all done?

它是如何运作的？

We install all the required tools in your server (Docker, your project’s deploy keys, higgsfield binary). Then we generate deploy & run workflows for your experiments. As soon as it gets into Github, it will automatically deploy your code on your nodes. Then you access your experiments’ run UI through Github, which will launch experiments and save the checkpoints. 我们在你的服务器上安装所有必需的工具（Docker、项目的部署密钥、Higgsfield 二进制文件）。然后，我们为你的实验生成部署和运行工作流。一旦推送到 GitHub，它就会自动将代码部署到你的节点上。随后，你可以通过 GitHub 访问实验的运行界面，从而启动实验并保存检查点。

Design

设计

We follow the standard pytorch workflow. Thus you can incorporate anything besides what we provide, deepspeed, accelerate, or just implement your custom pytorch sharding from scratch. 我们遵循标准的 PyTorch 工作流。因此，除了我们提供的功能外，你还可以集成 DeepSpeed、Accelerate，或者从零开始实现自定义的 PyTorch 分片。

Environment hell

环境地狱

No more different versions of pytorch, nvidia drivers, data processing libraries. You can easily orchestrate experiments and their environments, document and track the specific versions and configurations of all dependencies to ensure reproducibility. 告别不同版本的 PyTorch、NVIDIA 驱动程序和数据处理库带来的困扰。你可以轻松编排实验及其环境，记录并追踪所有依赖项的具体版本和配置，以确保可复现性。

Config hell

配置地狱

No need to define 600 arguments for your experiment. No more yaml witchcraft. You can use whatever you want, whenever you want. We just introduce a simple interface to define your experiments. We have even taken it further, now you only need to design the way to interact. 无需为实验定义 600 个参数，也不再需要复杂的 YAML 配置。你可以随时随地使用任何你想要的东西。我们只是引入了一个简单的接口来定义实验。我们甚至更进一步，现在你只需要设计交互方式即可。

Compatibility

兼容性

We need you to have nodes with: Ubuntu, SSH access, Non-root user with sudo privileges (no-password is required). 我们需要你的节点具备：Ubuntu 系统、SSH 访问权限、具有 sudo 权限的非 root 用户（无需密码）。

Clouds we have tested on: Azure, LambdaLabs, FluidStack. Feel free to open an issue if you have any problems with other clouds. 我们已测试的云平台：Azure、LambdaLabs、FluidStack。如果你在使用其他云平台时遇到问题，欢迎提交 Issue。

Getting started

入门指南

Setup: Here you can find the quick start guide on how to setup your nodes and start training. 设置：你可以在此处找到关于如何设置节点并开始训练的快速入门指南。

Initialize the project
初始化项目
Setup the environment
设置环境
Setup git
设置 Git
Time to setup your nodes!
设置节点的时间到了！
Run your very first experiment
运行你的第一个实验
Fasten your seatbelt, it’s time to deploy!
系好安全带，准备部署！

Tutorial

教程

API for common tasks in Large Language Models training.
大语言模型训练中常见任务的 API。
Working with distributed model
使用分布式模型
Preparing Data
准备数据
Optimizing the Model Parameters
优化模型参数
Saving Model
保存模型
Training stabilization techniques
训练稳定性技术
Monitoring
监控