Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

查看原文 / View Original 编辑 / Edit

Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

通过 100 步 GRPO 微调 350M 模型以实现更好的结构化输出

This guide is a fully public, inexpensive recipe for making a small model substantially better at structured-output compliance. We fine-tune LFM2.5-350M with Group Relative Policy Optimization (GRPO) using the TRL library and evaluate it on the IFStruct benchmark. The full run takes around 500 samples and 100 training steps, small enough for a free-tier Colab or Kaggle GPU, and is available on GitHub. The results show that even a light fine-tuning procedure improves performance from 22.6% to 29.7% on the IFStruct benchmark.

本指南提供了一套完全公开且低成本的方案，旨在显著提升小模型在结构化输出合规性方面的表现。我们使用 TRL 库通过组相对策略优化（GRPO）对 LFM2.5-350M 模型进行微调，并在 IFStruct 基准测试上进行评估。整个运行过程仅需约 500 个样本和 100 个训练步数，完全可以在免费版的 Colab 或 Kaggle GPU 上运行，相关代码已在 GitHub 上开源。结果显示，即使是轻量级的微调过程，也能将模型在 IFStruct 基准测试上的表现从 22.6% 提升至 29.7%。

Structured output is one of the most common real-world tasks for LLMs, yet most benchmarks fold it into broader reasoning or extraction scores rather than measuring it on its own. Whether a model reliably returns valid, parseable output in the requested format and shape — schema compliance — is often what decides whether it can be wired into a downstream system at all.

结构化输出是大型语言模型（LLM）在现实世界中最常见的任务之一，但大多数基准测试将其归入更广泛的推理或提取评分中，而不是对其进行单独衡量。模型是否能可靠地返回符合要求格式和形状的可解析输出（即模式合规性），往往决定了它能否被接入下游系统。

Note that the training pipeline described here is not the one used to train the RL model described in the IFStruct blog. This notebook doesn’t aim to recreate the IFStruct benchmark score, but to show how task-specific fine-tuning of smaller models can improve performance and match that of far larger models.

请注意，此处描述的训练流程并非 IFStruct 博客中所述的强化学习模型训练流程。本笔记本的目的并非重现 IFStruct 的基准测试分数，而是展示如何通过针对特定任务对小模型进行微调，从而提升性能并媲美规模大得多的模型。

Prerequisites

前置条件

This guide has two halves that run in different places: Fine-tuning runs on a GPU. The accompanying notebook is sized for a free-tier Colab or Kaggle GPU. Evaluation can run locally on a MacBook (here, a MacBook Pro with an Apple M5 Max and 36 GB of unified memory) through llama.cpp, which exposes an OpenAI-compatible server that the IFStruct evaluator talks to. We will need uv for the Python tooling and llama.cpp for serving. Following the Liquid AI llama.cpp deployment docs, install llama.cpp with Homebrew and verify that llama-server is available:

本指南分为两部分，在不同环境下运行：微调在 GPU 上进行。配套的笔记本已针对免费版 Colab 或 Kaggle GPU 进行了适配。评估可以在 MacBook（此处使用配备 Apple M5 Max 和 36 GB 统一内存的 MacBook Pro）上通过 llama.cpp 本地运行，它会暴露一个与 OpenAI 兼容的服务器，供 IFStruct 评估器调用。我们需要使用 uv 作为 Python 工具，并使用 llama.cpp 进行服务部署。按照 Liquid AI 的 llama.cpp 部署文档，使用 Homebrew 安装 llama.cpp 并验证 llama-server 是否可用：

brew install llama.cpp
llama-server --version

IFStruct Evaluation on LFM2.5-350M (Base model)

LFM2.5-350M（基础模型）的 IFStruct 评估

Before we begin, let’s evaluate LFM2.5-350M on the IFStruct benchmark and see whether we can reproduce the reported score of 21.1%. IFStruct is a benchmark for testing the validity of LLM outputs and schema adherence. The benchmark is open-source in Liquid4All/ifstruct, with the public benchmark dataset available on Hugging Face at LiquidAI/ifstruct-v1.0.

在开始之前，让我们先在 IFStruct 基准测试上评估 LFM2.5-350M，看看能否复现其报告的 21.1% 的分数。IFStruct 是一个用于测试 LLM 输出有效性和模式遵循能力的基准测试。该基准测试在 Liquid4All/ifstruct 中开源，公共基准数据集可在 Hugging Face 的 LiquidAI/ifstruct-v1.0 上获取。

git clone https://github.com/Liquid4All/ifstruct.git

For the eval comparison, we serve the model locally on the MacBook with llama.cpp. We will use the BF16 GGUF (LiquidAI/LFM2.5-350M-GGUF). Then we start the base-model server with the following command:

为了进行评估对比，我们在 MacBook 上使用 llama.cpp 本地部署模型。我们将使用 BF16 GGUF 版本 (LiquidAI/LFM2.5-350M-GGUF)。然后，使用以下命令启动基础模型服务器：

llama-server \
 -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \
 -c 32768 \
 -np 4 \
 -ngl 99 \
 --alias LiquidAI/LFM2.5-350M \
 --host 127.0.0.1 \
 --port 8080

  • --alias: model name IFStruct sends to the OpenAI-compatible endpoint

  • -ngl 99: asks llama.cpp to offload all layers to the GPU when available

  • -np 4: serves four requests in parallel

  • -c 32768: size of the prompt context

  • --alias: IFStruct 发送到 OpenAI 兼容端点的模型名称

  • -ngl 99: 要求 llama.cpp 在可用时将所有层卸载到 GPU

  • -np 4: 并行处理四个请求

  • -c 32768: 提示词上下文大小

Once the server is running, we can run the full benchmark with 2000 samples:

服务器运行后，我们可以使用 2000 个样本运行完整的基准测试：

uv run ifstruct-eval \
 --model LiquidAI/LFM2.5-350M \
 --base-url http://localhost:8080/v1 \
 --api-key dummy \
 --dataset data/test.jsonl \
 --results-file results/lfm2.5-350m-llamacpp-base.json \
 --n-threads 4 \
 --max-tokens 2048 \
 -v

(Results summary omitted for brevity)

(结果摘要略)

The IFStruct release blog reports 21.1% for LFM2.5-350M. Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.

IFStruct 发布博客报告 LFM2.5-350M 的得分为 21.1%。我们本地的 llama.cpp/BF16 设置测得为 22.6%，与博客中报告的 21.1% 相近。我们将此本地结果作为同一服务栈对比的基准。

GRPO Fine-tuning with TRL on Structured Outputs

使用 TRL 进行结构化输出的 GRPO 微调

The full, runnable pipeline lives in the accompanying notebook. We will cover only the relevant pieces in this section.

完整且可运行的流水线位于配套的笔记本中。本节仅涵盖相关部分。

Training data 训练数据

We use nvidia/Nemotron-RL-instruction_following-structured_outputs, which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training. Because the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two…

我们使用 nvidia/Nemotron-RL-instruction_following-structured_outputs，它将每个提示词与目标 JSON Schema 和预期的字段计数配对。我们使用约 500 个样本进行训练。由于 Nemotron 的数据分布与 IFStruct 评估不同，我们对提示词进行了增强以缩小两者之间的差距……

← 返回新闻列表 / Back to news list