Fine-Tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

Fine-Tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

Structured output compliance is a frequent requirement in production language-model applications. A model may generate useful content, but downstream systems often depend on whether that content is valid, parseable, and shaped according to a requested schema.

This guide fine-tunes LFM2.5-350M with Group Relative Policy Optimization (GRPO), using the TRL library, and evaluates the result on the IFStruct benchmark. The training run uses about 500 samples and 100 steps, making it small enough for a free-tier Colab or Kaggle GPU. On the benchmark, the model's score increases from 22.6% to 29.7%.

The training pipeline described here is not the one used for the reinforcement-learning model discussed in the IFStruct release. It is intended to demonstrate how task-specific fine-tuning can improve a smaller model's structured-output performance. It does not attempt to reproduce the published IFStruct score.

Prerequisites

The workflow has two parts that can run in different environments:

• Fine-tuning runs on a GPU. The accompanying notebook is designed for a free-tier Colab or Kaggle GPU.
• Evaluation can run locally on a MacBook. The reference setup uses a MacBook Pro with an Apple M5 Max and 36 GB of unified memory, running llama.cpp as an OpenAI-compatible server for the IFStruct evaluator.

The tooling requires uv for Python workflows and llama.cpp for model serving. On macOS, install llama.cpp with Homebrew and check that llama-server is available:

brew install llama.cpp
llama-server --version

Evaluating the LFM2.5-350M Base Model

The first step is to evaluate LFM2.5-350M on IFStruct. The reported score for this model is 21.1%, and the local evaluation checks whether a similar result can be obtained.

IFStruct measures output validity and schema adherence. Its implementation is open source in Liquid4All/ifstruct, and the public dataset is available as LiquidAI/ifstruct-v1.0.

git clone https://github.com/Liquid4All/ifstruct.git

For evaluation, the model is served locally with llama.cpp using the BF16 GGUF file from LiquidAI/LFM2.5-350M-GGUF:

llama-server \
  -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --alias LiquidAI/LFM2.5-350M \
  --host 127.0.0.1 \
  --port 8080

The relevant options are:

• --alias: the model name sent by IFStruct to the OpenAI-compatible endpoint
• -ngl 99: requests GPU offloading for all layers when available
• -np 4: serves four requests concurrently
• -c 32768: sets the prompt context size

Run the full benchmark with 2,000 samples:

uv run ifstruct-eval \
  --model LiquidAI/LFM2.5-350M \
  --base-url http://localhost:8080/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-llamacpp-base.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v

The local run produces these headline results:

Model: LiquidAI/LFM2.5-350M
Overall: 452/2000 passed (22.6%)
Average latency: 1453ms

By format:
  JSON: 180/1000 passed (18.0%)
  YAML: 272/1000 passed (27.2%)

By top-level structure:
  Wrapper key 288/1011 passed (28.5%)
  Bare list   164/989 passed (16.6%)

The most frequent errors are missing required fields, incorrect item counts, type mismatches, unclosed code blocks, and extraneous fields:

7228x required field missing
738x wrong item count
540x type mismatch
317x Unclosed code block
190x extraneous field 'notes'
181x extraneous field 'path'
175x extraneous field 'constraints'
170x extraneous field 'type'
170x missing code block
100x expected bare list, got wrapper

The IFStruct release reports 21.1% for LFM2.5-350M. The local llama.cpp and BF16 setup reports 22.6%, which is used as the baseline for the matching serving-stack comparison.

GRPO Fine-Tuning with TRL

The complete runnable pipeline is provided in the accompanying notebook. The main components are the training data, a LoRA adapter, structured-output reward functions, and a short GRPO training run.

Training Data

The procedure uses nvidia/Nemotron-RL-instruction_following-structured_outputs. Each example pairs a prompt with a target JSON Schema and an expected field count. Approximately 500 samples are used for training.

The Nemotron data distribution differs from IFStruct, so the prompts are augmented to address two evaluation requirements:

• For 40% of the examples, the instruction to return the output inside a fenced code block is appended. This teaches the model to follow the requested presentation format rather than always emitting raw JSON.
• A disjoint 20% of the examples are converted into top-level-array tasks. The schema is wrapped in an array with a required item count, training bare-list output and item-count compliance.

Model and LoRA Configuration

LiquidAI/LFM2.5-350M is loaded with a LoRA adapter. Because LFM2.5 uses a hybrid attention and convolution architecture, the adapter targets LFM-specific module names:

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
        "w1", "w2", "w3",
    ],
)

The configuration trains approximately 6 million parameters, or about 1.66% of the model.

Reward Functions

Three reward functions score each completion on a scale from 0 to 1. They focus on whether the extracted structure is correct:

• json_format_reward: checks whether the output is parseable and uses the requested form. The requested form, fenced or raw, receives 1.0; a different but parseable form receives 0.2; unparseable output receives 0.0.
• field_count_reward: checks whether the object contains the expected number of top-level fields. An exact match receives 1.0, while the score decreases linearly as the count differs from the target.
• schema_validation_reward: checks whether the output validates against the row's JSON Schema. It counts constraint violations and limits partial credit according to required-key coverage.

The rewards are combined with:

reward_weights = [1.0, 0.5, 2.0]

These weights correspond to JSON format, field count, and schema validation, respectively.

Training Configuration

The model is trained for 100 steps with eight generations per prompt group. The configuration is sized for a 16 GB GPU:

from trl import GRPOConfig

training_args = GRPOConfig(
    output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",
    learning_rate=5e-5,
    max_steps=100,
    warmup_steps=10,
    num_generations=8,              # completions sampled per prompt group
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,  # 4 prompt groups per optimizer step
    steps_per_generation=2,
    max_completion_length=1024,     # room for nested JSON
    mask_truncated_completions=False,
    temperature=1.1,                # hotter sampling keeps groups varied
    beta=0.01,                      # KL penalty toward the reference model
    reward_weights=[1.0, 0.5, 2.0], # json_format, field_count, schema_validation
    logging_steps=1,
    save_steps=100,
)

During the run, all three reward components increase. The KL divergence from the reference model rises from zero after warmup, while the fraction of truncated completions remains close to zero.

Merging and Saving the Model

After training, the LoRA adapter is merged into the base weights. The result is saved as a self-contained checkpoint that can be converted to GGUF for serving:

MERGED_DIR = f"{training_args.output_dir}-merged"

merged_model = trainer.model.merge_and_unload()
merged_model.save_pretrained(MERGED_DIR)
tokenizer.save_pretrained(MERGED_DIR)

Evaluating the GRPO-Tuned Model

The merged checkpoint must first be converted to a BF16 GGUF file. The conversion script is included with the llama.cpp source, and its gguf package can be installed as follows:

git clone --depth 1 https://github.com/ggml-org/llama.cpp
pip install ./llama.cpp/gguf-py

mkdir -p models
python llama.cpp/convert_hf_to_gguf.py \
  PATH_TO_YOUR_MERGED_MODEL \
  --outfile ./models/lfm25-350m-grpo-bf16.gguf \
  --outtype bf16

Serve the converted model on port 8081:

llama-server \
  -m ./models/lfm25-350m-grpo-bf16.gguf \
  --alias lfm25-350m-grpo-structured-output \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8081

Run the same 2,000-sample IFStruct evaluation against the fine-tuned model:

uv run ifstruct-eval \
  --model lfm25-350m-grpo-structured-output \
  --base-url http://localhost:8081/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm25-350m-grpo.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v

The tuned model reports:

Model: lfm25-350m-grpo-structured-output
Overall: 594/2000 passed (29.7%)
Average latency: 1518ms

By format:
  JSON: 319/1000 passed (31.9%)
  YAML: 275/1000 passed (27.5%)

By top-level structure:
  Wrapper key 300/1011 passed (29.7%)
  Bare list   294/989 passed (29.7%)

The most frequent errors are:

7331x required field missing
890x wrong item count
555x type mismatch
102x expected bare list, got wrapper
62x extraneous field 'metadata.tone'
55x 6 is greater than maximum 5
49x extraneous field 'speaker_labels'
47x extraneous field 'tone'
44x 'cups' not in allowed values ['mg', 'g', 'kg', 'oz', 'lb', 'ml', 'l', 'cl', 'dl']
44x extraneous field 'notes'

Comparing the Results

The two evaluations use the same serving configuration:

The largest improvements occur in the areas targeted during training. JSON performance increases by 13.9 percentage points, from 18.0% to 31.9%, while YAML performance changes only slightly. Bare-list performance rises by 13.1 points, from 16.6% to 29.7%.

The tuned result remains below the Qwen3.5-2B score of 33.15%, but the experiment demonstrates the effect of a short, task-specific fine-tuning run on a 350M-parameter model.

Conclusion

A GRPO run using approximately 500 samples and 100 training steps raises LFM2.5-350M from 22.6% to 29.7% on IFStruct. The training process uses a small LoRA adapter and reward functions focused on output format, field counts, and JSON Schema validation.

The results show that targeted optimization can improve a small model's reliability on structured-output requirements, particularly for JSON formatting and bare-list tasks. The benchmark implementation, dataset, and training notebook provide the components needed to reproduce or extend the experiment.