Granite 4.2 LLMs: Architecture, Training, and Usage

Granite 4.2 LLMs: Architecture, Training, and Usage

Granite 4.2 is IBM's first family of dense, decoder-only reasoning language models. The release includes 3B, 8B, and 30B variants. Each model is pretrained from scratch on approximately 15 trillion tokens, fine-tuned on reasoning and agentic data, and post-trained through a multi-stage reinforcement learning pipeline.

The models support thinking and non-thinking modes, a low-effort thinking option, native tool calling, and a 512K-token context window introduced during pretraining. The 8B and 30B versions also receive agentic reinforcement learning that teaches them to use tools inside sandboxed environments. All Granite 4.2 models are released under the Apache 2.0 license.

Overview

Granite 4.2 extends the Granite language-model family with explicit reasoning. In thinking mode, a model can produce a chain of thought before its final answer. Non-thinking mode omits deliberate reasoning, while low-effort thinking applies a short reasoning budget to relatively simple questions.

The 3B, 8B, and 30B models use the same general architecture and training sequence: pretraining from scratch, supervised fine-tuning (SFT), and multi-stage reinforcement learning. The main difference appears during post-training. The 8B and 30B models receive an additional agentic RL block that trains them to call tools, edit and execute code, operate terminals, and search the web in real environments.

All three models support native tool calling. When served through an OpenAI-compatible endpoint, such as vLLM, they produce tool calls in the OpenAI function-calling format. Granite 4.2 is also supported by SGLang.

Model Architecture

Granite 4.2 uses a dense, decoder-only transformer architecture with these components:

• Attention: Grouped Query Attention (GQA) with 40 attention heads and 8 KV heads
• Position embeddings: Rotary Position Embedding (RoPE), with θ = 10,000,000
• Feed-forward network: MLP with SwiGLU activation
• Normalization: RMSNorm, with ε = 1e-5
• Embeddings: Separate input and output embeddings
• Precision: bfloat16

Pretraining

Granite 4.2 was trained from scratch on approximately 15 trillion tokens through five phases:

1. Foundational pretraining
2. Continued foundational pretraining
3. Mid-training with higher-quality data
4. Further data annealing and quality refinement
5. Long-context training

The fifth phase extends the context window to 512K tokens. Across the phases, the data mixture and learning-rate schedule shift from broad web-scale material toward more curated sources.

The recipe follows the previous generation's approach to data blending, phase scheduling, and context extension.

Supervised Fine-Tuning

SFT converts the base model into an instruction-following assistant that can reason and use tools. The mixture contains approximately 7.2 million samples, representing roughly 100B tokens, of which about 65B are trainable. Agentic data accounts for 31.6%, and non-agentic data accounts for 68.4%.

Agentic SFT data

The agentic corpus includes:

• Software engineering: 69%
• Tool calling: 12.1%
• Terminal use: 8.0%
• Mathematics: 3.5%
• Search: 0.8%
• Actions: 0.2%

The trajectories were generated with agent scaffolds and harnesses including OpenHands, OpenCode, Terminus-2, SWE-agent, OpenResearcher, MiniSWE, OpenSeeker, EnvScaler, Gemini CLI, Hermes, Codex, and Goose. The corpus combines open-source datasets with synthetic data generated in RL environments.

Non-agentic SFT data

The non-agentic corpus includes:

• Instruction following: 18.8%
• Coding: 18.8%
• Mathematics: 14.6%
• Multilingual data: 7.0%
• Science: 5.4%
• Reasoning: 3.0%
• Safety: 0.8%

Data quality control

Before entering the final mixture, samples pass through several quality-control steps. Data from different sources is normalized into a consistent OpenAI Chat format, including standardized conversation and tool-interaction structures.

GPT-OSS-120B and Gemma 4 serve as language-model judges. Samples may be removed for low quality, fabricated information, invalid tool interactions, or calls to undefined functions. Dataset-specific heuristic rules are also applied.

Local and global deduplication use SHA-256 hashes calculated from the combination of the tools and messages fields.

SFT training configuration

The corpus is globally shuffled, divided into equally sized .parquet shards, tokenized with the model tokenizer and chat template, and prepared for distributed training. Hyperparameter sweeps cover learning-rate schedules, initial learning rates, and warm-up ratios.

Additional SFT for 30B

The 30B model receives a second SFT phase focused on agentic coding. Agentic, SWE, and coding data are upsampled, while approximately 16% of the original SFT mixture is retained as replay data.

The model is then trained for approximately one more epoch at a learning rate of 3.0e-6. This increases exposure to agentic coding trajectories while retaining capabilities from the first SFT phase.

Multi-Stage Reinforcement Learning

After SFT, Granite 4.2 goes through a multi-stage, multi-environment RL pipeline. Separate stages target mathematics, coding, science, instruction following, tool use, structured output, software engineering, terminal operation, and web search. Each stage starts from the checkpoint produced by the previous stage.

SFT -> RLVR -> Skill boosters -> SWE agent -> Terminal -> Search -> RLHF
      |---------- foundational RL ---------| |------ agentic RL, 8B / 30B ------|

The 8B and 30B models follow the complete sequence. The 3B model uses foundational RL and alignment, without the agentic block.

Training methodology

Each stage uses asynchronous GRPO, or Group Relative Policy Optimization. Generation workers continuously produce trajectories and place them in a shared buffer. When enough samples are available for a training step, the trainer updates the model and streams the new parameters back to the generation workers.

Workers reuse their KV cache during parameter refreshes. They are limited to being no more than one update behind the trainer. Truncated importance sampling handles the remaining difference between the generation and training policies by limiting the log-probability ratio.

Advantages use a leave-one-out baseline. Each response is compared with the average reward of the other responses generated for the same prompt, eliminating the need for a separate value network.

In the RLVR stage, each step uses 256 prompts with 16 responses per prompt, producing a 4,096-example batch.

Shared RL configuration

The shape of each run varies by stage, including prompt count, generations per prompt, context length, rollout turns, KL value, and learning rate.

These settings describe the 30B model. The 3B and 8B models use the same general recipe, but with fewer stages.

Reward signals

Verifiable rewards are used early because they are objective and difficult to game. Judge and preference rewards address qualities that fixed checkers cannot capture. Agentic rewards are generally sparse and may consist of a single outcome signal after a long tool-use trajectory.

Foundational RL

RLVR

RLVR is the broad foundational stage. Its data includes multiple verifiable domains:

• Mathematics with boxed-answer checking and formal proving in Lean
• Competitive programming checked against hidden tests in a sandbox
• STEM and graduate-level science multiple-choice questions
• General knowledge
• Instruction following, structured-output, and inverse-instruction tasks
• Single-step tool and function calling
• Reasoning puzzles and abstention tasks

Each task type uses its own verifier. RLVR runs for two rounds on the 3B and 8B models, and three rounds on the 30B model. Each round is a new warm-started run with a reweighted combination of public and internally curated RL data.

Skill boosters

After RLVR, focused booster stages improve selected capabilities:

• Instruction following: Multi-turn chat, inverse-IFEval, and structured outputs
• Code: Competitive programming

These short runs use a light KL penalty to keep the model close to its current behavior while emphasizing the selected skill.

Agentic RL for 8B and 30B

The agentic stages train models to call tools, observe results, and iterate in real environments. They use multi-turn tool interaction, sparse outcome rewards, GRPO, and warm starts from the coding-boosted checkpoint. The stages run in this order: SWE, Terminal, and Search.

SWE agent

Each task is a real software repository in a sandbox. Using the OpenHands harness, the model reads code, edits files, and runs test suites over multiple turns. The reward is based on whether hidden tests pass. Tasks come from open-source SWE datasets, with each instance backed by a repository-specific container image.

Terminal agent

Terminal tasks run in a live shell through the Harbor and Terminus-2 harnesses. The model plans commands, observes their output, and recovers from errors. The reward is assigned when the task is completed successfully. Rollouts can include up to 64 environment turns.

Search agent

The search stage addresses difficult, multi-hop questions through live web-search calls in a browsing loop. The model gathers evidence, reasons over information from multiple search steps, and produces an answer. An LLM judge evaluates the final response because correctness is open-ended.

Alignment with RLHF

Every model finishes with RLHF for human preference and safety. The stage optimizes against a generative reward model for preference and a safety reward covering jailbreak resistance and appropriate refusals.

RLHF uses the pipeline's highest KL penalty. It also applies a reasoning-length penalty to discourage excessively verbose reasoning acquired during earlier stages.

Differences Between Model Sizes

The 3B model receives foundational RL and alignment. The 8B and 30B models add agentic RL for tool use in real environments.

Infrastructure for Scalable RL

Granite 4.2's RL system uses NeMo-RL for training and NeMo-Gym for rollouts. This infrastructure supports simultaneous policy training and live environments, including code repositories, terminals, and web-search tools.

NeMo-RL

• Megatron-Core provides the training backend.
• vLLM generates rollouts.
• Megatron-Bridge converts weights between Megatron and Hugging Face formats.

Each RL stage can therefore export a Hugging Face checkpoint for the next stage.

NeMo-Gym

NeMo-Gym exposes environments as Resources, including verifiers, tools, sandboxes, and reward models. SWE repositories, terminal harnesses, and web-search tools use the same general interface as a mathematical verifier.

Generation and policy updates run on separate GPU pools. This allows generation workers and live agentic environments to remain active while optimizer steps occur.

Results

Granite 4.2 was evaluated on agentic coding, general agentic and tool use, reasoning, chat and instruction following, and long-context tasks.

Supported languages are English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.

Quantization

Four quantized variants were released for inference with vLLM. FP8, NVFP4, and MXFP4 versions are produced with LLM Compressor. GGUF versions are created with the llama.cpp framework.

FP8

The FP8 version uses dynamic per-channel weight quantization and per-token activation quantization. No calibration is used.

FP4

NVFP4 and MXFP4 versions use GPTQ calibration on 2K samples from the SFT dataset. Calibration uses a maximum context length of 2K.

GGUF formats

Available GGUF formats include:

• Q8_0
• Q6_K
• Q5_K_S
• Q5_K_M
• Q5_1
• Q5_0
• Q4_K_S
• Q4_K_M
• Q4_1
• Q4_0
• Q3_K_S
• Q3_K_M
• Q3_K_L
• Q2_K

Training Infrastructure

Hardware

Training took place on an NVIDIA GB200 NVL72 cluster hosted by CoreWeave. The system includes:

• A 72-GPU NVLink domain for high-speed intra-rack communication
• A non-blocking Fat-Tree NDR 400 Gb/s InfiniBand fabric for inter-rack connectivity
• Thousands of GPUs operating at cluster scale

Software stack

Training software is packaged in .sqsh container images that provide reproducible environments with SBSA-compatible CUDA targets, Linux aarch64 Python wheels, and pinned GPU-specific binaries.

Large-scale SFT runs use an NGC PyTorch base image with Ubuntu 22.04, CUDA 12.8, and Python 3.12. RL runs use a separate NeMo-RL container.

Getting Started with Transformers

Installation

pip install torch
pip install accelerate transformers

Thinking mode

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "ibm-granite/granite-4.2-3b"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="cuda",
    torch_dtype=torch.bfloat16,
)
model.eval()

messages = [
    {"role": "user", "content": "How many r's are in the word 'strawberry'?"},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=8192,
        temperature=1.0,
        top_p=0.95,
        do_sample=True,
    )

print(tokenizer.decode(
    output[0][inputs.input_ids.shape[-1]:],
    skip_special_tokens=False,
))

A response may include a <think> section followed by the final answer:

<think>
Okay, let's see. The problem is to find how many 'r's are in the word 'strawberry'.

First, I need to write out the word: s t r a w b e r r y.

There are three 'r' letters.
</think>
There are 3 r's in the word "strawberry".
<|im_end|>

Non-thinking mode

messages = [
    {"role": "user", "content": "What is the capital of France?"},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=1.0,
    top_p=0.95,
    do_sample=True,
)
print(tokenizer.decode(
    output[0][inputs.input_ids.shape[-1]:],
    skip_special_tokens=False,
))

The output can be:

<think></think>The capital of France is Paris.<|im_end|>

Low-effort thinking

messages = [
    {"role": "user", "content": "What is 2 + 2?"},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
    low_effort=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=1.0,
    top_p=0.95,
    do_sample=True,
)
print(tokenizer.decode(
    output[0][inputs.input_ids.shape[-1]:],
    skip_special_tokens=False,
))

Tool calling

Tools use the OpenAI function definition schema. Granite can reason about which tool to call before producing the call.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather for a specified city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "Name of the city",
                    }
                },
                "required": ["city"],
            },
        },
    }
]

messages = [
    {"role": "user", "content": "What's the weather like in Boston right now?"},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    tools=tools,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=1.0,
    top_p=0.95,
    do_sample=True,
)
print(tokenizer.decode(
    output[0][inputs.input_ids.shape[-1]:],
    skip_special_tokens=False,
))

A tool call uses Granite's tool-call structure:

<think>
The user is asking for the weather in Boston. I should call get_current_weather with Boston as the city.
</think>
<tool_call>
<function=get_current_weather>
<parameter=city>
Boston
</parameter>
</function>
</tool_call>
<|im_end|>

A tool response can then be added to the conversation:

messages = [
    {"role": "user", "content": "What's the weather like in Boston right now?"},
    {
        "role": "assistant",
        "content": "<think>\nThe user wants to know the current weather in Boston. I should call get_current_weather.\n</think>",
        "tool_calls": [
            {
                "function": {
                    "name": "get_current_weather",
                    "arguments": {"city": "Boston"},
                }
            }
        ],
    },
    {
        "role": "tool",
        "content": '{"temperature": "72°F", "condition": "Partly cloudy", "humidity": "65%"}',
    },
]

Multi-turn conversations

By default, previous thinking can be removed to preserve context:

messages = [
    {"role": "user", "content": "What is 15 * 37?"},
    {
        "role": "assistant",
        "content": "<think>\n15 * 37 = 15 * 30 + 15 * 7 = 450 + 105 = 555\n</think>\n15 * 37 = 555",
    },
    {"role": "user", "content": "Now divide that by 5"},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
    truncate_history_thinking=True,
)

To retain the complete previous reasoning, set truncate_history_thinking=False.

Thinking and final-answer sections can be separated with a regular expression:

import re

def parse_model_output(text):
    """Separate thinking content from final answer."""
    think_match = re.search(r'<think>(.*?)</think>', text, re.DOTALL)
    if think_match:
        thinking = think_match.group(1).strip()
        answer_start = text.find('</think>') + len('</think>')
        answer_end = text.find('<|im_end|>', answer_start)
        answer = (
            text[answer_start:answer_end].strip()
            if answer_end != -1
            else text[answer_start:].strip()
        )
    else:
        thinking, answer = "", text.strip()
    return thinking, answer

Agentic Coding Harnesses

Because Granite 4.2 supports reasoning and tool calling through an OpenAI-compatible API, it can be connected to agentic coding harnesses after starting a vLLM server.

OpenCode

OpenCode is a terminal-based AI coding agent.

Install it with:

curl -fsSL https://opencode.ai/install | bash

Configure ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "model": "local/granite-4.2-30b",
  "provider": {
    "local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "vLLM (local)",
      "options": {
        "baseURL": "http://localhost:8000/v1",
        "apiKey": "EMPTY"
      },
      "models": {
        "granite-4.2-30b": {
          "name": "Granite 4.2 30B",
          "limit": {
            "context": 131072,
            "output": 8192
          }
        }
      }
    }
  }
}

Run OpenCode with:

opencode
opencode run "your task description"

Pi

Pi is a terminal-based agent harness that supports custom providers through models.json.

curl -fsSL https://pi.dev/install.sh | sh

A vLLM provider can be configured in ~/.pi/agent/models.json:

{
  "providers": {
    "vllm": {
      "baseUrl": "http://localhost:8000/v1",
      "api": "openai-completions",
      "apiKey": "EMPTY",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [
        {
          "id": "granite-4.2-30b",
          "name": "Granite 4.2 30B",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 131072,
          "maxTokens": 8192,
          "samplingParams": {
            "temperature": 1.0,
            "top_p": 0.95
          },
          "cost": {
            "input": 0,
            "output": 0,
            "cacheRead": 0,
            "cacheWrite": 0
          }
        }
      ]
    }
  }
}

Start Pi with:

pi

The granite-4.2-30b model can then be selected with /model or Ctrl+L.

OpenHands

OpenHands is an AI software-engineering environment that can plan, write code, and execute commands.

After installing OpenHands, configure the LLM with:

• Model: granite-4.2-30b
• Base URL: http://localhost:8000/v1
• API key: The value specified by the vLLM --api-key option

The openai/ prefix is required when connecting to OpenAI-compatible endpoints such as vLLM.

Categories

• AI/ML
• Programming Languages