Transformers Adds Support for llama.cpp GGUF Quantized Models

Transformers Adds Support for llama.cpp GGUF Quantized Models

Hugging Face is adding support for running GGUF models efficiently in transformers. This lets users select checkpoints sized for their laptop's available memory, load them through the familiar transformers APIs, and generate text locally.

Users can choose a GGUF file from the Hub, load it with from_pretrained, and run inference on their own machine. The initial implementation focuses on local inference on Apple Silicon, beginning with the Qwen3.5 architecture.

GGUF and local inference

Local model execution has become more practical in part because of llama.cpp, whose inference engine powers tools including Ollama, LM Studio, and Jan. Alongside projects such as MLX, it has helped make local inference accessible for everyday workloads.

GGUF, developed by the llama.cpp team, packages model weights and metadata in a single file. This can include tokenizer information and an optional chat template. GGUF also supports multiple quantization levels, allowing users to trade some precision for a smaller memory footprint.

The llama.cpp team publishes quantized checkpoints through ggml-org on the Hub. Other publishers, including Unsloth, LM Studio Community, and bartowski, provide GGUF checkpoints in several quantization formats. These models have been downloaded millions of times.

To make the transformers implementation practical, the project reuses ggml kernels through the kernels library and reduces overhead in generate.

How GGUF quantization affects file size

Quantization reduces the precision used to store model weights. Variants such as Q4_K_M combine tensor precisions, using mostly 4-bit weights while retaining higher precision for more sensitive tensors.

The following examples show file sizes for Unsloth's Qwen3.5-4B:

Q4_K_M is a reasonable starting point for local inference. Users with additional memory can try Q5_K_M or Q6_K. More aggressive quantization can help larger models fit into memory, but the quality impact depends on the model and task. The Hub's GGUF documentation describes the available quantization types.

Loading GGUF with Transformers

The initial setup requires:

• An Apple Silicon Mac.
• A PyTorch version supported by the published ggml-quantization kernel builds, usually one of the two latest PyTorch releases.
• The latest version of transformers, currently from the main branch, and a compatible version of kernels.

pip install -U "git+https://github.com/huggingface/transformers.git" kernels

To load a GGUF model, pass the Hub model_id and filename through gguf_file:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    gguf_file=filename
)

When the weights remain packed on Metal, transformers automatically loads compatible ggml and Metal layer kernels and uses ggml-org/ggml-attn for attention. If that kernel cannot be fetched, the model falls back to "sdpa" with a warning. It is also possible to select "sdpa" explicitly with attn_implementation="sdpa".

After loading, generation uses the standard transformers API:

messages = [{"role": "user", "content": "Explain why the sky is blue in a few sentences."}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=256)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Without a compatible quantization kernel, the loader dequantizes the model, which requires more memory.

Serving GGUF through an OpenAI-compatible API

The same checkpoint can be served with transformers serve:

pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels

transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

The model argument follows the format <model_id>:<filename>.gguf. In this example, unsloth/Qwen3.5-4B-GGUF identifies the Hub repository and Qwen3.5-4B-Q4_K_M.gguf selects the file within it.

For models with chat templates that support reasoning, the server accepts these options:

• --reasoning off disables reasoning.
• --reasoning on enables reasoning.
• --reasoning auto follows the chat template's default behavior.

Clients such as Jan and Pi can connect through a custom OpenAI-compatible provider with the following settings:

In this arrangement, transformers runs the model on the Mac while the client supplies the conversation interface. Other clients that support the same API can use the endpoint as well.

Performance comparison with llama.cpp

The reference point for local inference performance is llama.cpp. The comparison covers three GGUF checkpoints: a small dense model, a larger dense model, and a mixture-of-experts model.

The llama.cpp results use llama-bench, build 5f55650a7, release b10200, with the Metal backend from ggml 0.18.0. The command was:

llama-bench -m <file> -p 0 -n 128 -r 3

This reports tg128, the token-generation rate for 128 decoded tokens averaged over three repetitions, excluding prompt processing. The transformers measurement uses generate to produce the same 128 tokens from a 12-token prompt, with the best result from three warmed runs. It includes prefill.

Measurements were taken on a MacBook Pro M2 Max with 32 GB of unified memory, macOS 26.6, PyTorch 2.12.1, and kernels 0.17.0, while connected to power.

The benchmark script was:

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id, filename = "unsloth/Qwen3.5-4B-GGUF", "Qwen3.5-4B-Q4_K_M.gguf"

model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
inputs = tokenizer("The capital of France is Paris. The capital of Germany is", return_tensors="pt")
inputs = inputs.to(model.device)

with torch.inference_mode():
    model.generate(**inputs, max_new_tokens=8, min_new_tokens=8, do_sample=False)  # warm up
    torch.mps.synchronize()
    for _ in range(3):
        time.sleep(90)  # let the machine cool: back-to-back runs decay by 10% or more
        start = time.perf_counter()
        model.generate(**inputs, max_new_tokens=128, min_new_tokens=128, do_sample=False)
        torch.mps.synchronize()
        print(f"{128 / (time.perf_counter() - start):.1f} tok/s")

The corresponding llama.cpp command was:

llama-bench -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M -p 0 -n 128 -r 3

The results show transformers close to llama.cpp across all three checkpoints. The measurements do not use identical conditions: the transformers result includes prefill, while llama-bench reports decode-only throughput.

How Transformers and llama.cpp complement each other

When GGML and llama.cpp joined Hugging Face, the projects were described as having complementary roles. llama.cpp provides a foundation for local inference, while transformers provides a foundation for model definitions. GGUF support brings those roles closer together.

llama.cpp remains the recommended engine when efficient local inference is the primary requirement. Its runtime, memory management, and hardware support are designed for that purpose. GGUF support in transformers provides another way to work with the same checkpoints:

• Experiment with GGUF in Python and PyTorch. Developers can inspect intermediate activations with hooks, modify a model's forward pass, and prototype custom layers.
• Evaluate GGUF models. Existing transformers evaluation workflows can measure the quality of quantized checkpoints.
• Validate GGUF conversions. Loading an original checkpoint and its GGUF conversion makes it easier to verify that weights were converted correctly while accounting for quantization error.
• Test decoding methods. Custom logits processors and stopping criteria can be used with generate, or developers can write a custom generation loop in Python.
• Fine-tune from a GGUF checkpoint. The weights can be dequantized before continuing with a standard transformers training workflow.

For the last use case, load the model with GgufConfig(dequantize=True):

import torch
from transformers import AutoModelForCausalLM, GgufConfig

model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen3.5-4B-GGUF",
    gguf_file="Qwen3.5-4B-Q4_K_M.gguf",
    quantization_config=GgufConfig(dequantize=True),
    dtype=torch.bfloat16,
)

Applying ggml kernels beyond GGUF

The same kernel work can also bring ggml performance to models that llama.cpp does not support. transformers already contains PyTorch implementations for these architectures. With ggml kernels and quantization schemes available in PyTorch, supported operations can be accelerated without first implementing the entire model in llama.cpp.

This is relevant to new architectures, research models, and custom variants that may not receive dedicated llama.cpp implementations. The opportunity is not limited to GGUF files. Kernels operate on tensors, so the same building blocks can be integrated into other transformers models and loading workflows.

Compatible attention, normalization, and matrix multiplication kernels could also be reused by computer vision, audio, and multimodal models. Each architecture still requires integration and validation. The initial GGUF examples focus on text generation.

Fast local inference with Python and PyTorch

The implementation keeps the model and generation loop in Python while using specialized kernels for the expensive GPU operations. The kernels perform the heavy computation, and the generation loop avoids unnecessary synchronization so the GPU can remain active.

The work targets fast eager execution without requiring torch.compile. For interactive use, this avoids compilation pauses and recompilation when input shapes change. The two main areas are the kernels and the generate loop.

Reusing ggml's Metal kernels

A kernel is a small program that performs an operation on the GPU. PyTorch provides general-purpose implementations, while specialized kernels can reduce work, combine operations, or read quantized weights directly in their stored format.

The kernels library distributes compatible builds of ggml's Metal kernels on the Hub and makes them available to transformers models.

The first four packages build on ggml kernels. The top-k kernel addresses a separate bottleneck in mixture-of-experts routing. Together, these kernels reduce the GPU work required for each generated token.

To measure the contribution of the layer kernels, the same packed GGUF checkpoints were compared with and without them. The quantization kernel remained enabled in both configurations, because disabling it would also change the weight representation and measure a different tradeoff.

Coordinating CPU and GPU work

Faster kernels are useful only when the GPU has work queued. During generation, the CPU schedules GPU operations and controls the loop that produces the next token. Reading a result from the GPU can force the CPU to wait for queued operations to finish. Repeating even a small wait for every token can reduce throughput.

Two changes to generate address this issue and improve generation for all transformers models, not only GGUF models:

• Drop an unnecessary attention mask early (#48814). When a supported decoder-only input has no padding, its all-ones padding mask can be removed at the start of generation. Downstream attention code no longer needs to repeatedly inspect the mask to determine whether it can be skipped. Causal attention remains unchanged.
• Defer the stopping check (#47975). On supported paths, generate copies the stopping decision asynchronously and consumes it during the following step. This allows the CPU to continue scheduling work while the GPU runs. Streaming tokens use the same approach, and any extra step after the stopping condition is removed from the output.

These changes reduce overhead around the model and therefore apply beyond GGUF. They complement the kernel work: specialized kernels reduce the cost of individual operations, while fewer synchronization points allow CPU scheduling and GPU execution to overlap.

Current limitations and next steps

The initial target is a single interactive conversation on Apple Silicon. Current limitations include:

• The packed inference path is MPS-only. GGUF import through dequantization remains a separate option. Support for the file format does not mean packed kernels are available on every device.
• Padding and batching need additional work. Unpadded inputs benefit from the mask optimization. Padded batches cannot use the same shortcut and may have lower performance. Support for generate_batch on MPS is planned for future work.
• Architecture coverage is limited. The packed loader currently supports the Qwen3.5 dense and mixture-of-experts architectures, including compatible Qwen3.8 checkpoints. Additional architectures can be integrated progressively.

Acknowledgments

The project involved contributions from Arthur Zucker, Cyril Vallez, Sayak Paul, the llama.cpp team, Bertrand Chevalier, Aritra Roy Gosthipaty, Pedro Cuenca, and Lysandre Debut.