Transformers Model Integration Contracts

Transformers Model Integration Contracts

Published November 3, 2026

On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, the UMT5 decoder could attend to future tokens during training, allowing it to see the answer it was meant to predict. Nothing crashed, and the bug remained in the main branch for almost two months.

When the affected commits were replayed later, a small UMT5 model running on a CPU detected the problem in every case. This article describes that model and the tests built around it.

Why bugs reach the main branch

Transformers supports more than 750 model types. Its test suite includes more than 88,000 CPU tests on main and more than 200,000 GPU tests in a daily run. Model tests are only one part of the suite, which also covers pipelines, training, distributed execution, quantization, and attention kernels.

The integration tests load real checkpoints and exercise model APIs on real hardware. Running all of these tests for every pull request would require keeping many GPUs active throughout the day, which is impractical. A scheduled run may therefore find a regression only after it has already reached the main branch.

The alternative is to reduce the model size. A fixture is a small model created from the configuration of a real checkpoint. It uses fewer and narrower layers while retaining every type of layer present in the original model. Its tokenizer and processor files remain unchanged. After being published once, the fixture can be reused in fast integration tests on an inexpensive CPU runner.

In a CPU study, the experimental UMT5 fixture used 66 MB of weights, compared with 1,227 MB for the source checkpoint. This approach also makes it possible to test very large models that do not fit on standard GPU runners, because their fixtures can run on CPUs.

Transformers model integrators already create small test checkpoints, such as the tiny-random-* models. The goal of Model Integration Contracts is to provide a common recipe and checks that go beyond loading a model and performing a forward pass. The UMT5 regression would have passed a basic forward-pass test.

Fixtures do not replace full integration tests. They provide an additional opportunity to detect regressions during pull-request testing, before those regressions reach the main branch.

Introducing Model Integration Contracts

transformers-mic proposes a standard format for packaging these tests. A Model Integration Contract, or MIC, contains three parts:

• A fixture, consisting of the reduced model described above, published once and pinned.
• A usage path, which is a runnable Markdown example of a typical model operation, such as text generation or a training step.
• Baselines, which are the outputs recorded when the usage path is run and reviewed by a person.

A contract is considered valid when running the usage path on the fixture continues to produce the recorded baselines.

A contract covers an architecture and a usage path rather than one specific checkpoint. Compatible checkpoints can share a fixture, while each checkpoint retains its own expected results. Contracts are published in huggingface-mic.

For example, the SmolLM contract generates text from prompts such as The capital of France is and compares the resulting token IDs with a recorded answer. After the model and tokenizer are loaded, the simplified generation code for one prompt is:

inputs = tokenizer("The capital of France is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=4, do_sample=False)
new_tokens = outputs[:, inputs["input_ids"].shape[1]:]
print(tokenizer.batch_decode(new_tokens, skip_special_tokens=True))

The contract also compares training loss, checks that gradients are finite, and verifies that an optimizer step changes the model weights. The complete example is available in usage.md for the SmolLM contract.

A fleet dataset pins each MIC to a frozen revision and records the package versions used to produce its baselines. Each CI run records the fleet revision it used, allowing the run to be reproduced later.

For a pull request, MICs are selected according to the files changed by the patch.

The UMT5 regression

Frozen MICs were replayed over five months of Transformers commits. The small UMT5 fixture detected the regression from its introduction on July 30, 2026 until its fix on September 28.

Label log-probabilities differed from the reviewed baseline by 1.22, while training loss differed by approximately 0.20.

The attention implementation used by the contract was important. It used SDPA, the default attention path. Running only the eager attention variant did not expose the problem. A fixture is useful only when it follows the same code path as the real checkpoint.

The fix was then reverted on a proof branch in Transformers CI. The check failed with the same differences, and the selected T5 contract's runner step took 13.4 seconds on a CPU, excluding setup.

How models are shrunk

The initial shrinking method reduced dimensions and retained fewer layers. That was insufficient for mixed architectures. For example, retaining two layers is not useful if both use sliding attention and all full-attention layers are removed.

The current process has four steps:

1. Shrink the configuration. Start with a real checkpoint and reduce hidden sizes, attention heads, and depth. Preserve different layer types, including combinations of sliding and full attention or dense and mixture-of-experts layers. Some architectures require additional rules to keep their components compatible. The model's existing Transformers tests are also consulted. The ModelTester in test_modeling_<model>.py often defines a small configuration manually, so those sizes are used when a generic rule does not recognize a particular width.

2. Keep preprocessing unchanged. Preserve the vocabulary, special tokens, and input dimensions. Tokenizer and processor files are copied byte for byte because loading and preprocessing are part of the behavior being tested.

3. Build and validate the weights. Begin with seeded random weights and vary the inputs to check whether the model responds. If it does not, train it on the contract's small batch of examples. Reload the saved model and verify that outputs, loss, and gradients are finite. Failed builds are rejected.

4. Freeze and record. Save the accepted weights and measure repeatable outputs on a known-good Transformers build. Review baselines for each supported platform and precision, then publish a pinned snapshot. CI reuses that snapshot instead of rebuilding the fixture for every pull request.

The original weights file format is retained as well. This ensures that conversion code executed when a user loads the real checkpoint is not skipped.

The Llama 4 Scout shrink report demonstrates the process on a large model. Its 108.6 billion parameters are reduced to 44.7 million. The vision stack is reduced from 34 layers to 2, and the language stack from 48 layers to 4, while preserving the checked layer types and mixture-of-experts routing style.

Building MICs with agents

The tmic command-line tool creates a contract and builds its fixture. For a new model, an LLM drafts the usage code and manifest, while CPU and GPU runs measure candidate baselines. The LLM produces the draft but does not approve it. Changes to expected results require human review.

A local workflow can use a source checkpoint in place of org/model:

pip install "transformers-mic[runtime]" "hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git"
hf auth login
tmic run org/model --local-only --fixtures-only --no-history --bill-to ''

This workflow drafts the contract, builds the fixture, and records baselines locally. It asks for review at each step and stops before uploading anything. For a new model, the draft is produced by an LLM through Inference Providers and billed to the user's own account with --bill-to ''.

Agents were used to build MICs across model families. When an edge case was found, the recipe itself was updated rather than only modifying an individual fixture, with assistance from model experts. Published models and remaining gaps are listed in the coverage space.

Random weights versus overfitting

With random weights, the Whisper fixture responded only weakly to changes in audio. It ran successfully, but it was not a strong test of the audio path.

The fixture was therefore trained on a few tones paired with words such as red and green. The purpose was not to teach speech recognition, but to make audio changes visible in the outputs. In a weight-initialization study, random Whisper fixtures detected 18 of 24 injected faults, while overfit fixtures detected all 24.

There are two ways to build fixture weights:

• Start with random weights and a fixed seed.
• Overfit on a few examples, such as The capital of France is producing Paris.

The Whisper result does not establish that overfitting is generally better. The faults were synthetic, and no real regression has yet been found that was detected only by the overfit version. Both versions missed an assisted-decoding bug that was exposed by the real Whisper checkpoint.

For this reason, random weights are retained when they pass the checks. Overfitting is used when greater sensitivity or readable answers are needed. Training takes seconds, but it introduces another possible convergence failure and another maintenance requirement.

Open questions

As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking pull requests. Whether it should block them depends on several measurements that are not yet available:

• How many real bugs does it detect in pull requests rather than in replays?
• How often does it produce false alarms?
• How much time do baseline reviews require?

It is also not known how many bugs, such as the Whisper assisted-decoding issue, appear only with real weights, or whether overfitting detects real bugs that random weights miss.

Coverage remains partial across model families and the APIs used with them. When a regression is missed, the missing usage path can be added and replayed against broken and fixed versions. Existing tests will remain until their coverage has been accounted for.

Another planned connection is between original Hub model pages and their MICs. This would allow a developer or an agent using a checkpoint to begin with an example tested against that checkpoint, its environment, and its expected outputs.