Hugging Face has proposed Model Integration Contracts (MICs), a way to run lightweight checks on Transformers models during pull requests. The aim is to catch regressions before they reach the main codebase, without needing to run the full integration suite on GPUs for every change.
The proposal follows a bug involving T5 attention. On July 30, 2026, Transformers enabled PyTorch’s scaled dot-product attention for T5, which also allowed UMT5’s decoder to attend to future tokens during training. The issue remained in the main branch until it was fixed on September 28. When the team replayed the intervening commits, a small UMT5 fixture running on CPU flagged the regression.
Transformers supports more than 750 model types. Its main branch has over 88,000 CPU tests, while a daily run includes more than 200,000 GPU tests. Full integration tests load real checkpoints and exercise models on real hardware, but running all of them on every pull request would require GPUs to remain busy throughout the day. Scheduled tests can also find a problem only after it has reached the main branch.
MICs address that trade-off by testing smaller fixtures built from real checkpoint configurations. These retain the different types of layers in a model but use fewer, narrower layers; tokenizer and processor files remain unchanged. The fixtures are published once and reused in fast integration tests on CPU. In the UMT5 study, the fixture’s weights took 66 MB, compared with 1,227 MB for the source checkpoint. The approach can also cover very large models that do not fit on standard GPU runners.
Each contract has three parts: a pinned fixture, a runnable example of a model task, and output baselines recorded from a known-good run and reviewed by a person. The check passes when the example run on the fixture matches those baselines. A contract is tied to an architecture and a usage path rather than one checkpoint, allowing compatible checkpoints to share a fixture while retaining their own expected results.
Examples can test more than whether a model loads and completes a forward pass. A SmolLM contract, for instance, generates text from a prompt and compares the resulting token IDs with a recorded answer. It also checks training loss, verifies that gradients are finite, and confirms that an optimizer step changes the weights. On pull requests, CI selects contracts according to the files changed.
The UMT5 replay showed why the tested path matters: label log-probabilities differed from the reviewed baseline by 1.22, and training loss by about 0.20. The contract used SDPA, the default attention path; testing only the eager attention variant would have missed the issue. Reverting the fix in a Transformers CI proof branch reproduced the failure, with the selected T5 contract’s runner step taking 13.4 seconds on CPU, excluding setup.
The fixture-building recipe has also been adapted for mixed architectures. It reduces model dimensions while retaining different layer types, including combinations such as sliding and full attention or dense and mixture-of-experts layers. It preserves preprocessing assets, including vocabulary and special tokens, and keeps the original weight-file format so that checkpoint conversion code is still exercised. Builds are validated for responsive outputs and finite results, then frozen with reviewed baselines for supported platforms and precisions.
The scale of the reduction is illustrated by a Llama 4 Scout fixture: 108.6 billion parameters become 44.7 million, while retaining the checked layer types and MoE routing style. The command-line tool tmic creates contracts and fixtures. For new models, an LLM can draft the usage code and manifest, but people must review changed expectations. The local workflow can build and test a contract without uploading it.
The team also compared random fixture weights with weights trained on a few examples. In a study using synthetic faults, random Whisper fixtures caught 18 of 24 injected issues, while fixtures trained on a small set of examples caught all 24. The authors caution that this does not establish which approach is better in practice: the tests used synthetic faults, and both versions missed an assisted-decoding bug exposed by the real Whisper checkpoint. The current approach keeps random weights when they pass validation and uses overfitting when greater sensitivity or readable outputs are needed.
As of November 1, 2026, the MIC job in Transformers CI reports failures but does not block pull requests. The team says it still needs to measure how many real bugs the checks catch, how often they produce false alarms, and how much work baseline reviews require. Coverage remains partial, and the project plans to add usage paths when regressions reveal gaps. Existing tests will remain until their coverage is accounted for.
Sources: Hugging Face Blog.