Hugging Face's 66 MB test model flagged a Transformers bug that sat in main for two months

Hugging Face has proposed Model Integration Contracts, small CPU-friendly test packages for its Transformers library, after a replay showed one of them would have flagged a UMT5 bug that stayed in the main branch from July 30 to September 28, 2026. The contract's model weighs 66 MB, against 1,227 MB for the real checkpoint, according to a November 3 post by Hugging Face engineer Tarek Ziadé.

The pitch is simple: Transformers now covers more than 750 model types, and running every real checkpoint on every pull request is not practical. A tiny model that keeps the original's structure can be checked in seconds.

A bug that crashed nothing

The regression arrived on July 30, when Transformers switched T5 to PyTorch's scaled dot-product attention, known as SDPA. As a side effect, the UMT5 decoder could look at future tokens, which meant that in training it could see the answer it was supposed to predict.

"Nothing crashed, and the bug stayed in main for almost two months," the post says. A fix landed on September 28.

When the team replayed frozen contracts across five months of commits, the UMT5 fixture flagged the problem on every commit between the change and the fix. Label log-probabilities drifted 1.22 away from the reviewed baseline, and training loss moved by about 0.20.

There was a catch inside the catch. The contract only noticed because it ran SDPA, the library's default attention path. A run limited to the older eager attention code missed the bug entirely.

How a 108.6 billion parameter model becomes a test

Each contract bundles three things: a shrunken fixture model, a runnable usage example and baseline outputs that a person has reviewed. Fixtures keep the real tokenizer and processor files and every kind of layer, but use fewer and narrower ones.

The most dramatic example in the post is Llama 4 Scout, cut from 108.6 billion parameters to 44.7 million. Its vision stack drops from 34 layers to 2 and its language stack from 48 layers to 4. Running the T5 contract on a CPU took 13.4 seconds once setup was done.

Random weights are not always enough. In a Whisper study, fixtures with random weights barely reacted when the audio changed and caught 18 of 24 injected faults. Fixtures overfit on a few tones paired with words like "red" and "green" caught all 24. The authors note the faults were synthetic. Neither kind of fixture spotted a real assisted-decoding bug, which only showed up with the full Whisper checkpoint.

The LLM drafts, a person approves

A command-line tool called tmic builds the contracts. When a new model is added, a language model writes a first version of the example code and manifest, then test runs on CPU and GPU propose baseline values. Any changed expectation then needs a human review, and the local tool stops before uploading anything.

As of November 1, 2026, the contract check in Transformers CI flags failures but does not stop a pull request from merging. The team says it still needs data on how many real bugs it catches on live pull requests, how many false alarms it raises and how much review work the baselines demand before deciding whether it should gate merges.