Ai2's StateBench: frontier models fall to 13% or less when long texts swap variables

Ai2's new StateBench benchmark found that four frontier models, including Claude Opus 5.5 and GPT-6 Sol, did well when a long document only overwrote values, but fell to 13% accuracy or lower when variables were swapped and copied by reference over the longest inputs. Ai2 published the benchmark, dataset and code on November 3, 2026.

The surprise came from much smaller models. Recurrent and hybrid models with 275 million parameters, trained directly on the task, kept near perfect accuracy on three of the four levels out to tens of thousands of assignments, after training on sequences of no more than 256.

Four kinds of bookkeeping

StateBench measures state tracking: whether a model can follow entities whose values change across a long text. It has four levels, each tied to a type of assignment and grounded in Krohn–Rhodes theory from algebra.

The easiest level only asks for a value that never changes. The next asks for the latest value after reassignments. The third uses self-referential updates such as adding 5 to a variable, so the model has to carry a running total. The hardest uses assignment by reference and swaps, which the authors link to the S5 problem, a classic example of hard state tracking.

All four levels are built from the same underlying sequence, with the same number of variables and assignments at the same positions. Only the type of assignment changes, so any drop in accuracy comes from the bookkeeping itself.

Where the frontier models broke

Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna handled the two easier levels well. On the third and fourth levels, accuracy fell steeply as inputs grew longer. Ai2 suggests one reason is sensitivity: at the harder levels more steps can change the final answer, so a single missed update is more likely to make it wrong.

Ai2's own pretrained open models, two 7B models (Olmo 3, a pure transformer, and Olmo Hybrid, a version mixing in Gated DeltaNet layers) plus 275M and 2.7B versions, did well on the easiest level, lost ground on the second as the text got longer, and never rose above chance on the hardest.

Format alone could flip the result. The 275M hybrid and Gated DeltaNet models were near the ceiling on easy problems written in the style of the RULER benchmark, but near chance when the same problems were presented as spreadsheet edits.

Training beat size, except for transformers

Ai2 then trained 275M-parameter models from scratch on StateBench, using sequences of up to 256 assignments and testing them on up to 64,000. The Gated DeltaNet and hybrid versions came close to perfect scores on the first three levels out to tens of thousands of assignments. On the hardest level, the Gated DeltaNet version whose memory could be updated more freely stayed near perfect to about 10 times its longest training length.

Transformers trained the same way learned the easy levels but degraded on longer sequences, and never learned the hardest level at all. The authors read that as an architectural limit, while the pretrained recurrent models' failures point to how they were trained.

The benchmark has one built-in simplification: it uses a fixed set of variables, while real conversations keep introducing new entities. Ai2 says that case is not yet covered. The dataset holds about 68,000 rows.