StateBench is a new benchmark designed to measure how well language models track information about people, objects or variables as it changes across long documents. Its creators evaluate four levels of difficulty, from retrieving an unchanged value to following chains of updates in which one variable can take another’s current value.
The benchmark reflects a range of practical challenges, from following variable assignments in code to tracking where characters and objects move in a story. Some tasks require only finding the latest update. More demanding ones require carrying earlier changes forward, so missing a single step can lead to an incorrect final answer.
In tests of open pretrained models, most performed well on the simplest tasks but lost accuracy as inputs grew on problems involving reassignment. They generally remained at chance on the most complex tasks. Results also depended on how a problem was presented: some models that performed strongly on one format were near chance when the same underlying task was framed as spreadsheet edits.
The researchers also tested Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. The models handled the two easier levels well, but accuracy fell sharply on tasks requiring cumulative or reference-based updates. On the longest general finite tasks, all four scored 13% or less.
To examine whether these difficulties stem from model architecture or pretraining, the researchers trained 275-million-parameter models from scratch on StateBench. The models learned from sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments.
Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, including sequences far longer than those used in training. Transformers degraded on longer sequences and did not learn the hardest level. A Gated DeltaNet variant with a more flexible memory update performed best on that level, remaining near perfect at roughly ten times the longest training sequence.
The findings suggest that direct training can help hybrid and recurrent models develop state-tracking skills that were absent in their pretrained versions. The researchers say transformers, by contrast, did not learn the hardest tasks even with targeted training.
StateBench currently assumes a fixed number of variables, so it does not cover settings where new people or objects appear over time. The researchers also identify open questions about the capabilities of different Gated DeltaNet variants and how numerical precision or attention layers may affect them.
Sources: Hugging Face Blog.