A study by Stanford researchers Owen Queen and James Zou suggests that research findings about large language models can become outdated quickly. The pair tested 50 widely cited claims against 20 models released between July 2024 and September 2026 by eight developers.
The original effect had largely disappeared for 44% of the claims, while another 34% had weakened significantly. Only 16% continued to hold, and 6% had become stronger. Among 22 findings for which the researchers estimated an expiration date, the average lifespan was 15.7 months. The share considered expired rose from 16% after one year to 39% after two years.
Claims about what models cannot do were the quickest to lose support. Five of nine findings related to reasoning were judged expired, and none had held up or strengthened. They included difficulties counting letters in “strawberry,” performing arithmetic in unfamiliar number systems, retaining information from the middle of long contexts, and maintaining correct answers when users challenged them. The researchers suggest these errors may be easier to address because they have clear correct answers that developers can train models to produce.
Claims about preferences and tendencies proved more durable. Of the 11 findings that held or strengthened, several concerned models acting as evaluators: preferring their own answers, rating accepted and rejected papers similarly, and being influenced by persuasive language in medical abstracts. Other persistent findings involved agreeing with users, a left-leaning political bias, similar responses to open-ended questions, and uneven distributions when generating random numbers.
The researchers also revisited a 2023 study of fabricated scientific citations, in which GPT-3.5 invented 55% of its references. Using the same prompts, they found a rate of 0% for GPT-6.1 Sol and 1% for Claude Sonnet 5.5. However, Qwen3.7-Flash fabricated 69% of its references in the test, showing that results can differ sharply between models.
Some weaknesses have declined but remain detectable. Claude Sonnet 5.5 still failed on about 10% of two-hop questions when it correctly answered both component questions separately; the original paper reported a 43% failure rate for ChatGPT. The “Reversal Curse” and contradictory answers to paired questions about time have also weakened, but have not disappeared. Social bias findings were mixed: gender stereotypes and implicit association biases weakened, while the political bias persisted and biases against speakers of African American English shifted direction in the trend data.
Differences between models were substantial even among releases from the same period. For models launched between July and September 2026, the number of expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. The researchers found that release date, developer, and model size did not reliably predict the results.
Queen and Zou argue that empirical claims about LLMs should be framed as observations about particular models at a particular time, rather than as universal truths. They recommend that researchers record the model, version, and date used, and publish prompts and analysis code so findings can be tested again as models change.
Sources: The Decoder.