OpenFrontiers Launches Massive Corpus of Open Research Problems and a Benchmark for Scientific Rediscovery — Artificial Intelligence

Researchers have introduced OpenFrontiers, a new cross-disciplinary resource featuring a corpus of over 12,000 open research problems and a specialized benchmark designed to evaluate large language models on scientific rediscovery. The initiative addresses the lack of comprehensive datasets for testing AI on unsolved questions by combining verbatim problem statements with a rigorous evaluation protocol using Lean 4 formal proofs.

A new resource titled OpenFrontiers has been released to address the challenge of evaluating large language models (LLMs) on research-level problem solving. While LLMs have demonstrated increasing promise in theoretical science, there has been a lack of comprehensive collections of open problems and reliable methods for grading them. OpenFrontiers combines a structured corpus of unsolved questions with a retrospective benchmark of recently resolved mathematical problems to measure the ability of AI systems to rediscover scientific solutions.

The core of the project is a cross-disciplinary corpus containing 12,151 problem records sourced from 62 different collections across ten top-level fields and 32 sub-fields. The dataset preserves verbatim statements, context, progress updates, and licensing information. Mathematics dominates the collection with 93.4% of the records, followed by physics, computer science, and machine learning. Approximately half of the records are labeled as open, while 46% are marked as unassessed to reflect uncertainty in their current status rather than assuming they remain unsolved. Additionally, 1,946 records include attached Lean 4 statements.

To facilitate evaluation, the researchers developed OpenFrontiers-Bench, a subset of 200 research-level mathematics problems. This benchmark includes 110 problems that were resolved in 2026 and 90 that remain open. Each problem features a Lean 4 statement verified by automated probes and two independent reviews. A high-quality subset known as HQ-50 further filters these items based on quality scores. The evaluation protocol grades formal proofs using two independent checkers within an isolated sandbox, testing models under closed-book, frozen-retrieval, and live-web conditions.

Initial experiments evaluated seven LLM agents on the HQ-50 subset under a standardized budget. The results indicate that closed-book rediscovery is rare, with agents solving at most seven of 40 resolved problems, mostly those involving short constructions. In contrast, success rates rose to up to 27 when live-web access was permitted, though this primarily reflected the retrieval of published solutions rather than independent discovery. Notably, no open problem was solved without accessing external information. The dataset and benchmark are available on Hugging Face for further research into scientific rediscovery.

Sources: arXiv cs.AI.