Self-evolving language-model agents are designed to rewrite a persistent state after feedback, but a new arXiv paper argues that it has been largely untested whether this process lets an agent learn a rule nobody told it. The paper introduces CLR-Bench, a Curriculum of Latent Rules benchmark, to separate genuine rule induction from the simpler strategy of writing down each correction as an exception. In the benchmark, an agent that only keeps a growing list can pass practice tasks while missing unseen cases that the underlying rule should cover.
CLR-Bench is built from chains of aggregation tasks over realistic records, such as counting reimbursable flight segments by department or reviewable medication records by patient. Each chain adds one hidden rule per level on top of earlier rules. A rule operates on an opaque label that stands for a named grouping published on the Web, such as the founding members of an airline alliance. The task supplies one example value, practice feedback reveals one additional member and one non-member, and the test checks unseen members and similar non-members. For example, if the example is Lufthansa, practice may show that Thai Airways belongs to the same grouping and ITA Airways does not; the test then asks about United Airlines, which never appeared in feedback, while similar non-members must still be counted. The paper says success requires turning the evidence into the right category and storing that category in the agent's state.
The benchmark uses an automated pipeline to build and certify chains from a new data domain, with language-model probes checking that the base model knows the members of each grouping and does not guess it without feedback. Probes also certify that, under a named-grouping convention, the evidence identifies the grouping. Four other base models rarely guess the groupings either, according to the paper. The released set contains 20 chains over 10 domains, with four levels per chain, pairing each hidden rule with a practice task and a test task. In total, the benchmark includes 80 hidden rules, 80 practice tasks and 80 test tasks. The domains range from travel expense audits and pharmacy reviews to system logs, chemical imports and port state control.
The paper evaluates six recent self-evolution methods. With the certifying model, the methods barely learn: the best gain is at most 5 points over a frozen agent, and they exclude only 6–12% of unseen members from the count, fewer than the 20% reached by simply keeping the raw practice history. Their states hold the revealed member in 75–98% of cases and almost never the class name, the paper reports. When practice exposes a member that the list misses, the methods add that member to the list. A fixed lesson that states the named-grouping convention lifts the same methods by 14.5 to 16.5 points.
With the stronger GPT-6-Sol base model, ACE and Dynamic Cheatsheet learn from feedback alone, reaching 78% accuracy and solving 59–60% of the single cells, against 53% and 8% with the certifying model. They also surpass the raw-history reference. The paper says CLR-Bench therefore measures how well an agent learns and shows that this ability differs across models. Code and data are available at https://github.com/yausankit/clr-bench.
Sources: arXiv cs.AI.