A new benchmark called LOLPlay tests whether visual AI agents can complete multi-step objectives in League of Legends while the game continues to run during their reasoning and actions. It includes 36 champion-specific tasks covering resource management, combat, cooperation and map objectives.
Agents interact using screenshots and mouse-and-keyboard controls. Each task has a defined starting setup and success conditions, while 82 executable subgoal checks record partial progress as well as completion. The benchmark uses live game data to evaluate outcomes, rather than relying on an agent’s own report that it has finished.
In 324 evaluations of three models at three reasoning-effort settings, GPT-6 Astra recorded the highest task success rate, 25.0%, at both low and medium effort. Its highest subgoal success rate was 40.3% at medium effort. The study reports that high effort improved neither task nor subgoal success over low effort for any of the models.
For Astra, longer requests left fewer interaction rounds, pointing to a potential trade-off between additional reasoning and timely control. The researchers also found examples of actions being carried out after the observed scene had changed, as well as runs that met some requirements but not enough to complete the task.
The task suite includes objectives such as reaching specified levels, buying items, surviving until a deadline, securing kills with allied bots and completing map objectives in a required order. Budgets range from five to 20 minutes, and gameplay remains active while agents respond and execute their actions.
Sources: arXiv cs.AI.