A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
The article introduces A2RBench, an automated system designed to generate benchmarks for evaluating abstract reasoning in large language models (LLMs). It highlights the challenges of current benchmarks, which often rely on manual annotation or fail to measure genuine reasoning. The authors present a theoretical framework that ensures unique solutions through programmatic verification, revealing significant deficiencies in LLMs' abstract reasoning capabilities compared to humans.
- ▪A2RBench is an automated pipeline for generating and evaluating abstract reasoning benchmarks.
- ▪Current LLMs show significant underperformance in abstract reasoning compared to human benchmarks.
- ▪The study finds that LLMs struggle with high-dimensional tasks and that higher information complexity can simplify reasoning.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2605.17278 (cs) [Submitted on 17 May 2026] Title:A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation Authors:Qingchuan Ma, Yuexiao Ma, Yongkang Xie, Tianyu Xie, Xiawu Zheng, Rongrong Ji View a PDF of the paper titled A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation, by Qingchuan Ma and 5 other authors View PDF Abstract:Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
