Life After Benchmark Saturation: A Case Study of CORE-Bench
Siegel, Arvind Narayanan View a PDF of the paper titled Life After Benchmark Saturation: A Case Study of CORE-Bench, by Nitya Nadgir and 13 other authors View PDF HTML (experimental) Abstract:When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version. We use CORE-Bench Hard, a benchmark for computational reproducibility of scientific code, as a case study to demonstrate that measuring agents along these dimensions yields meaningful insights into agent performance even after accuracy saturates. First, we surface threats to construct validity in CORE-Bench Hard that are difficult to anticipate with less capable agents.
- ▪Siegel, Arvind Narayanan View a PDF of the paper titled Life After Benchmark Saturation: A Case Study of CORE-Bench, by Nitya Nadgir and 13 other authors View PDF HTML (experimental) Abstract:When a benchmark's accuracy saturates, it is oft
- ▪We use CORE-Bench Hard, a benchmark for computational reproducibility of scientific code, as a case study to demonstrate that measuring agents along these dimensions yields meaningful insights into agent performance even after accuracy satu
- ▪First, we surface threats to construct validity in CORE-Bench Hard that are difficult to anticipate with less capable agents.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2606.26158 (cs) [Submitted on 23 Jun 2026] Title:Life After Benchmark Saturation: A Case Study of CORE-Bench Authors:Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, Matilda Orona, Stephan Rabanser, Tilman Bayer, Abhishek Shetty, Yue Ling, Derrick Chan-Sew, Rumi Nakagawa, Saiteja Utpala, Zachary S. Siegel, Arvind Narayanan View a PDF of the paper titled Life After Benchmark Saturation: A Case Study of CORE-Bench, by Nitya Nadgir and 13 other authors View PDF HTML (experimental) Abstract:When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv.org.
Discussion
0 commentsMore from arXiv.org
