RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
The paper introduces RoadmapBench, a benchmark designed to evaluate long-horizon coding tasks in software development. It highlights the challenges faced by coding agents in handling multi-target development across multiple files and programming languages. The study reveals that current models struggle significantly with these tasks, indicating that long-horizon software development remains an unsolved issue.
- ▪RoadmapBench consists of 115 long-horizon coding tasks based on real open-source version upgrades.
- ▪The benchmark covers 17 repositories and 5 programming languages, requiring substantial modifications across multiple files.
- ▪Even the best-performing model, Claude-Opus-4.7, only resolves 39.1% of tasks, showcasing the difficulty of long-horizon software development.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Software Engineering arXiv:2605.15846 (cs) [Submitted on 15 May 2026] Title:RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades Authors:Xinbo Xu, Ruihan Yang, Haiyang Shen, Wendong Xu, Bofei Gao, Ruoyu Wu, Kean Shi, Weichu Xie, Xuanzhong Chen, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang View a PDF of the paper titled RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades, by Xinbo Xu and 15 other authors View PDF HTML (experimental) Abstract:Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
