RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
RepoMirage is a new evaluation suite designed to assess repository context reasoning in code agents. The study reveals significant performance drops when agents are required to access broader context information. The findings suggest that improving structure-aware methods could enhance the capabilities of these code agents.
- ▪RepoMirage introduces a two-stage evaluation suite to investigate repository context reasoning in code agents.
- ▪Performance declines significantly when agents face tasks requiring broader context access, dropping from 66.8% to 25.3%.
- ▪The study proposes RepoAnchor, a prototype workflow that separates repository exploration from problem-solving to improve performance.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Software Engineering arXiv:2605.26177 (cs) [Submitted on 25 May 2026] Title:RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations Authors:Hanyu Li, Yichi Zhang, Speed Zhu, Hang Su, Jun Zhu, Yinpeng Dong View a PDF of the paper titled RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations, by Hanyu Li and 5 other authors View PDF HTML (experimental) Abstract:Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects repository context reasoning, the ability to identify the task-relevant information across multiple files and reason over the relations among them.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
