The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection
The article discusses the limitations of traditional information retrieval systems in the context of large language models (LLMs). It introduces a new measure called Bits-over-Random (BoR) to evaluate retrieval selectivity, revealing that high success rates can sometimes indicate random performance. The findings suggest a need to report BoR alongside traditional metrics to better assess retrieval effectiveness.
- ▪Traditional information retrieval systems were designed for human users who could filter irrelevant information.
- ▪The introduction of LLMs has changed the landscape, as they lack the ability to filter results effectively.
- ▪The Bits-over-Random (BoR) measure indicates that high success rates may mask random-level performance in retrieval systems.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Information Retrieval arXiv:2605.18857 (cs) [Submitted on 14 May 2026] Title:The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection Authors:Vyzantinos Repantis, Harshvardhan Singh, Tony Joseph, Cien Zhang, Akash Vishwakarma, Svetlana Karslioglu, Michael Wyatt Thot, Ameya Gawde View a PDF of the paper titled The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection, by Vyzantinos Repantis and 7 other authors View PDF HTML (experimental) Abstract:For most of the history of information retrieval (IR), search results were designed for human consumers who could scan, filter, and discard irrelevant information on their own.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
