Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
The Insights Generator (IG) is a new multi-agent system designed to diagnose failures in LLM agents by analyzing execution traces. It aims to produce grounded natural-language insights that reveal systematic behavioral patterns across trace groups. Evaluation shows that IG significantly improves performance and provides high-quality evidence in its reports.
- ▪Diagnosing failures in LLM agents has traditionally been a manual process.
- ▪The Insights Generator formalizes corpus-level trace diagnostics to enhance understanding of execution traces.
- ▪Human experts using IG reports improved scaffold performance by 30.4 percentage points over the baseline.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2605.21347 (cs) [Submitted on 20 May 2026 (v1), last revised 21 May 2026 (this version, v2)] Title:Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents Authors:Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz, Yuan (Emily)Xue View a PDF of the paper titled Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents, by Akshay Manglik and 8 other authors View PDF HTML (experimental) Abstract:Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
