A curated, non-BS library of the best resources for evaluating agents
The Awesome Agent Evals library is a curated collection of resources for building and evaluating AI agents, including papers, blog posts, talks, courses, tools, and benchmarks. The library is annotated and verified, with every entry including a description of what it is and why it belongs, as well as checked URLs and quotes. The library was assembled through a combination of academic citation crawls, practitioner-web discovery, and gap audits with adversarial verification.
- ▪The library contains over 443 curated links and 146 deep reading notes, with markers indicating newly released or updated content and caveats where applicable.
- ▪The library includes a playbook with real, runnable code and worked examples for evaluating AI agents, including patterns for LLM-as-judge, pass@k/pass^k, error analysis, and more.
- ▪The library is maintained by BenchFlow and is open to contributions, with a contributing guide available for those interested in adding to the collection.
Opening excerpt (first ~120 words) tap to expand
Awesome Agent Evals A curated, opinionated, non-BS library of the best resources for building and evaluating AI agents — papers, blog posts, talks, courses, tools, and benchmarks. Maintained by BenchFlow · Most "awesome" lists are link dumps. This one is annotated and verified: every entry says what it is and why it belongs, URLs are checked, quotes are verbatim, and dead/abandoned tools are pruned (not silently listed). It was assembled by: a depth-4 recursive citation crawl (11.6k papers, ranked by in-degree) to surface the academic canon, targeted practitioner-web discovery for the industry sources citation graphs miss (Eugene Yan, Han-Chung Lee, Hamel Husain, Shreya Shankar, Nathan Lambert, …), 47 talks & podcasts transcribed and deep-noted (verbatim + timestamps), and per-section gap…
Excerpt limited to ~120 words for fair-use compliance. The full article is at GitHub.