One-time agent-history audit skill for individual developers
A single-player variant of the loop: a skill that crawls all of a developer's local agent conversations (Claude Code, Codex, Pi, etc.), grades them in batches against rubrics for token efficiency, produces a score report card, and then a self-improvement agent reads the failing notes and proposes concrete skill changes — a first pass to find low-hanging fruit before committing to continuous scoring.
From AI Engineer — Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp
Problem: Without watching traces closely, hidden inefficiencies — like an agent repeatedly banging against port numbers or silently working around outdated skill instructions — go unnoticed; a one-time audit surfaces them cheaply.
For: Individual developers ('single-player' setups) who use coding agents locally and have never inspected their conversation histories.
Products from this video
Examples
- Ben's sample run over his own code sessions: graded across efficiency and code quality, found verbosity fine, and proposed port-finding instructions in the end-to-end testing skill plus a more prescriptive changelog skill.
Other takes on AI evaluation and benchmarking
- Voice agent benchmarking harness for picking a model stack
- Evaluation pipeline for high-stakes AI agents (trace, curate datasets, expert-reviewed failure modes, CI quality gates)
- AI technical due diligence service for investors and acquirers evaluating automation claims
- Event-driven adversarial PR review service as a standing agent job
- An independent evaluation company for foundation models and agents
- Per-agent tracking and evaluation tables for ROI reporting