Turn an agent's recorded failure history into an evaluation suite, then hill-climb the product against it
Mine your own usage history for every time a model did the wrong thing with your tools, categorize those failures into an eval, and run agents against it. Then hill-climb: first get every model to a perfect score by fixing behavior, commit, then cut the tool descriptions to a fraction of their size while keeping the perfect score. This shrinks context massively and proves via the eval that the result is better, not worse.
From Theo - t3․gg — Agents Sucked At UI (Until Today) at 27:24
Problem: Models misuse agent-facing tools — spawning new threads instead of sub-agents, leaving threads visible — because the tool descriptions are vague or bloated, and nobody had a reliable way to test description changes.
For: Developers building agent-facing tools, MCPs, or APIs whose models misuse or misunderstand the tools.
Products from this video
Examples
- T3 Code's MCP: the eval PR was over 2,000 lines of code; shrinking descriptions cut context by 52% with tool search off and 78% with tool search on, now live in nightly builds.
Other takes on AI evaluation and benchmarking
- Voice agent benchmarking harness for picking a model stack
- Evaluation pipeline for high-stakes AI agents (trace, curate datasets, expert-reviewed failure modes, CI quality gates)
- AI technical due diligence service for investors and acquirers evaluating automation claims
- Event-driven adversarial PR review service as a standing agent job
- An independent evaluation company for foundation models and agents
- Per-agent tracking and evaluation tables for ROI reporting