Event-driven adversarial PR review service as a standing agent job
A multi-agent pull-request reviewer deployed as a job triggered by GitHub webhooks. A contract declares the reviewer persona and evidence-based review standard; fresh review sessions are spawned on each PR and benchmarked against the contract, giving building agents back-pressure so they cannot mark work complete until an independent reviewer passes it. This escapes the self-review bias where the model that wrote the code checks its own work.
From AI Engineer — Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse
Problem: Self-review is biased: the agent (or human) that built a feature thinks it did a good job. Chat-based review is one-off and unreliable. This makes review repeatable, auditable, and running automatically on every event.
For: Software teams and solo developers shipping AI-generated code who want independent review on every pull request.
Products from this video
Examples
- The 'grumpy senior developer who doesn't like AI-generated code' reviewer deployed on the run.prose.md repo, producing dozens of reviews across PRs in a few days.
Other takes on AI evaluation and benchmarking
- Voice agent benchmarking harness for picking a model stack
- Evaluation pipeline for high-stakes AI agents (trace, curate datasets, expert-reviewed failure modes, CI quality gates)
- AI technical due diligence service for investors and acquirers evaluating automation claims
- An independent evaluation company for foundation models and agents
- Per-agent tracking and evaluation tables for ROI reporting
- Developing specialized AI benchmarks for professional workflows