Private repository benchmarking platform for enterprise coding-agent selection
I provide a product that turns an enterprise's existing GitHub codebase and completed work into an internal coding benchmark. The company can then compare coding agents and models on its own repository, choose tools for particular teams or projects, and control token usage according to the intelligence required by each task.
From a16z — Inside the Race to Measure Frontier Intelligence at 20:04
Problem: Public coding benchmarks do not reliably reveal which model or coding agent will perform best on a particular company's repository. Token usage can also become much larger than employee salary costs, making arbitrary tool access and usage limits difficult to justify.
For: Enterprises with substantial software engineering teams that are choosing among coding agents, models, harnesses, and pricing models.
Products from this video
Examples
- Vals: after a token-maxing experiment, it used repository and trace data to build Val Smith and found Cognition's Devin to be token efficient enough to adopt more.