VI codebench

VSC-Bench is an evaluation tool for testing agentic scenarios in Visual Studio Code. It runs product evaluations and supports adding new scenarios from GitHub issues.

Vals is a code-generation evaluation product that uses a company's GitHub codebase to build an internal coding benchmark. It evaluates coding models and agents against private, continuously evolving tasks to identify which systems perform best on the company's work and which offer the strongest return on investment. The platform is also described as supporting pre-release testing and evaluation of agentic coding work that may unfold over hours, days, or weeks, while accounting for factors such as token cost and latency.

Vals AI is an AI benchmarking and evaluation company that independently tests models on economically valuable real-world tasks, including software, finance, law, cybersecurity and other frontier-risk areas. It develops and runs its own evaluations, publishes benchmark results and model comparisons through the Vals Index and specialized benchmarks such as finance-agent, Excel-modeling, tax-agent and legal evaluations, and reports performance alongside cost, latency and token usage. The videos also describe Val Smith, an internal-tooling product that evaluates models against a firm's actual GitHub work to help choose models and reduce token spending.

Can I Vibecode It? is an open-source catalog that evaluates whether AI coding agents such as Claude Code, Codex, or Cursor can build a usable personal replacement for a paid SaaS application. Each app entry provides a YES, KINDA, or NOT REALLY verdict, the exact prompt for attempting the replacement, and the tradeoffs of leaving the original service, including lost network effects, data, or infrastructure. App entries are contributed as one JSON file per app, with the schema and verdict criteria documented in the repository. The site uses Astro server output with a Node adapter to render fully server-rendered HTML, SQLite via better-sqlite3 for vote counters and waitlist data, and vanilla JavaScript and CSS for interactions. Satori and resvg generate Open Graph images at build time. It runs locally with npm install, npm run dev, npm run build, and npm start, requires no environment variables for local development, and supports optional analytics, sign-in, payments, email, and media-storage configuration. It can be deployed on a VPS behind a reverse proxy, with DATA_DIR used to keep user data outside the repository. The project is released under the MIT License, and its prompts are intended to remain free.

WolfBench is an interactive online leaderboard for evaluating AI-agent configurations on Terminal Bench runs. It compares average, best, and worst scores; consistency, measured by tasks solved in every run; ceiling, measured by tasks solved at least once; token usage; and total cost. Users can filter and reorder results, export charts, share leaderboard state through URLs, and inspect the underlying Weave traces for commands, messages, and failures. Its evaluation workflow launches Terminal Bench tasks through Harbor, filters incomplete or invalid runs in a Marimo dashboard, and turns valid repeated runs into leaderboard results. A configuration records the benchmark, runner, sandbox, agent, model service, and settings, so versions, provider routing, prompts, resource limits, concurrency, retries, and timeouts remain part of the comparison. The project distinguishes infrastructure failures from task failures and uses repeated runs to expose differences between occasional success and consistent behavior.