SWE-bench

VSC-Bench is an evaluation tool for testing agentic scenarios in Visual Studio Code. It runs product evaluations and supports adding new scenarios from GitHub issues.

Terminal Bench 4 is a benchmark for comparing coding-model performance across tasks with different difficulty levels.

FrontierHarness Eval is an open evaluation benchmark and repository for measuring how coding-agent harness configurations affect software-engineering task results when the underlying model and runtime are held constant. It publishes benchmark definitions, task instructions and metadata, harness-version metadata, normalized aggregate and task-level results, and an agent-neutral workflow for evaluating additional harnesses. The benchmark runs the same model through multiple harness configurations on 30 tasks, using a frozen golden checkpoint and a fresh restore for each task. Its scripts provision the environment, run trials, record verifier-based pass/fail outcomes, measure cost, cache behavior, and speed, then normalize results and generate charts and reports. The repository also provides a CLI and skill-based workflow for evaluating a third-party harness, with infrastructure failures marked separately from task failures. The project focuses on terminal-based software-engineering tasks and states that its results may not generalize to other knowledge-work domains. The repository is published under the frontier-harness-eval organization, with isolated runtimes and checkpoint restores provided by Runta.

VI codebench is a coding benchmark that measures how well AI models turn natural-language prompts into full-stack web applications. It is associated with VALS, an AI evaluation company that develops continuously evolving assessments for measuring model performance before release and in practical coding workflows.

CommerceAgentBench is a benchmark developed and maintained by the Accio team at Alibaba International for evaluating whether AI agents can complete long-horizon business workflows in high-fidelity, stateful, reproducible replicas of online services. Its tasks cover browser operations, native-style command-line tools, file and spreadsheet production, API and MCP workflows, public-web research, supplier analysis, product publishing, logistics, storefront configuration, and other commerce operations. Each task runs in a fresh container and is graded by a deterministic or LLM-assisted verifier. The benchmark uses local mock services representing commerce, SaaS, messaging, document, and operational systems; runs preserve the resolved configuration, agent trajectory, verifier result, artifacts, logs, and container metadata for auditing and reproducibility. The repository describes 107 tasks spanning CLI, browser, file, and API/MCP interfaces, with text-only, browser-text-capable, and vision-required capability slices. The project was previously called RealReplicaBench.