Terminal Bench 4

Google Benchmark is a C++ microbenchmark support library for measuring the runtime of code snippets, including operations and data-structure changes during performance optimization. Benchmarks are written as functions that perform setup, execute the code under timing inside a `benchmark::State` loop, and are registered and run through the library's benchmark and `BENCHMARK_MAIN` APIs. It can be used with C++11, requires C++17 to build, and is built with CMake; Google Test is required for building and running its tests unless those tests are disabled.

CommerceAgentBench is a benchmark developed and maintained by the Accio team at Alibaba International for evaluating whether AI agents can complete long-horizon business workflows in high-fidelity, stateful, reproducible replicas of online services. Its tasks cover browser operations, native-style command-line tools, file and spreadsheet production, API and MCP workflows, public-web research, supplier analysis, product publishing, logistics, storefront configuration, and other commerce operations. Each task runs in a fresh container and is graded by a deterministic or LLM-assisted verifier. The benchmark uses local mock services representing commerce, SaaS, messaging, document, and operational systems; runs preserve the resolved configuration, agent trajectory, verifier result, artifacts, logs, and container metadata for auditing and reproducibility. The repository describes 107 tasks spanning CLI, browser, file, and API/MCP interfaces, with text-only, browser-text-capable, and vision-required capability slices. The project was previously called RealReplicaBench.

Assistant Benchmark is a comparative evaluation of AI assistants created by David Pawlan. It gives different assistants the same real-world prompts, including tasks such as email management, travel booking, and financial administration, and compares their outcomes, follow-up questions, speed, and performance across 16 dimensions. The benchmark is based on ongoing testing of multiple assistants and examines both task execution and the degree of autonomy assistants can usefully take.

VI codebench is a coding benchmark that measures how well AI models turn natural-language prompts into full-stack web applications. It is associated with VALS, an AI evaluation company that develops continuously evolving assessments for measuring model performance before release and in practical coding workflows.

SWE-bench is an execution-based benchmark for evaluating large language models on real-world software issues collected from GitHub. Given a repository codebase and an issue, a model must generate a patch intended to resolve the problem; the patch is evaluated by running repository tests in Docker-based environments. The project provides datasets and an evaluation command-line tool, with dataset variants including full, verified, multimodal, multilingual, and locally supplied datasets. It is distributed as open-source code through its GitHub repository and can also be loaded from Hugging Face.