WolfBench

WolfBench is an interactive online leaderboard for evaluating AI-agent configurations on Terminal Bench runs. It compares average, best, and worst scores; consistency, measured by tasks solved in every run; ceiling, measured by tasks solved at least once; token usage; and total cost. Users can filter and reorder results, export charts, share leaderboard state through URLs, and inspect the underlying Weave traces for commands, messages, and failures. Its evaluation workflow launches Terminal Bench tasks through Harbor, filters incomplete or invalid runs in a Marimo dashboard, and turns valid repeated runs into leaderboard results. A configuration records the benchmark, runner, sandbox, agent, model service, and settings, so versions, provider routing, prompts, resource limits, concurrency, retries, and timeouts remain part of the comparison. The project distinguishes infrastructure failures from task failures and uses repeated runs to expose differences between occasional success and consistent behavior.

CommerceAgentBench is a benchmark developed and maintained by the Accio team at Alibaba International for evaluating whether AI agents can complete long-horizon business workflows in high-fidelity, stateful, reproducible replicas of online services. Its tasks cover browser operations, native-style command-line tools, file and spreadsheet production, API and MCP workflows, public-web research, supplier analysis, product publishing, logistics, storefront configuration, and other commerce operations. Each task runs in a fresh container and is graded by a deterministic or LLM-assisted verifier. The benchmark uses local mock services representing commerce, SaaS, messaging, document, and operational systems; runs preserve the resolved configuration, agent trajectory, verifier result, artifacts, logs, and container metadata for auditing and reproducibility. The repository describes 107 tasks spanning CLI, browser, file, and API/MCP interfaces, with text-only, browser-text-capable, and vision-required capability slices. The project was previously called RealReplicaBench.

Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing AI agents and language models in container environments. It can run arbitrary agents, including Claude Code, OpenHands, and Codex CLI, against shared or custom benchmarks and environments such as Terminal-Bench, SWE-Bench, and Aider Polyglot. Harbor launches benchmark runs locally with Docker or in parallel through cloud and sandbox providers, and can generate rollouts for reinforcement-learning optimization. It is distributed as a Python package installable with uv or pip and provides command-line tools for running datasets, selecting agents and models, and listing supported benchmarks.

W&B Weave is an AI evaluation and observability tool that stores and displays per-run trajectories behind leaderboard results. In the WolfBench workflow, it preserves inspectable traces of commands, messages, failures, and other run evidence so reviewers can trace a score back to the conditions that produced it.

Multi-Agent Workbench is a local-first browser control room for orchestrating coding agents such as Claude Code, Codex CLI, and generic PTY-based command-line agents. Agents work in task rooms under editable YAML role cards that define permissions, speaking style, and decision boundaries; a rule-driven orchestrator pauses execution and risky commands for human approval, while persistent sessions expose terminal activity and status. The system uses a local event-driven architecture with a React interface, Fastify backend, REST API, server-sent events, agent adapters, an approval service, and a SQLite event store. Every action passes through the API and is recorded in an append-only event log, whose events are projected into timelines, decisions, terminal views, artifacts, and diff views. It also provides task templates, Markdown reports and decision logs, real-time room updates, and configurable risk resolutions. Code-writing agents are intended to use isolated Git worktrees, with diff and patch review before changes reach the main workspace; the repository marks this worktree and merge flow as in progress.