Weights & Biases (W&B)

W&B Weave is an AI evaluation and observability tool that stores and displays per-run trajectories behind leaderboard results. In the WolfBench workflow, it preserves inspectable traces of commands, messages, failures, and other run evidence so reviewers can trace a score back to the conditions that produced it.

Weights & Biases is a software service for training observability that can export metrics through an API-key integration.

W&B Inference is a serverless model-serving endpoint for running agents against selected open-weight models.

W&B Serverless Sandboxes are isolated per-task environments used in evaluation pipelines. Each evaluation runs in a clean sandbox, helping keep benchmark conditions consistent and separating infrastructure effects from task failures.

WolfBench is an open-source Apache-2.0 toolkit and evaluation methodology by Wolfram Ravenwolf for collecting, curating, analyzing, and publishing AI-agent benchmark results. It evaluates agents on Terminal-Bench 2.0 through Harbor orchestration and Daytona sandboxes, preserving configuration metadata, per-task results, token usage, timing, failures, and trajectory traces. Rather than reporting only an average, WolfBench calculates five metrics: the union of tasks solved across runs (Answers), the best single-run score, the mean across valid runs, the worst single-run score, and the tasks solved in every run (Solid). Repeated runs under consistent benchmark, runner, sandbox, agent, model-service, and settings configurations expose variance and distinguish dependable performance from isolated successes. The repository includes a Marimo dashboard that scans local and remote run directories over SSH, deduplicates and classifies runs, filters incomplete or invalid results, downloads data, saves curated JSON snapshots, uploads evaluations and trajectory traces to W&B Weave, and generates interactive HTML leaderboard charts. The core collector and chart generator use only the Python standard library; the dashboard requires Python 3.12+, Marimo, Weave/W&B, and SSH access for initial collection. Benchmark data is licensed CC-BY-4.0.