W&B Weave

Weights & Biases (W&B) is the platform account used in the evaluation workshop, providing inference credits and sandboxes for running evaluation workflows. The workflow uses W&B's Weave tooling to preserve agent trajectories, including commands, messages, and failures, so reviewers can inspect the evidence behind leaderboard scores.

WolfBench is an open-source Apache-2.0 toolkit and evaluation methodology by Wolfram Ravenwolf for collecting, curating, analyzing, and publishing AI-agent benchmark results. It evaluates agents on Terminal-Bench 2.0 through Harbor orchestration and Daytona sandboxes, preserving configuration metadata, per-task results, token usage, timing, failures, and trajectory traces. Rather than reporting only an average, WolfBench calculates five metrics: the union of tasks solved across runs (Answers), the best single-run score, the mean across valid runs, the worst single-run score, and the tasks solved in every run (Solid). Repeated runs under consistent benchmark, runner, sandbox, agent, model-service, and settings configurations expose variance and distinguish dependable performance from isolated successes. The repository includes a Marimo dashboard that scans local and remote run directories over SSH, deduplicates and classifies runs, filters incomplete or invalid results, downloads data, saves curated JSON snapshots, uploads evaluations and trajectory traces to W&B Weave, and generates interactive HTML leaderboard charts. The core collector and chart generator use only the Python standard library; the dashboard requires Python 3.12+, Marimo, Weave/W&B, and SSH access for initial collection. Benchmark data is licensed CC-BY-4.0.

WolfBench is an interactive online leaderboard for evaluating AI-agent configurations on Terminal Bench runs. It compares average, best, and worst scores; consistency, measured by tasks solved in every run; ceiling, measured by tasks solved at least once; token usage; and total cost. Users can filter and reorder results, export charts, share leaderboard state through URLs, and inspect the underlying Weave traces for commands, messages, and failures. Its evaluation workflow launches Terminal Bench tasks through Harbor, filters incomplete or invalid runs in a Marimo dashboard, and turns valid repeated runs into leaderboard results. A configuration records the benchmark, runner, sandbox, agent, model service, and settings, so versions, provider routing, prompts, resource limits, concurrency, retries, and timeouts remain part of the comparison. The project distinguishes infrastructure failures from task failures and uses repeated runs to expose differences between occasional success and consistent behavior.

W&B Inference is a serverless model-serving endpoint for running agents against selected open-weight models.

Weave is a meeting intelligence tool that listens to spoken conversations and generates a live, editable map of the thinking during a discussion (decisions, open questions, and who said what). Maps are saved, replayable, and shareable for review and asynchronous collaboration.