W&B Inference

Weights & Biases (W&B) is the platform account used in the evaluation workshop, providing inference credits and sandboxes for running evaluation workflows. The workflow uses W&B's Weave tooling to preserve agent trajectories, including commands, messages, and failures, so reviewers can inspect the evidence behind leaderboard scores.

W&B Weave is an AI evaluation and observability tool that stores and displays per-run trajectories behind leaderboard results. In the WolfBench workflow, it preserves inspectable traces of commands, messages, failures, and other run evidence so reviewers can trace a score back to the conditions that produced it.

W&B Serverless Sandboxes are isolated per-task environments used in evaluation pipelines. Each evaluation runs in a clean sandbox, helping keep benchmark conditions consistent and separating infrastructure effects from task failures.

Bindwidth is a browser-based calculator for sizing on-premises LLM inference infrastructure and estimating total cost of ownership. It evaluates three constraints for a resident text model—KV-cache memory, combined prefill and decode serving capacity, and the runtime session ceiling—and identifies which constraint binds for a workload. The tool accounts for interactive users and autonomous agents as different load types, then compares owned hardware, eligible sovereign rentals, and enterprise subscriptions with per-token APIs for agents. It distinguishes measured from estimated hardware and model data through confidence levels and profile-specific overrides, flags domain and model-capability limits, and exports a decision record as Markdown or the full scenario as JSON. The repository describes it as a directional sensitivity estimator rather than a procurement quote. It is a frontend-only application with no build step, server, database, account, or telemetry; calculations run in the browser and entered data is not transmitted.

WolfBench is an open-source Apache-2.0 toolkit and evaluation methodology by Wolfram Ravenwolf for collecting, curating, analyzing, and publishing AI-agent benchmark results. It evaluates agents on Terminal-Bench 2.0 through Harbor orchestration and Daytona sandboxes, preserving configuration metadata, per-task results, token usage, timing, failures, and trajectory traces. Rather than reporting only an average, WolfBench calculates five metrics: the union of tasks solved across runs (Answers), the best single-run score, the mean across valid runs, the worst single-run score, and the tasks solved in every run (Solid). Repeated runs under consistent benchmark, runner, sandbox, agent, model-service, and settings configurations expose variance and distinguish dependable performance from isolated successes. The repository includes a Marimo dashboard that scans local and remote run directories over SSH, deduplicates and classifies runs, filters incomplete or invalid results, downloads data, saves curated JSON snapshots, uploads evaluations and trajectory traces to W&B Weave, and generates interactive HTML leaderboard charts. The core collector and chart generator use only the Python standard library; the dashboard requires Python 3.12+, Marimo, Weave/W&B, and SSH access for initial collection. Benchmark data is licensed CC-BY-4.0.