BenchPod

Runpod is an AI infrastructure cloud that provides on-demand GPUs for training, fine-tuning, inference, batch workloads, and other compute tasks. Its platform combines persistent GPU Pods for development and stateful workloads, multi-GPU Clusters for distributed training and large-batch inference, and Serverless endpoints that autoscale GPU workers and scale to zero when idle. Users can deploy their own containers, frameworks, code, and inference handlers, with managed queuing, task distribution, real-time logs, monitoring, metrics, and persistent network storage.

GuideLLM is an open-source platform for benchmarking and evaluating real-world LLM inference deployments. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, using real or synthetic text and multimodal datasets and configurable execution profiles such as synchronous, concurrent, throughput, constant-rate, Poisson, and sweep workloads. It measures SLO-oriented latency and token statistics, including time to first token, inter-token latency, and end-to-end behavior, then produces console, JSON, CSV, or HTML reports for deployment tuning, capacity planning, and regression tracking. The tool provides both a CLI and API, with multiprocessing, threading, asynchronous execution, and Hugging Face, file, synthetic, or custom data sources.

WolfBench is an interactive online leaderboard for evaluating AI-agent configurations on Terminal Bench runs. It compares average, best, and worst scores; consistency, measured by tasks solved in every run; ceiling, measured by tasks solved at least once; token usage; and total cost. Users can filter and reorder results, export charts, share leaderboard state through URLs, and inspect the underlying Weave traces for commands, messages, and failures. Its evaluation workflow launches Terminal Bench tasks through Harbor, filters incomplete or invalid runs in a Marimo dashboard, and turns valid repeated runs into leaderboard results. A configuration records the benchmark, runner, sandbox, agent, model service, and settings, so versions, provider routing, prompts, resource limits, concurrency, retries, and timeouts remain part of the comparison. The project distinguishes infrastructure failures from task failures and uses repeated runs to expose differences between occasional success and consistent behavior.

Runpod Flash is an open-source Python SDK and application framework from Runpod for developing cloud-native AI applications and running serverless GPU and CPU workloads without manually writing Dockerfiles or building and pushing container images. Developers define remote functions with Python decorators, specifying hardware, dependencies, workers, and concurrency, then use `flash deploy` to package code, build cross-platform worker artifacts, install dependencies, and provision endpoints. Its `flash dev` mode runs a local FastAPI server while provisioning live ephemeral Runpod workers. Flash supports queue-based endpoints for asynchronous and batch workloads, load-balanced endpoints with HTTP routes, custom Docker images, endpoint-to-endpoint calls, connections to existing Runpod endpoints by ID, automatic dependency installation, and worker scaling from zero to the configured maximum. It supports multi-endpoint applications in which functions on different endpoints call one another through service discovery and routing handled by the runtime. It requires Python 3.10 or later on macOS or Linux.

WolfBench is an open-source Apache-2.0 toolkit and evaluation methodology by Wolfram Ravenwolf for collecting, curating, analyzing, and publishing AI-agent benchmark results. It evaluates agents on Terminal-Bench 2.0 through Harbor orchestration and Daytona sandboxes, preserving configuration metadata, per-task results, token usage, timing, failures, and trajectory traces. Rather than reporting only an average, WolfBench calculates five metrics: the union of tasks solved across runs (Answers), the best single-run score, the mean across valid runs, the worst single-run score, and the tasks solved in every run (Solid). Repeated runs under consistent benchmark, runner, sandbox, agent, model-service, and settings configurations expose variance and distinguish dependable performance from isolated successes. The repository includes a Marimo dashboard that scans local and remote run directories over SSH, deduplicates and classifies runs, filters incomplete or invalid results, downloads data, saves curated JSON snapshots, uploads evaluations and trajectory traces to W&B Weave, and generates interactive HTML leaderboard charts. The core collector and chart generator use only the Python standard library; the dashboard requires Python 3.12+, Marimo, Weave/W&B, and SSH access for initial collection. Benchmark data is licensed CC-BY-4.0.