Runpod Flash documentation

Runpod Flash is an open-source Python SDK and application framework from Runpod for developing cloud-native AI applications and running serverless GPU and CPU workloads without manually writing Dockerfiles or building and pushing container images. Developers define remote functions with Python decorators, specifying hardware, dependencies, workers, and concurrency, then use `flash deploy` to package code, build cross-platform worker artifacts, install dependencies, and provision endpoints. Its `flash dev` mode runs a local FastAPI server while provisioning live ephemeral Runpod workers. Flash supports queue-based endpoints for asynchronous and batch workloads, load-balanced endpoints with HTTP routes, custom Docker images, endpoint-to-endpoint calls, connections to existing Runpod endpoints by ID, automatic dependency installation, and worker scaling from zero to the configured maximum. It supports multi-endpoint applications in which functions on different endpoints call one another through service discovery and routing handled by the runtime. It requires Python 3.10 or later on macOS or Linux.

Runpod Flash Examples is a GitHub repository of example applications for Runpod Flash, a Python framework for running functions on Runpod Serverless infrastructure with distributed GPU and CPU computing. The examples cover GPU, CPU, and mixed workers, dependency management, local modules, model serving, load-balanced HTTP routes, autoscaling, and network volumes. The repository demonstrates the Flash workflow, including the `@Endpoint` decorator, local development with hot reload, deployment with `flash build` and `flash deploy`, and endpoint-to-endpoint client calls. It is maintained as an open-source example collection with contribution and development guides, and the README indicates that additional examples are planned. The repository is licensed under MIT.

Runpod is an AI infrastructure cloud that provides on-demand GPUs for training, fine-tuning, inference, batch workloads, and other compute tasks. Its platform combines persistent GPU Pods for development and stateful workloads, multi-GPU Clusters for distributed training and large-batch inference, and Serverless endpoints that autoscale GPU workers and scale to zero when idle. Users can deploy their own containers, frameworks, code, and inference handlers, with managed queuing, task distribution, real-time logs, monitoring, metrics, and persistent network storage.

BenchPod is a Runpod benchmarking dashboard and template built on vLLM. It packages vLLM in a Docker image with a FastAPI layer, centralizes GPU statistics, cold-start measurements, inter-token latency, time to first token, and time per output token, and allows vLLM configurations to be changed and tested live without waiting for restarts. It provides OpenAI-compatible endpoints for agents and includes a watchdog intended to recover failed instances. The video describes it as not yet released and expected to be distributed as a Runpod template.

pi pod is a self-hosted platform for running pi coding-agent sessions in isolated remote sandboxes called pods. Its command-line client and iOS and Android apps connect to a server that manages pod lifecycles through a REST API, session gateway, lifecycle workers, and Postgres; a native sandbox service runs multiple isolated pods in one container, with pi inside each pod behind a small shim. The CLI can launch and attach to a pod for the current project, while the server uses Zitadel for OIDC identity and stores no passwords. The repository includes a Docker Compose deployment, CLI package, server, sandbox service, mobile apps, and self-hosting tools. It requires a Linux host with Docker Compose, at least 8 GB of RAM, and Node 22.19 or later for the CLI, and is licensed under AGPL-3.0-only.