AI product Open source
vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support.
Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism.
The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.
21 uses taken from transcripts — each links to the moment in the video.
Provides an open-source inference engine used as part of the standard model-support stack.
Runs language models efficiently at production scale on hardware accelerators, supporting batching, paged attention, broad hardware and model compatibility, and OpenAI-compatible endpoints.
Open-source inference engine for turning GPUs and other accelerators into model-serving endpoints, supporting model architectures, optimizing serving, and benchmarking hardware.
Open-source inference engine providing KV cache, paged attention, prefix caching, chunked prefill, and speculative decoding capabilities for scaling LLM inference.
Serves the deployed model on the GPU rack.
An LLM inference engine that loads model weights, manages GPU memory and KV caches, batches requests, and serves an API. The video uses it to demonstrate how quantizing model weights and the KV cache increases concurrent serving capacity.
Runs batch inference for synthetic-data generation and is tuned through parameter sweeps for higher throughput.
Provided a reference inference engine and deployment architecture for comparison with SGLang.
An LLM serving engine used to run the demo. The speaker implemented speculative decoding with it, running baseline and speculative setups side by side on a single NVIDIA Blackwell GPU, and notes its documentation is good for getting started.
Inference engine for serving models on GPUs with continuous batching, paged attention and OpenAI-compatible API
Local model inference on the GPU cluster for data generation
LLM inference server used to serve the victim model; its optional speculative decoding flag produces the step-timing signal that enables workload inference.
Mature LLM-serving environment that uses batching and continuous batching to improve inference performance
An open-source library for running and serving large language models, utilizing techniques like page detention and continuous batching to serve many requests efficiently.
LLM serving engine whose configs (e.g. max batch tokens) are benchmarked and tuned; BenchPod is built on top of it
Open-source inference stack for spinning up your own LLM inference service in your own environment.
Inference engine used for serving rollouts during RL training.
Inference engine where KV-cache anomaly detection can be embedded for real-time alerting.
An LLM inference engine and server described as the JVM of language models. It implements the runtime for each released model architecture, uses paged attention for GPU memory, and serves high-throughput inference via continuous batching, exposing endpoints for inference calls. llm-d builds on top of it to scale across a cluster.
Self-hosted inference engine that hosts the model on GPUs, batches requests and maintains the KV cache; the admission layer is placed in front of it unchanged
LLM serving engine whose configuration (the serving recipe YAML) the agent optimizes
22 in the library.