LMCache

CacheGen is a system for compressing and streaming key-value (KV) data generated by AI inference engines. It is intended to make the real-time movement of large KV objects feasible, addressing the size of KV data when it must be transferred between components or systems.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

NVIDIA Dynamo is a developer toolkit from NVIDIA for managing distributed key–value (KV) context caches used in large‑language‑model inference. It provides KV‑aware routing, cache offloading, and prefill/decode disaggregation to move KV cache and related activation/state across GPU clusters, reducing memory pressure and supporting scalable LLM serving. It is intended for integration into GPU‑based inference and serving stacks.

Intrinsec AI is an AI-inference security company developing products that use KV-cache data as a security signal for production LLM inference. It learns the normal computational shape of a workload from its inference traffic and surfaces unusual KV pages, rather than inspecting prompts or relying on a separate detector for each attack or failure. Its Tessera product analyzes KV cache at rest, while Plato acts on anomalous KV data in flight; the system is intended to cover text, voice, tool-using, and agent workloads and can integrate with inference engines or KV-cache management layers.

Akamai LKE is the Kubernetes environment used in the workshop to run inference workloads on dedicated GPU nodes. It provides the namespace and deployment structure for a Jupyter notebook and a vLLM endpoint, and supports changing model-serving deployments and evaluating their latency, throughput, memory use, and output quality.