Akamai

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

APEX is an open, verification-first RTL design for an LLM-inference tile developed with Sigmantic AI. It implements one transformer decoder layer—including attention, softmax, RMSNorm, RoPE, SwiGLU, residual processing, and KV-cache compression—and runs Qwen2.5-0.5B through the verified pipeline on FPGA hardware. The KV-cache codec sits in the datapath: cached keys and values are compressed as they are produced and decompressed when consumed, while an importance unit allocates precision according to which parts of the context matter. Each RTL block is checked against an executable NumPy golden model with bit-exact comparisons and mutation-tested testbenches. The repository covers the inference tile rather than a complete system; DRAM control, PCIe, and NoC are out of scope.

Cast AI is a Kubernetes optimization platform for cloud-native teams. It uses workload, infrastructure, cost, and SLO signals to automate actions such as pod rightsizing, node scaling, GPU and Spot optimization, workload replica and resource tuning, bin packing, and operational remediation with approval workflows. The platform connects to EKS, AKS, GKE, and on-premises clusters without requiring cluster changes. The videos also identify Cast AI as the company that created Kimchi, an open-source coding harness with tools for autonomous coding tasks, persistent agent sessions, and team collaboration.

OpenAI is an AI model provider whose language models are available through an API and can serve as one provider in multi-model routing architectures. The videos describe its models being used for AI replies and extraction, Amazon-review analysis and customer-service email generation, legal workflows, complex software development and debugging, and experimental or auxiliary agent tasks.

LiteLLM is an open-source AI gateway and Python SDK from BerriAI for calling more than 100 LLM providers through a unified OpenAI-compatible interface. It supports direct library integration and a self-hosted proxy server, with provider-specific formats translated into common endpoints such as chat completions, responses, embeddings, image generation, audio, batches, reranking, messages, and agent-to-agent calls. The proxy provides centralized authentication and authorization, virtual keys, per-project and per-user spend tracking, logging, guardrails, caching, load balancing, routing with retry and fallback logic, and an administration dashboard. LiteLLM also exposes MCP tools and A2A agents through the gateway and can be deployed as a componentized stack with separate gateway, backend, and UI services. The repository describes it as open source and documents Docker, local development, and Terraform deployment options for AWS and Google Cloud.