vLLM Semantic Router

llm-d semantic classifier is an incubating llm-d project for routing conversations to an appropriate language model rather than selecting models only for inference efficiency. Its routing can use factors such as cost or conversation topic, including directing requests to a fine-tuned specialized model or to a smaller, less expensive model instead of a larger one. It is described as a sibling of the vLLM semantic router project.

RouteLLM is an open-source framework from lm-sys for serving and evaluating routers that direct requests between stronger and cheaper language models. It provides trained routers, a drop-in replacement for the OpenAI client, and an OpenAI-compatible server; routing decisions use configurable cost thresholds that can be calibrated for a target share of requests sent to the stronger model. The framework supports comparing routers across benchmarks and extending it with additional routers, model pairs, and providers.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

Vercel AI Gateway is a managed service from Vercel that routes application requests to language-model providers, centralizes API key management, and provides observability, caching, and rate-limiting for LLM usage. It is intended to simplify integration of multiple LLMs and enforce access controls for apps and autonomous agents.

rLLM is an open-source framework from UC Berkeley's Sky Computing Lab for training language agents with reinforcement learning. It connects agent code to evaluation and training through a decorator-based Python API or CLI workflow: agents run in sandboxes, the model gateway captures token IDs and log probabilities, traces are organized into episodes and trajectories, evaluators compute rewards, and a training backend updates the model. It supports existing or custom agent harnesses, Docker, Daytona, Modal, and local sandboxes, with training backends including verl, tinker, and Fireworks, and methods including GRPO, REINFORCE, RLOO, supervised fine-tuning, and on-policy distillation. The project also provides integrated benchmarks and requires Python 3.11 or later.