AI product Open source
Switchyard is an NVIDIA Rust library and proxy for routing LLM requests among models and providers while preserving native OpenAI and Anthropic API compatibility. Its routing algorithms choose among efficient and capable models using mechanisms such as LLM judging, stage-based response evaluation, escalation, advisor approval, random selection, and custom target-selection policies. It can run as a standalone proxy, integrate with NeMo Relay or LiteLLM, or be embedded in a custom gateway or harness; the proxy exposes OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses endpoints, along with statistics and Prometheus metrics for requests, errors, latency, tokens, and routing overhead. The repository provides Rust components for routing, provider-neutral protocols, HTTP model calls, and request, response, and stream translation. It is pre-1.0 software under the Apache 2.0 license, with the standalone server documented as suitable for demonstrations and evaluation rather than production.
1 use taken from transcripts — each links to the moment in the video.
A Rust proxy and library for routing and translating LLM traffic between OpenAI Chat, Anthropic Messages, and OpenAI Responses APIs. It allows native client APIs to be served by compatible local or hosted model endpoints and records Prometheus metrics.
1 in the library.