AI product Open source

SGLang

SGLang is an open-source serving framework and inference engine for large language models and multimodal models, developed by the SGLang project. It provides an alternative serving stack for deployed models, with runtime and kernel optimizations for inference workloads.

View repository Visit site Mentioned in 9 videos ↓

Overview

Its documented capabilities include RadixAttention, a zero-overhead batch scheduler, cache-aware load balancing, structured-output decoding, speculative decoding, disaggregated prefill and decode, model parallelism, and support for GPU and TPU backends. The project also publishes integrations and optimizations for current open models and multimodal, image, and video-generation workloads.

What SGLang is used for

9 uses taken from transcripts — each links to the moment in the video.

  • Provides an open-source inference engine whose runtime behavior and kernel issues are compared with other serving engines.

  • Provides an alternative inference-serving stack for the deployed model.

  • An inference runtime tested alongside vLLM for synthetic-data generation.

  • Served the vision-language model behind the museum placard application and provided the reference inference-engine architecture.

  • Alternative serving runtime good for shared long prefixes and multi-turn chats

  • LLM-serving system that uses batching and continuous batching to improve performance

  • Inference stack for serving open-weight models internally.

    unverified InfoQ22:04
  • Inference engine listed as an alternative integration point inside the engine.

  • Alternative self-hosted inference engine the same ingress admission policy works with; deeper decision points would require maintaining a modified fork

Videos mentioning SGLang

9 in the library.