DeepInfer

DeepSeek (深度求索) is an AI research project and model provider that develops and open-sources large foundation models (notably named DeepSeek-V4 and DeepSeek-R1) and provides hosted inference via a chat interface (“DeepSeek AI”) and an API. It publishes open-source model releases and offers a hosted API for text-based prompt inference; video reports state DeepSeek models can be exposed as external backends and routed into the Codex model picker via Codex Router.

DwarfStar is a self-contained, model-specific native inference engine for running DeepSeek V4 Flash, GLM 5.2, and, on high-memory systems, DeepSeek V4 PRO locally. It combines model loading, prompt rendering, tool calls, persistent KV state, an HTTP server, and a coding agent rather than serving as a general GGUF runner. The engine supports Metal on Macs, CUDA including multi-GPU systems, and ROCm on Strix Halo hardware; it also provides SSD streaming for machines with insufficient RAM, distributed tensor and pipeline parallelism, and server-side micro-batching for multi-user inference. The repository includes tooling and data for GGUF, imatrix, quality, and speed testing, and acknowledges code and design contributions from llama.cpp and GGML.

DeepSelect is a high-performance CUDA and PyTorch implementation of TopK kernels for DeepSeek Sparse Attention and sampling workloads. It replaces vanilla `torch.topk` for supported inputs, covering bfloat16 lightning-indexer workloads and float32 sampling workloads, with configurable sorting, index type, value output, variable-length rows, and NaN handling. The repository reports 2–20× speedups against `torch.topk` on its benchmarks. It is distributed as an installable Python package from the `deepseek-ai` GitHub repository. Supported TopK values are limited to 4096 or less, and input tensors must meet specified stride and contiguity requirements; the optimized sampling case targets vocabularies of approximately 128K.

deepseek-recipe is a collection of Rust libraries and Python bindings for converting API requests in Messages, Chat Completions, and Responses formats into a shared conversation representation, encoding conversations into prompts or token IDs for DeepSeek V4 and V4.1 models, and converting model output back into protocol-specific responses. It supports streaming, text and image inputs, thinking and reasoning settings, function tools, tool calls, JSON object output, and stop sequences; its image component provides DeepSeek V4.1 preprocessing with OpenCV. The package parses model output and handles prompt encoding, while inference, tool execution, and HTTP transport are supplied externally. It includes separate components for core conversation types, prompt encoding, image processing, Python bindings, and example Axum and FastAPI servers with mock inference. Project code and public documentation are licensed under the MIT License.

Mini-SGLang is a compact implementation of the SGLang large-language-model inference and serving architecture. Its roughly 5,000-line, type-annotated Python codebase is intended as both a usable inference engine and a readable reference for understanding LLM serving systems. The framework serves models through an OpenAI-compatible API or an interactive shell. Its implementation includes radix caching for reusing KV cache across shared prefixes, chunked prefill, overlap scheduling to run CPU scheduling alongside GPU computation, tensor parallelism across multiple GPUs, and optimized kernels using FlashAttention and FlashInfer. Mini-SGLang runs on Linux with NVIDIA CUDA dependencies and supports installation from source or via Docker. The repository documents deployment on one or more GPUs, including models such as Qwen and Llama, and provides offline and online benchmark configurations.