Colibrì

lumabri is a pure-C system for running large mixture-of-experts models across a swarm of peer machines using the colibri engine. A machine hosting a model serves it to other machines; model bytes needed during inference are fetched from peers on first use and retained in a shared content-addressed store. With Segment available, peers retain assigned layer ranges and their state while clients receive the tokenizer, embeddings, final transform or head, and conversation state; the system can instead fall back to the expert/CAS engine or local execution when a complete Segment route is unavailable. It supports CPU- and SSD-first participation as well as GPUs, and provides serving and chat commands, a terminal UI, configurable disk and compute donation, machine and health diagnostics, swarm monitoring, and low-priority peer-to-peer or signed relay transfers. The video also describes routing of missing model blocks or expert activations between peers, with hashes, signatures, and optional encryption used for protection.

edge0 is an open-source streaming Mixture-of-Experts inference framework for running sparse MoE models on Apple Silicon Macs. Its MLX backend memory-maps expert weights from SSD storage and loads them on demand, so peak memory depends on the active experts rather than the model's total parameter count. A trained prerouter predicts the next expert routes so storage reads can overlap the forward pass, while Recover-LoRA adapters distill from the full-precision model to compensate for quantization loss; the base weights remain read-only and the adapters stay separate. The framework ships two end-to-end model tiers with bundled 4-bit checkpoints, LoRA adapters, and prerouter heads. It provides a command-line interface and Python API, supports demo, chat, and server modes, and exposes an OpenAI-compatible `/v1/chat/completions` endpoint. The current backend supports macOS on Apple Silicon M1 through M4; CUDA support is identified as planned. The repository is licensed under Apache-2.0.

Kimi K3 in C is a portable C99 inference engine for running the 2.78-trillion-parameter Kimi K3 base model on CPUs without BLAS, an inference framework, or a GPU. It reads the 1.56 TB checkpoint from disk, keeps a selected depth of the 93-layer dense trunk in memory, and streams the remainder during generation; its routed experts are packed in 4-bit form, multiplied directly from that representation, and managed with an LRU cache. The implementation includes KDA attention with a non-growing memory, MLA attention with a latent representation, and selection of 16 experts from 896. The repository provides a command-line interface, weightless and full-checkpoint validation gates, and measurement data; it reports operation from 8 GB of RAM upward, with additional memory changing speed rather than the generated output. The base model has no chat template, so prompts produce continuations rather than chat replies. Reference requirements include an AVX2-and-FMA CPU, about 1.7 TB of free storage, and a C compiler; Linux/x86-64 is the reference platform, while macOS/arm64 and Windows/x86-64 are also documented as building and passing the tests.

WARP (Weight-Aware Runtime and Paging), formerly WASTE, is an embeddable inference engine written in dependency-free C by SQLiteAI. It runs large mixture-of-experts models beyond available RAM by keeping the shared model trunk in memory, streaming selected experts from NVMe or other fast storage, and using remaining RAM as a bounded expert cache. Its container layout gives each expert an aligned read; reads overlap computation, while a lookahead router prefetches experts for the next layer without changing the decisions made by the actual router. Experts use 3-bit residual vector quantization, while more sensitive shared weights use 4- or 8-bit storage. The project is designed for local inference of frontier-scale models such as Kimi K3 and GLM-5.3-Flash on consumer hardware.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.