lumabri

Colibrì is a pure-C inference engine and open research platform for running large mixture-of-experts models on consumer and heterogeneous hardware without engine dependencies. It treats VRAM, system RAM, and storage as a unified multitier inference hierarchy, streaming expert weights from disk and coordinating placement, storage I/O, scheduling, kernels, speculation, and CPU/GPU overlap. The project provides chat, server, and web front ends, with support for CPU, CUDA, Metal, and dual-SSD streaming; its design rules prohibit silently changing model precision or router semantics when fast memory is insufficient.

MoonEP is an expert-parallelism communication library from Moonshot AI for mixture-of-experts training and inference. It plans a small number of dynamic redundant experts from the current router outputs, prefetches their weights before expert computation, and reduces the resulting gradients back to the experts' home ranks during the backward pass. Its near-optimal GPU planning kernel, fused permute/unpermute operations, and zero-copy dispatch send tokens directly to expert-grouped positions on remote ranks and return buffer views to computation. The library uses a fixed S × K token buffer per rank, where S is the input-token count and K is the routed top-k, keeping communication and computation shapes static across layers. The repository documents support for NVIDIA GPUs and lists Zhenwu PPU support as under review.

Murmell is a cloud platform for running and coordinating AI coding agents (for example Codex, Claude, and Kimi). It provides a shared cloud sandbox for collaborative coding, real-time file-path locking to prevent merge conflicts, and tools to deploy from any terminal.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

edge0 is an open-source streaming Mixture-of-Experts inference framework for running sparse MoE models on Apple Silicon Macs. Its MLX backend memory-maps expert weights from SSD storage and loads them on demand, so peak memory depends on the active experts rather than the model's total parameter count. A trained prerouter predicts the next expert routes so storage reads can overlap the forward pass, while Recover-LoRA adapters distill from the full-precision model to compensate for quantization loss; the base weights remain read-only and the adapters stay separate. The framework ships two end-to-end model tiers with bundled 4-bit checkpoints, LoRA adapters, and prerouter heads. It provides a command-line interface and Python API, supports demo, chat, and server modes, and exposes an OpenAI-compatible `/v1/chat/completions` endpoint. The current backend supports macOS on Apple Silicon M1 through M4; CUDA support is identified as planned. The repository is licensed under Apache-2.0.