Deltafin

Kimi K3 in C is a portable C99 inference engine for running the 2.78-trillion-parameter Kimi K3 base model on CPUs without BLAS, an inference framework, or a GPU. It reads the 1.56 TB checkpoint from disk, keeps a selected depth of the 93-layer dense trunk in memory, and streams the remainder during generation; its routed experts are packed in 4-bit form, multiplied directly from that representation, and managed with an LRU cache. The implementation includes KDA attention with a non-growing memory, MLA attention with a latent representation, and selection of 16 experts from 896. The repository provides a command-line interface, weightless and full-checkpoint validation gates, and measurement data; it reports operation from 8 GB of RAM upward, with additional memory changing speed rather than the generated output. The base model has no chat template, so prompts produce continuations rather than chat replies. Reference requirements include an AVX2-and-FMA CPU, about 1.7 TB of free storage, and a C compiler; Linux/x86-64 is the reference platform, while macOS/arm64 and Windows/x86-64 are also documented as building and passing the tests.

kimi-k3-mlx is an MLX port for Apple Silicon of Moonshot’s Kimi K3 native-multimodal mixture-of-experts model. It provides a stock `mlx-lm`-compatible text model definition, a 3D video-capable MoonViT vision tower with a patch-merger projector, and an `mlx-vlm`-shaped wrapper that joins the language and vision components. The model has a 1-million-token context and uses routed and shared experts, Kimi Delta Attention, gated multi-head latent attention, Attention Residuals, latent-dimension experts, and the SiTU-GLU activation; routed experts are distributed in MXFP4 while other weights use bf16. The multimodal processor expands each image placeholder into a feature block rather than performing a same-length one-token-per-image scatter. The repository includes a streaming converter because standard `mlx-lm` conversion would materialize the multi-terabyte bf16 model, along with REAP expert-pruning scripts and expert-overlap analysis using Chinese, English, code, or mixed calibration data. It also contains vision-parity tests against Moonshot’s reference implementation. The repository states that the full model cannot run on a single Mac: even its smallest deployment tier requires hundreds of gigabytes of memory and exceeds the capacity of the largest Apple Silicon machine.

edge0 is an open-source streaming Mixture-of-Experts inference framework for running sparse MoE models on Apple Silicon Macs. Its MLX backend memory-maps expert weights from SSD storage and loads them on demand, so peak memory depends on the active experts rather than the model's total parameter count. A trained prerouter predicts the next expert routes so storage reads can overlap the forward pass, while Recover-LoRA adapters distill from the full-precision model to compensate for quantization loss; the base weights remain read-only and the adapters stay separate. The framework ships two end-to-end model tiers with bundled 4-bit checkpoints, LoRA adapters, and prerouter heads. It provides a command-line interface and Python API, supports demo, chat, and server modes, and exposes an OpenAI-compatible `/v1/chat/completions` endpoint. The current backend supports macOS on Apple Silicon M1 through M4; CUDA support is identified as planned. The repository is licensed under Apache-2.0.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.

DwarfStar is a self-contained, model-specific native inference engine for running DeepSeek V4 Flash, GLM 5.2, and, on high-memory systems, DeepSeek V4 PRO locally. It combines model loading, prompt rendering, tool calls, persistent KV state, an HTTP server, and a coding agent rather than serving as a general GGUF runner. The engine supports Metal on Macs, CUDA including multi-GPU systems, and ROCm on Strix Halo hardware; it also provides SSD streaming for machines with insufficient RAM, distributed tensor and pipeline parallelism, and server-side micro-batching for multi-user inference. The repository includes tooling and data for GGUF, imatrix, quality, and speed testing, and acknowledges code and design contributions from llama.cpp and GGML.