kimi-k3-mlx

Kimi K3 in C is a portable C99 inference engine for running the 2.78-trillion-parameter Kimi K3 base model on CPUs without BLAS, an inference framework, or a GPU. It reads the 1.56 TB checkpoint from disk, keeps a selected depth of the 93-layer dense trunk in memory, and streams the remainder during generation; its routed experts are packed in 4-bit form, multiplied directly from that representation, and managed with an LRU cache. The implementation includes KDA attention with a non-growing memory, MLA attention with a latent representation, and selection of 16 experts from 896. The repository provides a command-line interface, weightless and full-checkpoint validation gates, and measurement data; it reports operation from 8 GB of RAM upward, with additional memory changing speed rather than the generated output. The base model has no chat template, so prompts produce continuations rather than chat replies. Reference requirements include an AVX2-and-FMA CPU, about 1.7 TB of free storage, and a C compiler; Linux/x86-64 is the reference platform, while macOS/arm64 and Windows/x86-64 are also documented as building and passing the tests.

oMLX is an open-source LLM inference server for Apple Silicon Macs, managed through a macOS menu-bar application. It serves text, vision, OCR, embedding, and reranking models through compatible API endpoints, with model discovery and management, monitoring, continuous batching, and caching. Its caching system keeps KV cache in a hot in-memory tier and a cold SSD tier, allowing cached context to remain reusable across requests even when the conversation context changes. The project can be installed as a macOS app, through Homebrew, or from source, and includes a CLI for controlling the app-managed server. The repository states that it requires macOS 15 or later, Python 3.11–3.13, and Apple Silicon hardware.

MLX is an array framework for machine learning on Apple silicon, developed by Apple machine learning research. It provides Python, C++, C, and Swift APIs modeled on NumPy, plus higher-level packages such as mlx.nn and mlx.optimizers with PyTorch-like interfaces. Its composable transformations support automatic differentiation, automatic vectorization, and computation-graph optimization; computations are lazy, graphs are built dynamically, and operations can run on supported CPUs and GPUs using shared unified memory rather than explicit data transfers. MLX is distributed through PyPI and supports model training and deployment workflows including language-model training and generation, LoRA fine-tuning, image generation, and speech recognition.

MTPLX is an open-source native Mac app and command-line tool for running local language models on Apple Silicon. It uses a model's built-in multi-token prediction (MTP) heads to draft several tokens, verifies the block in one batched forward pass, and commits tokens with exact rejection sampling and residual correction, without requiring a separate draft model or changing the model's sampling distribution. It provides local OpenAI-compatible and Anthropic-compatible APIs, native chat, model management, hardware-specific draft-depth tuning, and tooling for building and verifying MTP models. The server can also expose MLX embedding and reranking models. MTPLX requires an M1-or-newer Mac running macOS 14 or later, and the repository is licensed under Apache-2.0.

slotstream is an open-source macOS local-LLM inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on Apple Silicon Macs whose memory cannot hold the model. Implemented as a single Swift binary using MLX, it keeps the dense trunk resident and reads routed expert weights from an SSD into a shared cache of slots, sizing that cache to available memory instead of loading the full model. It supports prefix caching for follow-up turns and exposes Ollama-compatible and OpenAI-compatible chat APIs, as well as a command-line prompt mode; the documented API subset supports streaming, CORS, and sampling options but not tools, images, JSON-schema output, or logprobs. The project requires macOS 14 or later, Apple Silicon, and roughly 110 GB of free disk space for the model weights. Its installer provides signed release binaries, and the repository can also be built with Swift and the macOS Command Line Tools. The source is MIT-licensed; the model weights are distributed under the Qwen community license.