FreeToken

edge0 is an open-source streaming Mixture-of-Experts inference framework for running sparse MoE models on Apple Silicon Macs. Its MLX backend memory-maps expert weights from SSD storage and loads them on demand, so peak memory depends on the active experts rather than the model's total parameter count. A trained prerouter predicts the next expert routes so storage reads can overlap the forward pass, while Recover-LoRA adapters distill from the full-precision model to compensate for quantization loss; the base weights remain read-only and the adapters stay separate. The framework ships two end-to-end model tiers with bundled 4-bit checkpoints, LoRA adapters, and prerouter heads. It provides a command-line interface and Python API, supports demo, chat, and server modes, and exposes an OpenAI-compatible `/v1/chat/completions` endpoint. The current backend supports macOS on Apple Silicon M1 through M4; CUDA support is identified as planned. The repository is licensed under Apache-2.0.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.

Project Maya is an open-source local inference system for running large mixture-of-experts language models on personal hardware. Created and led by Peasant Smith and co-maintained with needmorevram, it provides model-specific engines for GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, and experimental MiniMax-M3.1-Flash, with support for NVIDIA or AMD GPUs, Linux, and experimental Windows configurations. Maya places frequently used experts in GPU memory, keeps additional experts in RAM, and reads the remainder from an NVMe SSD, moving them as inference proceeds. Its setup measures the available GPUs, CPU, RAM, and storage and tunes the configuration; layers can be split across multiple GPUs, and optional drafter, quantization, vision, KV-cache streaming, and conversation-checkpoint features are selected according to the model and hardware. The project includes a browser chat interface and hardware monitor, an OpenAI-compatible API, an Anthropic-compatible API, tool calls, image inputs, thinking-level controls, concurrent requests, and a Responses API with streaming and function-tool support. It is free and distributed under the MIT License; the repository notes that some optional model components have separate licenses or non-commercial restrictions.

MoonEP is an expert-parallelism communication library from Moonshot AI for mixture-of-experts training and inference. It plans a small number of dynamic redundant experts from the current router outputs, prefetches their weights before expert computation, and reduces the resulting gradients back to the experts' home ranks during the backward pass. Its near-optimal GPU planning kernel, fused permute/unpermute operations, and zero-copy dispatch send tokens directly to expert-grouped positions on remote ranks and return buffer views to computation. The library uses a fixed S × K token buffer per rank, where S is the input-token count and K is the routed top-k, keeping communication and computation shapes static across layers. The repository documents support for NVIDIA GPUs and lists Zhenwu PPU support as under review.

TensorFold is an Apache-2.0 LLM inference engine that serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs. It reads model checkpoints, tokenizes requests, and runs Metal or CUDA kernels directly without requiring Python or MLX for serving; the project page also identifies Vulkan support. Models can be downloaded into a Hugging Face cache with checkpoint integrity checks and served through an OpenAI-compatible HTTP API supporting chat completions, completions, Responses, Anthropic Messages, and model discovery. It also provides a local web interface, a desktop workspace bundling the engine and model management, and an experimental Living Weights mode that writes validated learned facts into a checkpoint directory.