edge0

slotstream is an open-source macOS local-LLM inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on Apple Silicon Macs whose memory cannot hold the model. Implemented as a single Swift binary using MLX, it keeps the dense trunk resident and reads routed expert weights from an SSD into a shared cache of slots, sizing that cache to available memory instead of loading the full model. It supports prefix caching for follow-up turns and exposes Ollama-compatible and OpenAI-compatible chat APIs, as well as a command-line prompt mode; the documented API subset supports streaming, CORS, and sampling options but not tools, images, JSON-schema output, or logprobs. The project requires macOS 14 or later, Apple Silicon, and roughly 110 GB of free disk space for the model weights. Its installer provides signed release binaries, and the repository can also be built with Swift and the macOS Command Line Tools. The source is MIT-licensed; the model weights are distributed under the Qwen community license.

Project Maya is an open-source local inference system for running large mixture-of-experts language models on personal hardware. Created and led by Peasant Smith and co-maintained with needmorevram, it provides model-specific engines for GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, and experimental MiniMax-M3.1-Flash, with support for NVIDIA or AMD GPUs, Linux, and experimental Windows configurations. Maya places frequently used experts in GPU memory, keeps additional experts in RAM, and reads the remainder from an NVMe SSD, moving them as inference proceeds. Its setup measures the available GPUs, CPU, RAM, and storage and tunes the configuration; layers can be split across multiple GPUs, and optional drafter, quantization, vision, KV-cache streaming, and conversation-checkpoint features are selected according to the model and hardware. The project includes a browser chat interface and hardware monitor, an OpenAI-compatible API, an Anthropic-compatible API, tool calls, image inputs, thinking-level controls, concurrent requests, and a Responses API with streaming and function-tool support. It is free and distributed under the MIT License; the repository notes that some optional model components have separate licenses or non-commercial restrictions.

MoonEP is an expert-parallelism communication library from Moonshot AI for mixture-of-experts training and inference. It plans a small number of dynamic redundant experts from the current router outputs, prefetches their weights before expert computation, and reduces the resulting gradients back to the experts' home ranks during the backward pass. Its near-optimal GPU planning kernel, fused permute/unpermute operations, and zero-copy dispatch send tokens directly to expert-grouped positions on remote ranks and return buffer views to computation. The library uses a fixed S × K token buffer per rank, where S is the input-token count and K is the routed top-k, keeping communication and computation shapes static across layers. The repository documents support for NVIDIA GPUs and lists Zhenwu PPU support as under review.

FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine for running open-weight, frontier-scale models on consumer hardware. It treats GPUs, CPUs, host memory, and interconnects as a unified inference platform, using bandwidth-adaptive CPU–GPU co-execution, double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format. Semantic anchor checkpoints let recurrent state and KV caches survive agentic context edits without redundant recomputation, while runtime VRAM reallocation moves memory between expert caches and KV storage without restarting the engine or reloading weights. FreeToken supports multiple MoE models, quantization formats, and Anthropic- and OpenAI-compatible APIs, and is distributed as a Windows or Linux desktop application and as a Python/CLI package. The project is released under the Apache License 2.0.

oMLX is an open-source LLM inference server for Apple Silicon Macs, managed through a macOS menu-bar application. It serves text, vision, OCR, embedding, and reranking models through compatible API endpoints, with model discovery and management, monitoring, continuous batching, and caching. Its caching system keeps KV cache in a hot in-memory tier and a cold SSD tier, allowing cached context to remain reusable across requests even when the conversation context changes. The project can be installed as a macOS app, through Homebrew, or from source, and includes a CLI for controlling the app-managed server. The repository states that it requires macOS 15 or later, Python 3.11–3.13, and Apple Silicon hardware.