Strata

slotstream is an open-source macOS local-LLM inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on Apple Silicon Macs whose memory cannot hold the model. Implemented as a single Swift binary using MLX, it keeps the dense trunk resident and reads routed expert weights from an SSD into a shared cache of slots, sizing that cache to available memory instead of loading the full model. It supports prefix caching for follow-up turns and exposes Ollama-compatible and OpenAI-compatible chat APIs, as well as a command-line prompt mode; the documented API subset supports streaming, CORS, and sampling options but not tools, images, JSON-schema output, or logprobs. The project requires macOS 14 or later, Apple Silicon, and roughly 110 GB of free disk space for the model weights. Its installer provides signed release binaries, and the repository can also be built with Swift and the macOS Command Line Tools. The source is MIT-licensed; the model weights are distributed under the Qwen community license.

Swiftlet is an open-source Swift and Metal runtime for running Qwen3-Next and Qwen3.5/3.6 mixture-of-experts models locally on Apple devices, including iPhones. It keeps the models' small dense core in memory and streams routed expert weights from storage on demand, allowing 35B and 80B models to run with comparatively low RAM. The project is available as a Swift package, command-line interface, OpenAI-compatible server, and iOS app; its repository describes the runtime as working end to end and notes that only about 3B parameters are active per token in the supported models.

An Apache-2.0-licensed serving setup for Qwen3.8-27B on a single 24 GB NVIDIA GPU, built around vLLM and exposing an OpenAI-compatible API with optional key authentication. Its preparation pipeline starts from a W4A16 AutoRound model, requantizes the language-model head and embedding matrices to int8, requantizes the MTP components, and applies vLLM patches for the serving stack. The runtime combines int8 tensor-core GEMMs with an fp16 recurrent state, continuous batching, split-KV verification attention, and speculative decoding through either Qwen's MTP path or the optional DFlash2 block drafter; DFlash2 can also draft from the request's cached context for document-reproduction workloads. The repository provides Docker Compose profiles for batch throughput and single-user latency, prebuilt container images, model-download and requantization scripts, benchmark and quality-test tools, and launchers for fast, long-context, and experimental KVarN cache modes. The standard configurations target roughly 64k to 150k tokens of context, while KVarN and alternative int4 KV-cache paths extend the claimed capacity to the 256k-token range with lossy KV quantization. The setup requires a recent NVIDIA driver and a compatible Ampere-or-newer GPU; the repository notes that its benchmark figures were measured on an RTX 3090 subject to a 250 W power limit.

Unsloth is an open-source local desktop and web tool for running, training, fine-tuning, and deploying language, diffusion, embedding, audio, and vision models. It supports Windows, macOS, Linux, WSL, multi-GPU setups, NVIDIA, AMD, and Intel GPUs, CPUs, and the Vulkan backend. The project is distributed as Unsloth Desktop, Unsloth Studio, and Unsloth Core. It can run local models, provide an OpenAI-compatible API, connect models to agents and MCP tools, perform search and RAG, and offer remote or LAN access. Its training functions cover reinforcement learning, LoRA, QLoRA, full fine-tuning, pretraining, GRPO, DPO, and FP8, with export formats including GGUF, NVFP4, and FP8. The repository claims that its fine-tuning workflows use two times less training time and 70% less VRAM.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.