qwen38-27b-rtx3090

slotstream is an open-source macOS local-LLM inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on Apple Silicon Macs whose memory cannot hold the model. Implemented as a single Swift binary using MLX, it keeps the dense trunk resident and reads routed expert weights from an SSD into a shared cache of slots, sizing that cache to available memory instead of loading the full model. It supports prefix caching for follow-up turns and exposes Ollama-compatible and OpenAI-compatible chat APIs, as well as a command-line prompt mode; the documented API subset supports streaming, CORS, and sampling options but not tools, images, JSON-schema output, or logprobs. The project requires macOS 14 or later, Apple Silicon, and roughly 110 GB of free disk space for the model weights. Its installer provides signed release binaries, and the repository can also be built with Swift and the macOS Command Line Tools. The source is MIT-licensed; the model weights are distributed under the Qwen community license.

Swiftlet is an open-source Swift and Metal runtime for running Qwen3-Next and Qwen3.5/3.6 mixture-of-experts models locally on Apple devices, including iPhones. It keeps the models' small dense core in memory and streams routed expert weights from storage on demand, allowing 35B and 80B models to run with comparatively low RAM. The project is available as a Swift package, command-line interface, OpenAI-compatible server, and iOS app; its repository describes the runtime as working end to end and notes that only about 3B parameters are active per token in the supported models.

NIGHTRUN is a local LLM runtime written in Rust that boots directly from a USB stick on ordinary x86_64 PCs or from an SD card on a Raspberry Pi 5, without a conventional operating system underneath. Its UEFI application loads a quantized model into RAM, renders a chat terminal through the framebuffer, and runs inference locally and offline without a network stack, host process, or kernel. The installer builds and flashes a bootable image, guides model selection and verified downloads, and supports GGUF inspection and conversion into NIGHTRUN's .nrm container. During boot, the model is streamed into RAM with checksummed sections; storage is then sealed so generation cannot read from disk. The runtime includes custom framebuffer rendering, USB keyboard input, multi-core inference, serial diagnostics, hand-written quantized kernels for x86_64 and Raspberry Pi architectures, and Q8_0, Q4_K, and Q6_K weight formats. It requires Linux, Rust, and approximately 6 GB of free disk space for the model and image.

APEX is an open, verification-first RTL design for an LLM-inference tile developed with Sigmantic AI. It implements one transformer decoder layer—including attention, softmax, RMSNorm, RoPE, SwiGLU, residual processing, and KV-cache compression—and runs Qwen2.5-0.5B through the verified pipeline on FPGA hardware. The KV-cache codec sits in the datapath: cached keys and values are compressed as they are produced and decompressed when consumed, while an importance unit allocates precision according to which parts of the context matter. Each RTL block is checked against an executable NumPy golden model with bit-exact comparisons and mutation-tested testbenches. The repository covers the inference tile rather than a complete system; DRAM control, PCIe, and NoC are out of scope.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.