NIGHTRUN

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, including on CPUs, GPUs, laptops, Apple silicon devices, and small computers such as Raspberry Pis. The project is built on the ggml library and uses integer quantization from 1.5-bit through 8-bit to reduce memory use and accelerate inference, with CPU, CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and other hardware backends. It supports hybrid CPU-GPU inference for models larger than available VRAM and includes command-line tools, a built-in web interface, and an OpenAI-compatible REST API server. The project is distributed as source code, Docker images, and pre-built binaries, with installation instructions at llama.app.

Unsloth is an open-source local desktop and web tool for running, training, fine-tuning, and deploying language, diffusion, embedding, audio, and vision models. It supports Windows, macOS, Linux, WSL, multi-GPU setups, NVIDIA, AMD, and Intel GPUs, CPUs, and the Vulkan backend. The project is distributed as Unsloth Desktop, Unsloth Studio, and Unsloth Core. It can run local models, provide an OpenAI-compatible API, connect models to agents and MCP tools, perform search and RAG, and offer remote or LAN access. Its training functions cover reinforcement learning, LoRA, QLoRA, full fine-tuning, pretraining, GRPO, DPO, and FP8, with export formats including GGUF, NVFP4, and FP8. The repository claims that its fine-tuning workflows use two times less training time and 70% less VRAM.

An Apache-2.0-licensed serving setup for Qwen3.8-27B on a single 24 GB NVIDIA GPU, built around vLLM and exposing an OpenAI-compatible API with optional key authentication. Its preparation pipeline starts from a W4A16 AutoRound model, requantizes the language-model head and embedding matrices to int8, requantizes the MTP components, and applies vLLM patches for the serving stack. The runtime combines int8 tensor-core GEMMs with an fp16 recurrent state, continuous batching, split-KV verification attention, and speculative decoding through either Qwen's MTP path or the optional DFlash2 block drafter; DFlash2 can also draft from the request's cached context for document-reproduction workloads. The repository provides Docker Compose profiles for batch throughput and single-user latency, prebuilt container images, model-download and requantization scripts, benchmark and quality-test tools, and launchers for fast, long-context, and experimental KVarN cache modes. The standard configurations target roughly 64k to 150k tokens of context, while KVarN and alternative int4 KV-cache paths extend the claimed capacity to the 256k-token range with lossy KV quantization. The setup requires a recent NVIDIA driver and a compatible Ampere-or-newer GPU; the repository notes that its benchmark figures were measured on an RTX 3090 subject to a 250 W power limit.

An open GitHub guide and configuration project by jamesob for running state-of-the-art large language models locally. It documents a multi-GPU system using PCIe switches so GPUs can communicate peer-to-peer, including hardware selection, BIOS bifurcation, link-speed and ASPM settings, kernel and GRUB parameters, ACS configuration, and GPU power limiting. The repository also provides Docker-based serving configurations, a local speech-to-text configuration, and a GPU peer-to-peer bandwidth and latency benchmark.

slotstream is an open-source macOS local-LLM inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on Apple Silicon Macs whose memory cannot hold the model. Implemented as a single Swift binary using MLX, it keeps the dense trunk resident and reads routed expert weights from an SSD into a shared cache of slots, sizing that cache to available memory instead of loading the full model. It supports prefix caching for follow-up turns and exposes Ollama-compatible and OpenAI-compatible chat APIs, as well as a command-line prompt mode; the documented API subset supports streaming, CORS, and sampling options but not tools, images, JSON-schema output, or logprobs. The project requires macOS 14 or later, Apple Silicon, and roughly 110 GB of free disk space for the model weights. Its installer provides signed release binaries, and the repository can also be built with Swift and the macOS Command Line Tools. The source is MIT-licensed; the model weights are distributed under the Qwen community license.