Flare (NVIDIA Spark Local Inference)

Project Maya is an open-source local inference system for running large mixture-of-experts language models on personal hardware. Created and led by Peasant Smith and co-maintained with needmorevram, it provides model-specific engines for GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, and experimental MiniMax-M3.1-Flash, with support for NVIDIA or AMD GPUs, Linux, and experimental Windows configurations. Maya places frequently used experts in GPU memory, keeps additional experts in RAM, and reads the remainder from an NVMe SSD, moving them as inference proceeds. Its setup measures the available GPUs, CPU, RAM, and storage and tunes the configuration; layers can be split across multiple GPUs, and optional drafter, quantization, vision, KV-cache streaming, and conversation-checkpoint features are selected according to the model and hardware. The project includes a browser chat interface and hardware monitor, an OpenAI-compatible API, an Anthropic-compatible API, tool calls, image inputs, thinking-level controls, concurrent requests, and a Responses API with streaming and function-tool support. It is free and distributed under the MIT License; the repository notes that some optional model components have separate licenses or non-commercial restrictions.

DwarfStar is a self-contained, model-specific native inference engine for running DeepSeek V4 Flash, GLM 5.2, and, on high-memory systems, DeepSeek V4 PRO locally. It combines model loading, prompt rendering, tool calls, persistent KV state, an HTTP server, and a coding agent rather than serving as a general GGUF runner. The engine supports Metal on Macs, CUDA including multi-GPU systems, and ROCm on Strix Halo hardware; it also provides SSD streaming for machines with insufficient RAM, distributed tensor and pipeline parallelism, and server-side micro-batching for multi-user inference. The repository includes tooling and data for GGUF, imatrix, quality, and speed testing, and acknowledges code and design contributions from llama.cpp and GGML.

NVIDIA DGX Spark is a compact AI workstation from NVIDIA for local development, fine-tuning, and inference of generative-AI models. It is built around the NVIDIA GB10 Grace Blackwell Superchip, with 128 GB of unified memory and up to 1 petaflop of FP4 AI performance, and runs NVIDIA DGX OS.

FriendliAI is an inference cloud and API for running frontier open-weight and custom AI models in production, with support for long-context inference, real-time streaming, tool calling, and agent workloads. Its stack combines custom GPU kernels, continuous batching, parallel inference, prefix caching with cache-aware routing, and a GPU-, CPU-, and NVMe-based KV-cache hierarchy for repetitive or long agent inputs. It also uses hybrid speculative decoding that adapts to live traffic, alongside multi-cloud scaling and autoscaling. The service provides model APIs and one-click deployment for models from Hugging Face, as well as deployment support for proprietary or fine-tuned models. FriendliAI advertises 99.99% uptime SLAs and reports performance claims including up to 7x faster output token speed and lower inference costs.

Fireworks AI's inference engine for serving AI models. It uses custom-built CUDA kernels that can be optimized for a customer's workload to target latency and performance requirements. The video credits FireAttention and related Fireworks serving work with enabling 13× faster inference for Cursor's Composer 2.