Magnitude

MTPLX is an open-source native Mac app and command-line tool for running local language models on Apple Silicon. It uses a model's built-in multi-token prediction (MTP) heads to draft several tokens, verifies the block in one batched forward pass, and commits tokens with exact rejection sampling and residual correction, without requiring a separate draft model or changing the model's sampling distribution. It provides local OpenAI-compatible and Anthropic-compatible APIs, native chat, model management, hardware-specific draft-depth tuning, and tooling for building and verifying MTP models. The server can also expose MLX embedding and reranking models. MTPLX requires an M1-or-newer Mac running macOS 14 or later, and the repository is licensed under Apache-2.0.

codebase-memory-mcp is an MCP server from DeusData that indexes a codebase into a persistent local knowledge graph for AI coding agents. It parses source with tree-sitter AST grammars and uses Hybrid LSP semantic type resolution for selected languages to model functions, classes, call chains, HTTP routes, and cross-service links. Its MCP tools support code search, structural and Cypher-style queries, call tracing, architecture overviews, impact analysis, dead-code detection, and graph visualization. The project is distributed as native executables for macOS, Linux, and Windows, with no language runtime, hosted service, or API key required. Processing is local; the repository describes a RAM-first pipeline using compressed data, in-memory SQLite, and fused Aho-Corasick pattern matching, and reports sub-millisecond structural queries. A local visualization interface is available at localhost:9749.

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, including on CPUs, GPUs, laptops, Apple silicon devices, and small computers such as Raspberry Pis. The project is built on the ggml library and uses integer quantization from 1.5-bit through 8-bit to reduce memory use and accelerate inference, with CPU, CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and other hardware backends. It supports hybrid CPU-GPU inference for models larger than available VRAM and includes command-line tools, a built-in web interface, and an OpenAI-compatible REST API server. The project is distributed as source code, Docker images, and pre-built binaries, with installation instructions at llama.app.

Needle 2 is an open-source, 45-million-parameter foundation model for tool calling, device use, and structured extraction on small devices. Its weights are compressed into a single 14 MB binary and integrated with its inference engine; the repository reports that a full session uses about 28 MB of RAM and performs inference without network access after setup. The model converts text and tool descriptions into schema-constrained JSON calls using a byte-level grammar, attaches a learned confidence score for threshold-based escalation, and can retrieve the top five tools from a larger catalogue. It uses a 256-token sliding window with tools pinned as key-value sinks to bound memory, and its Simple Attention Network combines a Hadamard MLP, grouped-query attention, engram key-value memory, and multi-lane hyper-connections. The Python package supports inference, LoRA fine-tuning, and checkpoint export; the inference engine is fetched from Hugging Face and cached, while the training stack is an optional installation.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.