KTransformers

Kimi K3 in C is a portable C99 inference engine for running the 2.78-trillion-parameter Kimi K3 base model on CPUs without BLAS, an inference framework, or a GPU. It reads the 1.56 TB checkpoint from disk, keeps a selected depth of the 93-layer dense trunk in memory, and streams the remainder during generation; its routed experts are packed in 4-bit form, multiplied directly from that representation, and managed with an LRU cache. The implementation includes KDA attention with a non-growing memory, MLA attention with a latent representation, and selection of 16 experts from 896. The repository provides a command-line interface, weightless and full-checkpoint validation gates, and measurement data; it reports operation from 8 GB of RAM upward, with additional memory changing speed rather than the generated output. The base model has no chat template, so prompts produce continuations rather than chat replies. Reference requirements include an AVX2-and-FMA CPU, about 1.7 TB of free storage, and a C compiler; Linux/x86-64 is the reference platform, while macOS/arm64 and Windows/x86-64 are also documented as building and passing the tests.

An open GitHub guide and configuration project by jamesob for running state-of-the-art large language models locally. It documents a multi-GPU system using PCIe switches so GPUs can communicate peer-to-peer, including hardware selection, BIOS bifurcation, link-speed and ASPM settings, kernel and GRUB parameters, ACS configuration, and GPU power limiting. The repository also provides Docker-based serving configurations, a local speech-to-text configuration, and a GPU peer-to-peer bandwidth and latency benchmark.

Deltafin is a native binary for running the full, unpruned Kimi K3 mixture-of-experts model on consumer hardware, along with an OpenAI-compatible API server for local chat and coding agents. Its demonstration approach keeps the model's attention spine on local disk and fetches routed experts from Hugging Face's CDN as needed, allowing the full model to run on a 64-gigabyte MacBook rather than loading the entire expert bank into memory. The project emphasizes preserving Kimi K3's original expert weights and having K3 verify every token; it is presented as an experiment in pushing large-model inference on local hardware rather than as a practical chat setup.

AirLLM is an open-source Python library for running large language model inference on GPUs with limited memory. It reduces GPU memory use by loading and processing one model layer at a time instead of holding the entire model in GPU memory; for sparse mixture-of-experts models, it streams individual experts as needed. The project is designed to run large models without requiring quantization, distillation, or pruning, and provides a pip-installable package with support for multiple model families, CPU inference, macOS, and optional model compression and quantization features.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.