LLM Compressor

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, built on the ggml library. It supports integer quantization from 1.5-bit through 8-bit, including GGUF model workflows, and provides CPU, GPU, accelerator, and hybrid CPU–GPU backends across Apple Silicon, x86, RISC-V, NVIDIA, AMD, Intel, Vulkan, and other hardware targets, including small computers such as Raspberry Pis. Its command-line tools can download and run models, while llama-server provides an OpenAI-compatible REST API and a built-in web UI. The project is distributed as pre-built binaries, Docker images, install scripts, or source code.

llmfit is a terminal tool that evaluates which local language models fit and run well on a user's hardware. It detects system specifications including RAM, CPU, GPU, and multi-GPU configurations, then scores models across fit, estimated speed, quality, and context dimensions while accounting for quantization and mixture-of-experts architectures. It provides an interactive terminal UI and a classic CLI, supports local runtime providers including Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio, and can output recommendations as JSON. Its benchmarking workflow can measure real token-per-second performance on a machine and contribute those results to the project's fit data.

GuideLLM is an open-source platform for benchmarking and evaluating real-world LLM inference deployments. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, using real or synthetic text and multimodal datasets and configurable execution profiles such as synchronous, concurrent, throughput, constant-rate, Poisson, and sweep workloads. It measures SLO-oriented latency and token statistics, including time to first token, inter-token latency, and end-to-end behavior, then produces console, JSON, CSV, or HTML reports for deployment tuning, capacity planning, and regression tracking. The tool provides both a CLI and API, with multiprocessing, threading, asynchronous execution, and Hugging Face, file, synthetic, or custom data sources.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.