GuideLLM

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

llmfit is a terminal tool that evaluates which local language models fit and run well on a user's hardware. It detects system specifications including RAM, CPU, GPU, and multi-GPU configurations, then scores models across fit, estimated speed, quality, and context dimensions while accounting for quantization and mixture-of-experts architectures. It provides an interactive terminal UI and a classic CLI, supports local runtime providers including Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio, and can output recommendations as JSON. Its benchmarking workflow can measure real token-per-second performance on a machine and contribute those results to the project's fit data.

An open GitHub guide and configuration project by jamesob for running state-of-the-art large language models locally. It documents a multi-GPU system using PCIe switches so GPUs can communicate peer-to-peer, including hardware selection, BIOS bifurcation, link-speed and ASPM settings, kernel and GRUB parameters, ACS configuration, and GPU power limiting. The repository also provides Docker-based serving configurations, a local speech-to-text configuration, and a GPU peer-to-peer bandwidth and latency benchmark.

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, including on CPUs, GPUs, laptops, Apple silicon devices, and small computers such as Raspberry Pis. The project is built on the ggml library and uses integer quantization from 1.5-bit through 8-bit to reduce memory use and accelerate inference, with CPU, CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and other hardware backends. It supports hybrid CPU-GPU inference for models larger than available VRAM and includes command-line tools, a built-in web interface, and an OpenAI-compatible REST API server. The project is distributed as source code, Docker images, and pre-built binaries, with installation instructions at llama.app.

WebLLM is an in-browser large language model inference engine developed by the MLC community. It runs model inference directly in web browsers using WebGPU hardware acceleration, without server-side processing, and can be used as an npm, Yarn, or CDN package for building web applications. Its MLCEngine exposes an OpenAI-compatible chat-completions interface with streaming, JSON-mode structured generation, seeding, and preliminary function-calling support. It supports model families including Llama, Phi, Gemma, Mistral, and Qwen, and can load custom MLC-format models consisting of model artifacts and WebAssembly model libraries. WebLLM provides Web Worker and Service Worker integrations, Chrome-extension examples, browser caching through Cache API, IndexedDB, or OPFS, and optional Subresource Integrity verification for downloaded model artifacts.