NVIDIA Model Optimizer

An Anthropic reference release containing 36 drop-in inference-optimization kits for open protein- and genomics-machine-learning tools, covering structure prediction, cofolding, binder and sequence design, protein-language-model inference, and genomic prediction. Each kit carries a pinned upstream release in a stock directory and applies optimizations under a named mode without changing the upstream calling interface. Modes include off for the unmodified upstream tool, exact for identical outputs with faster inference, fast for documented numerical differences, and big for lower peak GPU memory; some kits also support splitting a prediction across multiple GPUs. The kits share runtime code for mode resolution, determinism, memory handling, multi-GPU execution, and GPU kernels, and can be installed through Docker, Apptainer, or a pinned Python environment. The repository is a non-maintained reference release accompanying Anthropic's biomolecular-modeling blog post; its original code is under Apache License 2.0, while the carried upstream projects and model weights have their own licenses and terms.

AirLLM is an open-source Python library for running large language model inference on GPUs with limited memory. It reduces GPU memory use by loading and processing one model layer at a time instead of holding the entire model in GPU memory; for sparse mixture-of-experts models, it streams individual experts as needed. The project is designed to run large models without requiring quantization, distillation, or pruning, and provides a pip-installable package with support for multiple model families, CPU inference, macOS, and optional model compression and quantization features.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.

An Apache-2.0-licensed serving setup for Qwen3.8-27B on a single 24 GB NVIDIA GPU, built around vLLM and exposing an OpenAI-compatible API with optional key authentication. Its preparation pipeline starts from a W4A16 AutoRound model, requantizes the language-model head and embedding matrices to int8, requantizes the MTP components, and applies vLLM patches for the serving stack. The runtime combines int8 tensor-core GEMMs with an fp16 recurrent state, continuous batching, split-KV verification attention, and speculative decoding through either Qwen's MTP path or the optional DFlash2 block drafter; DFlash2 can also draft from the request's cached context for document-reproduction workloads. The repository provides Docker Compose profiles for batch throughput and single-user latency, prebuilt container images, model-download and requantization scripts, benchmark and quality-test tools, and launchers for fast, long-context, and experimental KVarN cache modes. The standard configurations target roughly 64k to 150k tokens of context, while KVarN and alternative int4 KV-cache paths extend the claimed capacity to the 256k-token range with lossy KV quantization. The setup requires a recent NVIDIA driver and a compatible Ampere-or-newer GPU; the repository notes that its benchmark figures were measured on an RTX 3090 subject to a 250 W power limit.

Transformers is Hugging Face's open-source model-definition framework and Python library for inference and training with pretrained text, computer vision, audio, video, and multimodal machine-learning models. It centralizes model definitions behind a unified API and provides the Pipeline interface, which preprocesses inputs and returns task-specific outputs for applications such as text generation, automatic speech recognition, image classification, and visual question answering. Models can be downloaded and cached from the Hugging Face Hub, and the library supports moving models between PyTorch, JAX, and TensorFlow workflows while exposing model internals for customization. It requires Python 3.10 or later and PyTorch 2.5 or later for the documented installation.