XLA

JAX is a Python library for accelerator-oriented array computation and program transformation, designed for numerical computing and large-scale machine learning. It transforms Python and NumPy-style functions through automatic differentiation with `jax.grad`, automatic batching and vectorization with `jax.vmap`, and just-in-time compilation with `jax.jit`. It uses XLA to compile computations for GPUs, TPUs, and other accelerators. JAX supports compiler-driven multi-device execution: `Mesh`, `PartitionSpec`, and `NamedSharding` declare array placement across device meshes, allowing distributed training code to include automatically managed parallel work such as gradient synchronization. Its compilation tools also include ahead-of-time compilation, portable artifact export with `jax.export`, and conversion to TensorFlow SavedModels through `jax2tf` for serving integrations. The project describes itself as a research project rather than an official Google product.

vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.

MLX is an array framework for machine learning on Apple silicon, developed by Apple machine learning research. It provides Python, C++, C, and Swift APIs modeled on NumPy, plus higher-level packages such as mlx.nn and mlx.optimizers with PyTorch-like interfaces. Its composable transformations support automatic differentiation, automatic vectorization, and computation-graph optimization; computations are lazy, graphs are built dynamically, and operations can run on supported CPUs and GPUs using shared unified memory rather than explicit data transfers. MLX is distributed through PyPI and supports model training and deployment workflows including language-model training and generation, LoRA fine-tuning, image generation, and speech recognition.

Flax NNX is a neural-network API for JAX developed by the Flax team in collaboration with the JAX team. Released in 2024, it uses Python reference semantics so models can be expressed as regular Python objects with reference sharing and mutability, supporting inspection, debugging, and analysis. Its API includes layers and modules such as Linear, convolution, normalization, attention, recurrent cells, and dropout, along with patterns for replicated training, serialization, and checkpointing. In distributed training, it can manage model state while JAX handles computation across a device mesh.

An open GitHub guide and configuration project by jamesob for running state-of-the-art large language models locally. It documents a multi-GPU system using PCIe switches so GPUs can communicate peer-to-peer, including hardware selection, BIOS bifurcation, link-speed and ASPM settings, kernel and GRUB parameters, ACS configuration, and GPU power limiting. The repository also provides Docker-based serving configurations, a local speech-to-text configuration, and a GPU peer-to-peer bandwidth and latency benchmark.