CUDA (Compute Unified Device Architecture)

NVIDIA Dynamo is a developer toolkit from NVIDIA for managing distributed key–value (KV) context caches used in large‑language‑model inference. It provides KV‑aware routing, cache offloading, and prefill/decode disaggregation to move KV cache and related activation/state across GPU clusters, reducing memory pressure and supporting scalable LLM serving. It is intended for integration into GPU‑based inference and serving stacks.

DeepSelect is a high-performance CUDA and PyTorch implementation of TopK kernels for DeepSeek Sparse Attention and sampling workloads. It replaces vanilla `torch.topk` for supported inputs, covering bfloat16 lightning-indexer workloads and float32 sampling workloads, with configurable sorting, index type, value output, variable-length rows, and NaN handling. The repository reports 2–20× speedups against `torch.topk` on its benchmarks. It is distributed as an installable Python package from the `deepseek-ai` GitHub repository. Supported TopK values are limited to 4096 or less, and input tensors must meet specified stride and contiguity requirements; the optimized sampling case targets vocabularies of approximately 128K.

DeepJIT is a lightweight, header-only C++20 library for just-in-time compilation, caching, loading, and launching of device kernels on NVIDIA CUDA GPUs and Huawei Ascend NPUs. It provides a shared runtime interface for both backends while keeping kernel source and compiler options backend-specific. The runtime hashes kernel source, tracked includes, compiler information, effective options, and application-provided dependency signatures to reuse compiled artifacts through in-memory and on-disk caches. Cache directories can be shared across users, processes, and nodes on local or distributed filesystems, with completed entries published through atomic directory renames. Lazy initialization defers device and compiler discovery until first use. DeepJIT integrates with PyTorch CUDA and torch_npu streams and can expose a runtime through pybind11. The CUDA backend uses NVCC and the CUDA Driver API to produce, load, and launch CUBIN files; the Ascend backend uses Bisheng, ld.lld, and ACL to compile, link, load, and launch kernels. It supports per-kernel compiler and launch options, compilation diagnostics, and optional PTX, SASS, or Ascend assembly dumps. The host environment requires Linux, a C++20 compiler with std::format support, Python, pybind11, and the dependencies for the selected backend; it is intended to be embedded in another C++ or Python extension as a header-only dependency.

NVIDIA GPUs are a family of graphics processing units produced by NVIDIA Corporation, intended for graphics rendering, high-performance computing, and hardware acceleration of workloads such as machine learning and scientific computing. Major product lines include GeForce for consumer gaming, RTX A (formerly Quadro) for professional visualization, and data-center GPUs (e.g., A100, H100) for AI training and inference.

Tensor Processing Unit (TPU) is a family of application-specific integrated circuits (ASICs) and managed services developed and operated by Google to accelerate machine learning workloads. TPUs are available as on-premises hardware and as a managed Cloud TPU service for training and inference of neural networks and deep learning models. First announced in 2016, TPUs are used inside Google's datacenters and offered to external customers via Google Cloud.