NVIDIA Dynamo

CUDA is a parallel computing platform and programming model developed by NVIDIA that enables general-purpose computing on NVIDIA GPUs. It includes the CUDA Toolkit (compilers, libraries, and developer tools) and APIs for C, C++ and Fortran to accelerate compute-intensive applications in areas such as scientific computing, HPC, graphics, and machine learning.

DwarfStar is a self-contained, model-specific native inference engine for running DeepSeek V4 Flash, GLM 5.2, and, on high-memory systems, DeepSeek V4 PRO locally. It combines model loading, prompt rendering, tool calls, persistent KV state, an HTTP server, and a coding agent rather than serving as a general GGUF runner. The engine supports Metal on Macs, CUDA including multi-GPU systems, and ROCm on Strix Halo hardware; it also provides SSD streaming for machines with insufficient RAM, distributed tensor and pipeline parallelism, and server-side micro-batching for multi-user inference. The repository includes tooling and data for GGUF, imatrix, quality, and speed testing, and acknowledges code and design contributions from llama.cpp and GGML.

NVIDIA DGX Spark is a compact AI workstation from NVIDIA for local development, fine-tuning, and inference of generative-AI models. It is built around the NVIDIA GB10 Grace Blackwell Superchip, with 128 GB of unified memory and up to 1 petaflop of FP4 AI performance, and runs NVIDIA DGX OS.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.

DeepJIT is a lightweight, header-only C++20 library for just-in-time compilation, caching, loading, and launching of device kernels on NVIDIA CUDA GPUs and Huawei Ascend NPUs. It provides a shared runtime interface for both backends while keeping kernel source and compiler options backend-specific. The runtime hashes kernel source, tracked includes, compiler information, effective options, and application-provided dependency signatures to reuse compiled artifacts through in-memory and on-disk caches. Cache directories can be shared across users, processes, and nodes on local or distributed filesystems, with completed entries published through atomic directory renames. Lazy initialization defers device and compiler discovery until first use. DeepJIT integrates with PyTorch CUDA and torch_npu streams and can expose a runtime through pybind11. The CUDA backend uses NVCC and the CUDA Driver API to produce, load, and launch CUBIN files; the Ascend backend uses Bisheng, ld.lld, and ACL to compile, link, load, and launch kernels. It supports per-kernel compiler and launch options, compilation diagnostics, and optional PTX, SASS, or Ascend assembly dumps. The host environment requires Linux, a C++20 compiler with std::format support, Python, pybind11, and the dependencies for the selected backend; it is intended to be embedded in another C++ or Python extension as a header-only dependency.