Triton

Inductor is PyTorch's default backend compiler for optimizing compiled models. It fuses operations, schedules work, and generates Triton kernels for GPUs or C++ code for CPUs as part of the torch.compile pipeline, alongside TorchDynamo and AOT Autograd.

Dynamo is the first compilation stage used by PyTorch's torch.compile system. It traces Python execution, captures tensor operations in an FX graph, and records guards—assumptions about the inputs—that determine when the captured graph can be reused.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.

torch.compile is PyTorch's model compilation tool. It inspects a model before execution and converts as much of its computation as possible into an optimized region, allowing compiler transformations such as kernel fusion, operation reordering, and memory planning. Python scalar reads such as `.item()` can cause graph breaks that fragment these regions and force synchronization between Python and the accelerator, reducing available optimization and potentially slowing execution.

DeepJIT is a lightweight, header-only C++20 library for just-in-time compilation, caching, loading, and launching of device kernels on NVIDIA CUDA GPUs and Huawei Ascend NPUs. It provides a shared runtime interface for both backends while keeping kernel source and compiler options backend-specific. The runtime hashes kernel source, tracked includes, compiler information, effective options, and application-provided dependency signatures to reuse compiled artifacts through in-memory and on-disk caches. Cache directories can be shared across users, processes, and nodes on local or distributed filesystems, with completed entries published through atomic directory renames. Lazy initialization defers device and compiler discovery until first use. DeepJIT integrates with PyTorch CUDA and torch_npu streams and can expose a runtime through pybind11. The CUDA backend uses NVCC and the CUDA Driver API to produce, load, and launch CUBIN files; the Ascend backend uses Bisheng, ld.lld, and ACL to compile, link, load, and launch kernels. It supports per-kernel compiler and launch options, compilation diagnostics, and optional PTX, SASS, or Ascend assembly dumps. The host environment requires Linux, a C++20 compiler with std::format support, Python, pybind11, and the dependencies for the selected backend; it is intended to be embedded in another C++ or Python extension as a header-only dependency.