GraalVM

TornadoVM is an add-on to the Java Virtual Machine that compiles Java code at runtime into GPU kernels, including NVIDIA PTX and SPIR-V targets. It lets developers write GPU programs in Java; the discussed example implements a GPU version of Llama 3 entirely in Java.

Java is a class-based, object-oriented programming language and computing platform originally developed by Sun Microsystems and now stewarded by Oracle and the OpenJDK community. Source code is compiled to JVM bytecode and packaged as .class, .jar and .war artifacts; the platform is distributed as JDK/JRE builds (OpenJDK, Oracle JDK, Eclipse Temurin, etc.), uses the Java Virtual Machine to run on multiple OSes, and has long-term-support releases such as Java 8 and Java 17 that are common targets for modernization and compatibility testing.

XLA (Accelerated Linear Algebra) is an open-source machine-learning compiler for GPUs, CPUs, and other ML accelerators. It takes models from frameworks such as JAX, PyTorch, and TensorFlow, traces and compiles them into optimized executable code for the target hardware. The project is developed under OpenXLA and is primarily used through the documentation and integration of the corresponding ML framework; its repository is intended mainly for compiler contributors and frontend or hardware-backend integrators.

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, built on the ggml library. It supports integer quantization from 1.5-bit through 8-bit, including GGUF model workflows, and provides CPU, GPU, accelerator, and hybrid CPU–GPU backends across Apple Silicon, x86, RISC-V, NVIDIA, AMD, Intel, Vulkan, and other hardware targets, including small computers such as Raspberry Pis. Its command-line tools can download and run models, while llama-server provides an OpenAI-compatible REST API and a built-in web UI. The project is distributed as pre-built binaries, Docker images, install scripts, or source code.

DwarfStar is a self-contained, model-specific native inference engine for running DeepSeek V4 Flash, GLM 5.2, and, on high-memory systems, DeepSeek V4 PRO locally. It combines model loading, prompt rendering, tool calls, persistent KV state, an HTTP server, and a coding agent rather than serving as a general GGUF runner. The engine supports Metal on Macs, CUDA including multi-GPU systems, and ROCm on Strix Halo hardware; it also provides SSD streaming for machines with insufficient RAM, distributed tensor and pipeline parallelism, and server-side micro-batching for multi-user inference. The repository includes tooling and data for GGUF, imatrix, quality, and speed testing, and acknowledges code and design contributions from llama.cpp and GGML.