Project Titania

Bonsai Demo is an open-source repository from Prism ML for running the Bonsai and Ternary-Bonsai language models locally. It supports 1-bit and ternary model families in several sizes, using llama.cpp or MLX across macOS Metal, Linux and Windows CUDA, Vulkan, ROCm, and CPU backends. Setup scripts download the models and binaries, while the included server and web interfaces provide local chat; the 27B models additionally support image, screenshot, and PDF input, OpenAI-style tool calls, MCP servers, configurable reasoning effort, and long-context conversations. The repository provides command-line launch scripts, model-family and size selection, and links to the corresponding Hugging Face collections and technical whitepapers.

An Apache-2.0-licensed serving setup for Qwen3.8-27B on a single 24 GB NVIDIA GPU, built around vLLM and exposing an OpenAI-compatible API with optional key authentication. Its preparation pipeline starts from a W4A16 AutoRound model, requantizes the language-model head and embedding matrices to int8, requantizes the MTP components, and applies vLLM patches for the serving stack. The runtime combines int8 tensor-core GEMMs with an fp16 recurrent state, continuous batching, split-KV verification attention, and speculative decoding through either Qwen's MTP path or the optional DFlash2 block drafter; DFlash2 can also draft from the request's cached context for document-reproduction workloads. The repository provides Docker Compose profiles for batch throughput and single-user latency, prebuilt container images, model-download and requantization scripts, benchmark and quality-test tools, and launchers for fast, long-context, and experimental KVarN cache modes. The standard configurations target roughly 64k to 150k tokens of context, while KVarN and alternative int4 KV-cache paths extend the claimed capacity to the 256k-token range with lossy KV quantization. The setup requires a recent NVIDIA driver and a compatible Ampere-or-newer GPU; the repository notes that its benchmark figures were measured on an RTX 3090 subject to a 250 W power limit.

TileLang is a Pythonic domain-specific language and compiler for developing high-performance GPU, CPU, and NPU kernels such as GEMM, quantized matrix multiplication, FlashAttention, and linear attention. Built on TVM, it uses tiled programming primitives for memory transfers, pipelining, matrix operations, parallel elementwise work, layouts, and target-specific code generation; its JIT specializes kernels for input shapes and compile-time arguments on first use. The project supports CUDA, ROCm/HIP, Apple Metal, Huawei Ascend, and experimental LLVM, CuTe DSL, and WebGPU backends, with additional ecosystem adapters for other accelerator platforms. It is distributed as a Python package through PyPI, with source builds and nightly wheels available.

NIGHTRUN is a local LLM runtime written in Rust that boots directly from a USB stick on ordinary x86_64 PCs or from an SD card on a Raspberry Pi 5, without a conventional operating system underneath. Its UEFI application loads a quantized model into RAM, renders a chat terminal through the framebuffer, and runs inference locally and offline without a network stack, host process, or kernel. The installer builds and flashes a bootable image, guides model selection and verified downloads, and supports GGUF inspection and conversion into NIGHTRUN's .nrm container. During boot, the model is streamed into RAM with checksummed sections; storage is then sealed so generation cannot read from disk. The runtime includes custom framebuffer rendering, USB keyboard input, multi-core inference, serial diagnostics, hand-written quantized kernels for x86_64 and Raspberry Pi architectures, and Q8_0, Q4_K, and Q6_K weight formats. It requires Linux, Rust, and approximately 6 GB of free disk space for the model and image.

mini-AGI is an experimental continual-learning, byte-level language model that trains from scratch on a CUDA-capable GPU with at least 8 GB of VRAM. It reads 256 byte values directly rather than using a conventional tokenizer, processes data one chunk at a time, and uses the same forward path for training and generation. Its architecture combines dense prelude blocks with a recurrent block applied up to 24 times, adaptive PonderNet-style halting, and per-application top-8 routing through a dynamically growing and pruning expert pool. Expert weights and Adam moments are stored as files on disk; a working set is paged through RAM and VRAM as needed, allowing the pool size to exceed available VRAM. The project includes Python commands for building corpora, reading files, streaming training, serving a local web interface, and inspecting or replicating continual-learning experiments. The repository describes the current model as toy-level rather than frontier-capable, and its weights are generated in the local weights directory rather than distributed with the source.