Bonsai Demo

MiniMind is an open-source tiny large-language-model project and end-to-end training tutorial developed by Jingyao Gong. Its main dense model is approximately 64M parameters, and the repository provides the model architecture, tokenizer, datasets, inference code, and training pipeline. The pipeline covers pretraining, supervised fine-tuning, hand-written LoRA, DPO, PPO, GRPO, CISPO, model distillation, tool calling, adaptive thinking, and agentic reinforcement learning. Core algorithms are implemented directly with native PyTorch rather than relying on high-level abstractions from third-party training libraries. Its agentic reinforcement-learning path performs multi-turn rollouts, executes generated tool calls, appends tool observations to the context, calculates trajectory-level rewards, and updates the policy; rollout can use local PyTorch generation or an SGLang server. MiniMind supports dense and mixture-of-experts variants, single- and multi-GPU training, YaRN-based RoPE length extrapolation, Transformers-format models, and inference through llama.cpp, vLLM, or Ollama. The repository also includes a Streamlit chat interface and a lightweight OpenAI-compatible API server with tool-call and reasoning fields. It is released under the Apache License 2.0.

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, including on CPUs, GPUs, laptops, Apple silicon devices, and small computers such as Raspberry Pis. The project is built on the ggml library and uses integer quantization from 1.5-bit through 8-bit to reduce memory use and accelerate inference, with CPU, CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and other hardware backends. It supports hybrid CPU-GPU inference for models larger than available VRAM and includes command-line tools, a built-in web interface, and an OpenAI-compatible REST API server. The project is distributed as source code, Docker images, and pre-built binaries, with installation instructions at llama.app.

oMLX is an open-source LLM inference server for Apple Silicon Macs, managed through a macOS menu-bar application. It serves text, vision, OCR, embedding, and reranking models through compatible API endpoints, with model discovery and management, monitoring, continuous batching, and caching. Its caching system keeps KV cache in a hot in-memory tier and a cold SSD tier, allowing cached context to remain reusable across requests even when the conversation context changes. The project can be installed as a macOS app, through Homebrew, or from source, and includes a CLI for controlling the app-managed server. The repository states that it requires macOS 15 or later, Python 3.11–3.13, and Apple Silicon hardware.

kimi-k3-mlx is an MLX port for Apple Silicon of Moonshot’s Kimi K3 native-multimodal mixture-of-experts model. It provides a stock `mlx-lm`-compatible text model definition, a 3D video-capable MoonViT vision tower with a patch-merger projector, and an `mlx-vlm`-shaped wrapper that joins the language and vision components. The model has a 1-million-token context and uses routed and shared experts, Kimi Delta Attention, gated multi-head latent attention, Attention Residuals, latent-dimension experts, and the SiTU-GLU activation; routed experts are distributed in MXFP4 while other weights use bf16. The multimodal processor expands each image placeholder into a feature block rather than performing a same-length one-token-per-image scatter. The repository includes a streaming converter because standard `mlx-lm` conversion would materialize the multi-terabyte bf16 model, along with REAP expert-pruning scripts and expert-overlap analysis using Chinese, English, code, or mixed calibration data. It also contains vision-parity tests against Moonshot’s reference implementation. The repository states that the full model cannot run on a single Mac: even its smallest deployment tier requires hundreds of gigabytes of memory and exceeds the capacity of the largest Apple Silicon machine.

Needle 2 is an open-source, 45-million-parameter foundation model for tool calling, device use, and structured extraction on small devices. Its weights are compressed into a single 14 MB binary and integrated with its inference engine; the repository reports that a full session uses about 28 MB of RAM and performs inference without network access after setup. The model converts text and tool descriptions into schema-constrained JSON calls using a byte-level grammar, attaches a learned confidence score for threshold-based escalation, and can retrieve the top five tools from a larger catalogue. It uses a 256-token sliding window with tools pinned as key-value sinks to bound memory, and its Simple Attention Network combines a Hadamard MLP, grouped-query attention, engram key-value memory, and multi-lane hyper-connections. The Python package supports inference, LoRA fine-tuning, and checkpoint export; the inference engine is fetched from Hugging Face and cached, while the training stack is an optional installation.