Soup CLI

Soup is an open-source Python CLI for fine-tuning and post-training large language models from a YAML configuration and a single training command. It supports supervised fine-tuning, LoRA, QLoRA, preference optimization, quantization, automatic batch-size and GPU detection, and local training without a cloud service. Its optional layer-streaming mode keeps the frozen base model out of VRAM and feeds it to the GPU one decoder layer at a time, reducing memory use for low-VRAM systems. The project describes training an 8B model on a 4 GB laptop GPU with this mode; layer streaming is marked beta.

An open-source Python package of shared utilities for Unsloth, supporting large-language-model training and fine-tuning workflows such as full fine-tuning, LoRA, quantized training, and reinforcement learning. The associated Unsloth project provides beginner-oriented notebooks, supports models including gpt-oss, Gemma, Qwen, Llama, and Mistral, and can export fine-tuned models to GGUF, Ollama, vLLM, or Hugging Face.

Unsloth is an open-source local desktop and web tool for running, training, fine-tuning, and deploying language, diffusion, embedding, audio, and vision models. It supports Windows, macOS, Linux, WSL, multi-GPU setups, NVIDIA, AMD, and Intel GPUs, CPUs, and the Vulkan backend. The project is distributed as Unsloth Desktop, Unsloth Studio, and Unsloth Core. It can run local models, provide an OpenAI-compatible API, connect models to agents and MCP tools, perform search and RAG, and offer remote or LAN access. Its training functions cover reinforcement learning, LoRA, QLoRA, full fine-tuning, pretraining, GRPO, DPO, and FP8, with export formats including GGUF, NVFP4, and FP8. The repository claims that its fine-tuning workflows use two times less training time and 70% less VRAM.

Bonsai Demo is an open-source repository from Prism ML for running the Bonsai and Ternary-Bonsai language models locally. It supports 1-bit and ternary model families in several sizes, using llama.cpp or MLX across macOS Metal, Linux and Windows CUDA, Vulkan, ROCm, and CPU backends. Setup scripts download the models and binaries, while the included server and web interfaces provide local chat; the 27B models additionally support image, screenshot, and PDF input, OpenAI-style tool calls, MCP servers, configurable reasoning effort, and long-context conversations. The repository provides command-line launch scripts, model-family and size selection, and links to the corresponding Hugging Face collections and technical whitepapers.

llama.cpp is a C/C++ implementation for running large language and vision-language model inference locally or in the cloud, including on CPUs, GPUs, laptops, Apple silicon devices, and small computers such as Raspberry Pis. The project is built on the ggml library and uses integer quantization from 1.5-bit through 8-bit to reduce memory use and accelerate inference, with CPU, CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and other hardware backends. It supports hybrid CPU-GPU inference for models larger than available VRAM and includes command-line tools, a built-in web interface, and an OpenAI-compatible REST API server. The project is distributed as source code, Docker images, and pre-built binaries, with installation instructions at llama.app.