Quantization recipe for fitting a 27-billion-parameter model on a 24 GB GPU

Problem: A 27-billion-parameter model may load on a 24 GB card but leave insufficient memory for useful batched inference.

For: Local-inference users and developers trying to run large language models on single gaming GPUs.

Examples

• Qwen 3.8 27B: the model being fitted to the single 24 GB card.

Related ideas

• GPU-free local inference engine for a large language model
• Local execution runtime for large Qwen models on iPhones
• SSD-streamed large-model inference for memory-limited Macs
• Offline command-line toolkit for fine-tuning large models on small GPUs
• Production-ready inference hosting for open AI models