AI product Open source · MIT

nanochat

nanochat is an open-source experimental harness for training and running small language models, developed by Andrej Karpathy. Designed for a single GPU node, it covers tokenization, pretraining, supervised fine-tuning, evaluation, inference, chat-oriented reinforcement learning, and command-line interaction. Model size is controlled primarily through a single transformer-depth setting; the harness derives other hyperparameters, including transformer width, attention heads, learning-rate adjustments, training horizons, and weight decay. It includes GPT-style model and tokenizer implementations, distributed training and gradient accumulation, mixed-precision execution, KV-cache inference, benchmark tasks, sandboxed Python execution, and scripts for CPU, Apple Silicon, and CUDA environments. The repository is distributed under the MIT license.

View repository