AI product Open source
DeepSeek V3 JAX is an educational implementation of the DeepSeek V3 architecture in about 190 lines of pure JAX. It uses plain dictionaries of arrays rather than Flax, Equinox, or PyTorch, and includes Multi-head Latent Attention with a compressed per-token cache, decoupled RoPE, DeepSeekMoE with one shared expert and routed top-k experts, auxiliary-loss-free expert balancing, sequence-wise balance loss, and multi-token prediction.
The repository includes training, cached generation, data preparation, tests, plotting, and checkpoint utilities. Its reference configuration was pretrained from scratch on public-domain detective fiction; the project reports a small model with 36.2 million parameters, while noting that this is an architecture demonstration rather than a strong language model. The implementation uses dense MoE computation and omits features such as FP8 training, DualPipe, node-limited routing, YaRN context extension, post-training, and sparse expert dispatch. The code is MIT-licensed; the training text is public domain in the United States.
1 use taken from transcripts — each links to the moment in the video.
A small educational implementation of DeepSeek v3's architecture in pure JAX, including latent attention, mixture-of-experts routing, load balancing, and multi-token prediction, with parameters stored as plain arrays.
1 in the library.