Gigatoken

What people use it for

An open-source tokenizer for language modeling that processes text at gigabytes per second. It is designed for large training corpora and supports common tokenizers on x86 and ARM, including compatibility with Hugging Face and TikToken code.

ManuAGI - AutoGPT Tutorials