Gigatoken: ~1000x faster tokenization than HuggingFace tokenizers
Gigatoken delivers ~1000x faster tokenization than HuggingFace tokenizers, offering GB/s throughput as a Rust-based drop-in replacement.
Gigatoken is a new Rust-based tokenization library positioned as a drop-in replacement for HuggingFace Tokenizers and tiktoken. It supports widely used tokenizer schemes (GPT-2, Llama, Qwen, DeepSeek, GLM, and more) and claims GB/s-scale throughput on modern CPUs. Published benchmarks show up to 989x speedup over HF tokenizers on a 144-core AMD EPYC 9565 and up to 1353x on an Apple M4 Max.
The library offers two usage paths: a compatibility mode that requires minimal changes to existing HF or tiktoken code, and a native Gigatoken API that lets Rust read files directly for maximum parallelism and lowest overhead. Much of the speedup comes from SIMD-optimized pretokenization — work usually delegated to regex engines — plus an aggressive caching strategy for repeated tokens despite the long-tailed distribution of pretokens.
Tokenization is often an overlooked bottleneck in LLM data pipelines. A speedup of this magnitude could turn multi-hour corpus preprocessing (e.g., tokenizing datasets the size of Common Crawl) into a matter of hours rather than days, meaningfully cutting the cost of data prep ahead of large-scale pretraining.