Gigatoken: Rust BPE Tokenizer Hits 24.53 GB/s, Up to 989x Faster
Rust-based Gigatoken hits 24.53 GB/s BPE tokenization using SWAR and dual-cursor techniques, vastly outpacing HuggingFace tokenizers and tiktoken.
Gigatoken, a new Rust-based BPE tokenizer, reaches 24.53 GB/s on a 144-core AMD EPYC 9565, compared to 24.8 MB/s for HuggingFace tokenizers and 36.0 MB/s for tiktoken on the same machine. Both baselines are multithreaded Rust implementations, so the gap comes from how the work is structured rather than the language used.
Instead of delegating pretokenization to a regex engine, Gigatoken implements it directly. A 256-byte lookup table classifies the first byte in O(1), SWAR loads 8 bytes as a u64 to check letter properties with branchless arithmetic, and two independent cursors starting from a safe split point let the out-of-order CPU overlap their instruction streams. Repeated words are served from a pretoken cache instead of being re-run through BPE, though the author notes the cache's rapid growth and long-tailed distributions make this the hardest part to get right.
On the 11.9 GB OpenWebText corpus with GPT-2, measured throughput was 24.53 GB/s on the EPYC 9565, 8.79 GB/s on an Apple M4 Max, and 6.27 GB/s on a Ryzen 7 9800X3D. Note that Gigatoken tokenizes the full file and finds its own boundaries, while the comparison tools were tested on presplit partial inputs.
The speedup matters for pretraining pipelines that retokenize a corpus after every mixture or filter change, and for serving stacks like vLLM and SGLang, where tokenization runs before token-chunk hashing and KV-cache lookup, directly affecting time-to-first-token.