» Tag
bpe
1 postsGigatoken: Rust BPE Tokenizer Hits 24.53 GB/s, Up to 989x Faster
Rust-based Gigatoken hits 24.53 GB/s BPE tokenization using SWAR and dual-cursor techniques, vastly outpacing HuggingFace tokenizers and tiktoken.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com