Unigram: A Rust Codec Turning Bytes Into Single-Token Words for LLMs

Unigram is a bijective Rust codec that maps arbitrary bytes — nonces, correlation IDs, and similar identifiers — into readable words drawn from a purpose-built 256-word alphabet. Its central claim: every word in that alphabet tokenizes to exactly one token, bare and space-prefixed, across GPT-2, GPT-3, GPT-3.5/4, GPT-4o, o200k, Llama, and Claude tokenizers. That makes the token cost of an N-byte value a flat N, with no worst-case variance — unlike hex, base64url, or base58, whose costs shift with the specific value and whose worst case is not known in advance.

The crate ships two parsing modes: a strict parse that accepts exactly one canonical spelling, and a tolerant recover that forgives casing, separator, and line-wrap differences introduced when text round-trips through a model. A CheckedUnigramId variant adds a single CRC-8 check word so a mutated value can't silently pass as valid.

The alphabet itself is engineered differently from existing wordlists like BIP39, Diceware, or what3words, none of which were built with tokenizer boundaries in mind — only 349 of BIP39's 2048 words survive a single-token filter across all tested families. For engineers building systems where identifiers pass through LLM prompts, logs, or error messages, unigram offers a way to make token budgets for those values fixed and predictable rather than probabilistic.