Why AI Models Perform Worse Outside English
Why do LLMs underperform outside English? Training data, tokenizer inefficiency, and instruction-tuning gaps explain the cost and accuracy divide.
The performance gap between English and other languages in large language models isn't about linguistic difficulty — it's about training data share, tokenizer efficiency, and how far instruction-tuning extended. Measured accuracy drops of up to 29% have been found between English and non-English enterprise retrieval tasks on identical documents, and for languages absent from training data entirely, model accuracy falls to roughly chance level (around 38% on three-way classification).
Speaker population doesn't predict this outcome: Bengali has far more speakers than German but a fraction of its digital footprint, because what gets scraped into training corpora tracks wealth, internet penetration, and colonial history rather than population. The less obvious and more expensive factor is the 'tokenizer tax' — non-English text, especially non-Latin scripts and agglutinative languages, can require 3-4x more tokens to express the same meaning. This simultaneously raises per-request cost, shrinks effective context windows, and degrades output quality since the model processes text at a coarser resolution.
For engineers, the takeaway is that most multilingual underperformance stems from correctable engineering decisions — data distribution and tokenizer design — not from properties of the languages themselves. Recent model families have adopted larger vocabularies to ease tokenizer fragmentation, but this mitigates rather than solves the problem, since vocabulary space remains finite and still skews toward the training distribution.