The Tokenizer Tax: Why non-English AI users pay more for tokens

A study by the Coalition for Language Equity in AI Research found that AI models can require far more tokens for some languages than for others, creating what researchers call a hidden tokenizer tax. The effect can make AI more expensive for people who use less widely spoken languages, especially when those languages have a smaller online presence. Tokens are the units of text or data that AI models process.

How token counts vary by language

Tokenizers break raw text, images, or audio into tokens. In one example, an English sentence was split into recognizable words, while the same sentence in Malayalam was broken into multiple fragments per word without respect for multi-byte characters. Even within the 20 most popular languages on the internet, the research found a gap in token usage: the most expensive of those languages used 1.5 to 5 times more tokens than the cheapest. When the set expanded to 200 languages, the token cost gap in one AI model grew by 17 times. The study said a startup in Bengaluru may pay a surcharge on every prompt and completion while its competitor in Austin does not.

Cost is not the only penalty

Higher token counts also translate into higher latency and reduced effective context capacity, the study said. Non-English languages leave less room for documents or conversation history and require more work to generate an answer, according to Pratyush Choudhury, co-founder of Activate AI. He compared the problem to a tutor retaining a student's earlier mistakes and explanations: if that history consumes the context window much faster in one language, the disadvantage reaches into service quality. The study said these penalties hit hardest where compute and connectivity are already limited, and that tokenizer inequity is a constraint on who can participate in the AI economy.

Script, data, and design all matter

The rarity of a language is not the only factor in token use; script plays a big role. Asturian, which has relatively few speakers and no Common Crawl data, benefits from the Latin-script advantage and representation from related Romance languages such as Italian. Languages written in scripts tokenizers rarely saw, including Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, were at the greatest disadvantage consistently across nearly all models in the analysis. Dheemanth Reddy, CEO of Maya Research, said many tokenizers remain disproportionately optimized for English and other high-resource languages, and that the underlying issue is the lack of high-quality digital data. One way to improve language processing is to improve underlying data. Raoul Nanavati, co-founder and CEO of Navana.ai, pointed to RESPIN, a collaborative initiative with the Indian Institute of Science funded by the Gates Foundation, which collected text from native speakers of 38 dialects along with read speech.

Fixing the gap

Better training data alone does not fix the tokenizer, Reddy warned. Token prices are falling quickly, so the direct cost impact should become less severe over time, and better multilingual and synthetic data should improve model quality. But actually fixing the tokenization gap will still require labs to deliberately design future tokenizers to handle these languages and scripts better. Choudhury said language efficiency should be a product requirement alongside accuracy, speed, and reliability. He said perfect equality may be unrealistic, but large persistent disparities should require an explanation and an improvement plan, and he asked every major lab to publish a language-by-language scorecard showing what it costs to complete comparable tasks successfully and to commit to measurable improvements for its worst-served languages.

Sources: Rest Of World · Techmeme