Token counter, in Indian languages.
Paste any text, or pick a sample, and see how a model’s tokenizer cuts it into the tokens it reads and charges for. The same sentence takes more tokens in Hindi or Tamil than in English. Here’s how many more.
Guess first
The same sentences in English and in Hindi. On GPT-4o and GPT-5’s tokenizer, how many times English’s tokens does Hindi take?
It’s 1.6×: 157 tokens of Hindi for every 100 of English, across 1,012 sentences. On GPT-4’s older tokenizer it was 4.8×, and on GPT-2’s 7.4×. On Gemma 4’s and Sarvam’s, 1.3×.
Count your own textPick a sample, or type your own
Tokenizer
34tokenson GPT-4o · GPT-5
Letters 77Words 22Tokens a word 1.5
One colour, one tokenA small number: a letter cut between that many tokens, drawn as one piece
Context window
34 of 128K tokens: 0.03% of the window. It would fit about 3,855 times.
The same text, every tokenizer
This sample, cut by each tokenizer in advance.
Samples: sentence 680 of FLORES-200’s test set, the same meaning in every language (Meta AI, CC BY-SA 4.0). Hinglish and code are ours.
The finding
Same meaning, more tokens.
Each language’s 1,012 FLORES-200 sentences, the same meaning in every language, counted by each tokenizer against English’s. Pick a tokenizer.
How many times English’s tokens, on GPT-4o · GPT-5’s tokenizer. Counted on 1 Nov 2026 by the tokenizers themselves (js-tiktoken 1.0.21, Hugging Face’s tokenizers.js 0.2.0), each sentence alone. Our GPT-4 figures are close to Petrov and others’ (NeurIPS 2023: Hindi 4.79×, Tamil 7.65×, on two of FLORES-200’s sets together).
Pieces in Indian scripts, in each tokenizer’s list
- 1GPT-2
- 43GPT-4
- 253Qwen 3
- 13,379GPT-4o · GPT-5
- 34,334Gemma 4 · Gemini
- 51,474Sarvam
Each list’s pieces holding a letter of Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada or Malayalam (Urdu’s script is left out), counted on 1 Nov 2026.
What a token is, and why it costs more in Hindi.
A token is a piece of text
Before a model reads anything, its tokenizer cuts the text into pieces from a fixed list it learned: 50,257 pieces for GPT-2, about 2,00,000 for GPT-4o and GPT-5, 2,62,144 for Gemma 4 and Sarvam. A common word is one piece; a rare one is cut into several.
Why Indian languages cost more
The lists were learned mostly from English. GPT-2’s holds just one piece in an Indian script, so it can cut each Hindi character, 3 bytes in UTF-8, into as many as 3 tokens. GPT-4o’s holds 13,379, which is why Hindi fell from 7.4× English’s tokens to 1.6×.
What it changes
Prices, speed and the context window are all counted in tokens. The same answer in Tamil takes 2× English’s tokens on GPT-5, so it costs about twice as much, takes about twice as long to write and fills the window twice as fast.
And Claude? Anthropic hasn’t published Claude’s tokenizer: it counts tokens through its API, and says Claude 4.7 and later make about 30% more tokens than earlier Claude models for the same text.
Sources
- OpenAI, tiktoken: which encoding each model uses (read 1 Nov 2026; every GPT-5 model uses o200k_base).
- js-tiktoken 1.0.21: OpenAI’s encodings in JavaScript, which count GPT’s tokens here (MIT licence).
- Google, Gen AI SDK for Python: it counts Gemini 3.1 and 3.5 with Gemma 4’s tokenizer (read 1 Nov 2026).
- google/gemma-4-E4B-it on Hugging Face: Gemma 4’s tokenizer, at revision ee0ef60 (Apache 2.0).
- Qwen/Qwen3-0.6B on Hugging Face: Qwen 3’s tokenizer, at revision c1899de (Apache 2.0).
- sarvamai/sarvam-30b on Hugging Face: Sarvam’s tokenizer, at revision 071ae95 (Apache 2.0).
- Hugging Face, tokenizers.js 0.2.0: runs the open models’ tokenizers here (Apache 2.0).
- NLLB Team, Meta AI: FLORES-200, the same 1,012 test sentences in 200 languages (CC BY-SA 4.0).
- Petrov and others, “Language Model Tokenizers Introduce Unfairness Between Languages”, NeurIPS 2023.
- Anthropic, Token counting (read 1 Nov 2026).