The model reads tokens, not words.

Lesson 1 of 6About 8 minutes5 steps

Before a language model sees your sentence, a tokenizer cuts it into pieces and swaps each piece for a number. Those pieces, tokens, decide what a model can see, what it costs and why it trips over some easy questions.

  • Say what a token is, and what the model actually receives
  • Explain why rare words and names split into pieces
  • Explain why Hindi often costs more tokens than English

You need: Nothing: this is where the series starts.

  1. 1Tokens
  2. 2Embeddings
  3. 3Attention
  4. 4Picking the next token
  5. 5Context and the KV cache
  6. 6Hallucination
0 of 5 steps

  1. Pieces and numbers

    Predict first

    GPT-4o’s tokenizer cuts “Skillcept Online helps you get hired.” into pieces. How many?

    Just show me

    8 pieces: “Skill” “cept” “Online” “helps” “you” “get” “hired” “.”. Not words, not letters: tokens.

    A tokenizer splits text into tokens and looks each one up in a fixed list, its vocabulary. GPT-4o’s and GPT-5’s list, o200k_base, has about 200 thousand entries. The model never sees the letters, only the numbers under each piece. (A dot marks a space that belongs to the token.)

    Skill30852cept953Online6910helps9335you481get717hired37883.13

    8 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    Quick check

    So what does the model actually receive?

    Token numbers. Everything the model does, it does with those numbers; turning them back into text is the tokenizer’s job.

  2. Common words whole, rare words in pieces

    Predict first

    Which of these does GPT-4o’s tokenizer cut into the most pieces?

    Just show me

    Bengaluru: 4 pieces. “unbelievable” takes 3 and “tokenization” 2.

    Tokenizers like this one are built by byte-pair encoding: start from single bytes, then keep merging the pair that appears most often in a huge body of text, until the list is full. Frequent words end up as one token; rarer words, names and new brands stay in pieces. “Skillcept” is “Skill” + “cept”.

    un373bel9880ievable45794

    3 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    token10346ization2860

    2 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    B33eng882al280uru9734

    4 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    Skill30852cept953

    2 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    Quick check

    Why is “Skillcept” cut in two?

    Merges follow how often pairs appear in the training text. A word the tokenizer rarely saw never got its own entry.

  3. Why models miscount letters

    Predict first

    Ask a model how many r’s are in “strawberry”. Why might it get it wrong?

    Just show me

    It sees “strawberry” as “st” + “raw” + “berry”: three numbers, with no letters inside them to count.

    The same happens to numbers. “₹19,999” is “₹” + “19” + “,” + “999”, and “2026” is “202” + “6”. Counting letters or doing digit-by-digit sums means working through pieces that don’t line up with letters or digits, which is why a calculator or a little code beats asking the model to do it in its head.

    st302raw1618berry19772

    3 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    ₹5069919858,119999130

    4 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    2021323621

    2 tokens on GPT-4o’s and GPT-5’s tokenizer (o200k_base)

    Quick check

    You need a model to add up a column of prices exactly. Best move?

    Tools do exact arithmetic; the model’s job is to decide what to compute.

  4. Hindi costs more tokens

    Predict first

    “I want to learn data science.” is 7 tokens on GPT-4’s tokenizer. Its Hindi, “मुझे डेटा साइंस सीखना है।”?

    Just show me

    26 tokens on GPT-4’s tokenizer, against 9 on GPT-4o’s newer one.

    Older tokenizers learned mostly from English text, so Hindi letters are often cut down to raw bytes: GPT-4’s list has 43 entries in Indian scripts, GPT-4o’s has 13,379. Over the 1,012 sentences of the FLORES-200 test set, Hindi takes 4.76 tokens for every English one on GPT-4, and 1.57 on GPT-4o. More tokens means a higher bill and less room in the context window for the same meaning.

    GPT-2 (2019)7.41×
    GPT-44.76×
    GPT-4o, GPT-51.57×

    Hindi’s tokens for every English token, over 1,012 sentences of the same meaning (FLORES-200)

    Count your own text in the token counter

    Quick check

    You build a Hindi support bot. What does a newer tokenizer change?

    Fewer tokens for the same text: a lower bill, a faster reply and more room in the context.

  5. What tokens decide

    Predict first

    An API charges per million tokens. Which costs more to send, a reply in English or the same reply in Hindi?

    Just show me

    Hindi, on most tokenizers: the bill counts tokens, not meaning.

    Tokens are the unit of almost everything: prices are per million input and output tokens; a model’s context window is a number of tokens (Qwen 3’s is 32,768 tokens on its own, from its model card); speed is tokens per second. When you estimate cost, latency or whether a document fits, count tokens with the model’s own tokenizer, not words.

    Quick check

    You want to know if a 40-page policy fits in a model’s context. What do you count?

    Context limits are in tokens, and each model family has its own tokenizer.

In the interview: tokens.

Ask Prism about this lesson

Sources

  1. Sennrich, Haddow and Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL 2016: byte-pair encoding for subwords.
  2. OpenAI, tiktoken, the o200k_base, cl100k_base and GPT-2 encodings, counted here with js-tiktoken.
  3. Petrov and others, “Language Model Tokenizers Introduce Unfairness Between Languages”, NeurIPS 2023.
  4. NLLB Team, FLORES-200, CC BY-SA 4.0: the same 1,012 sentences in 200 languages.

All six lessons