Meaning, as a list of numbers.

Lesson 2 of 6About 8 minutes5 steps

Inside the model, every token becomes a long list of numbers, an embedding. Texts that mean similar things get similar lists, which is how a search can find “money back” when you typed “refund”.

  • Say what an embedding is and how big one is
  • Compare two texts by cosine similarity
  • Explain how meaning search works

You need: Lesson 1, Tokens.

  1. 1Tokens
  2. 2Embeddings
  3. 3Attention
  4. 4Picking the next token
  5. 5Context and the KV cache
  6. 6Hallucination
0 of 5 steps

  1. Every token, a list of numbers

    Predict first

    How many numbers does Qwen 3 (0.6B) use to describe each token?

    Just show me

    1,024 numbers, from its config file (hidden size). 1,51,936 is the size of its vocabulary.

    The model keeps a table with one row per token in its vocabulary: 1,51,936 rows of 1,024 numbers, 15,55,82,464 numbers in all. That’s a big share of a small model: its model card puts it at 0.6B parameters, 0.44B of them outside this table. The numbers start random and are learned in training, so tokens used alike end up with similar rows.

    Quick check

    Where do a token’s embedding numbers come from?

    They start random and training nudges them until they’re useful for predicting text.

  2. Close in meaning, close in numbers

    Predict first

    Which pair does an embedding model rate closest?

    Just show me

    “How do I get a refund?” and “Can I get my money back?”: 0.87, with no word in common. The course fee scores 0.54, the cricket match 0.43.

    A sentence embedding model turns a whole text into one list: BAAI’s bge-small-en-v1.5, the one our RAG chunking tool uses, gives 384 numbers per text. Similar meaning, similar lists, so their similarity is high even when they share no words.

    Each phrase against “How do I get a refund?”, by bge-small-en-v1.5 (1 is the same meaning)

    Can I get my money back?0.87
    I was charged twice0.67
    What is the course fee?0.54
    Who won the cricket match?0.43
    A hot cup of masala chai0.37

    Quick check

    A keyword search for “refund” misses “money back”. A search by embeddings?

    Embeddings compare meaning, not spelling: that’s the point of meaning search.

  3. A map of meaning

    Predict first

    Draw 16 short phrases by their embeddings. What do you expect?

    Just show me

    Groups by topic: refunds and fees on one side, cricket and food on the other.

    The map squeezes each phrase’s 384 numbers down to two (by principal components) so it can be drawn; real searches use all 384. Tap a phrase to see its three nearest neighbours by the full numbers.

    RefundsFees and payingCricketFood
    Nearest to “How do I get a refund?”
    1. Can I get my money back? (0.87)
    2. I was charged twice (0.67)
    3. Cancel my registration (0.66)

    Quick check

    “Is there an EMI option?” sits between the groups. Why might that be?

    Two numbers can’t hold everything 384 can, and “EMI” is a word none of the other phrases use.

  4. How closeness is measured

    Predict first

    Two embeddings point the same way, but one is twice as long. Their cosine similarity?

    Just show me

    1: cosine similarity only looks at direction, not length.

    Cosine similarity is the cosine of the angle between two lists of numbers: multiply them position by position, add it up, and divide by both lengths. 1 means the same direction, 0 means unrelated. Embeddings are usually scaled to length 1 first (we did for these), and then the plain product is the cosine.

    Cosine similarity, symbol by symbol

    Quick check

    A score of 0.87 against 0.43 tells you what?

    Higher cosine, closer meaning, by this model’s reckoning.

  5. Where you’ll use them

    Predict first

    An assistant answers from your company’s policies. Where do embeddings come in?

    Just show me

    Finding the passages: retrieval, the first half of RAG.

    Embeddings power meaning search and RAG, recommendations (“people who read this also read”), grouping similar tickets, and spotting duplicates. Each use is the same move: turn things into lists of numbers, then compare.

    Search a real policy by meaning in RAG chunking

    Quick check

    Which task needs embeddings least?

    Arithmetic needs exact numbers, not meaning.

In the interview: embeddings.

Ask Prism about this lesson

Sources

  1. Mikolov and others, “Efficient Estimation of Word Representations in Vector Space”, 2013: word vectors.
  2. BAAI, bge-small-en-v1.5, MIT: the sentence embedding model, run here at full precision on 1 Nov 2026.
  3. Qwen, Qwen 3 (0.6B), Apache 2.0: its config file and model card for the embedding table’s size.

Back: lesson 1, Tokens