Cut the document, then ask it.
An AI assistant that answers from documents first finds the right pieces of them: retrieval-augmented generation, or RAG. How you cut the document decides what it can find. Here’s our own Refund policy, cut the way you choose, and four questions people really ask.
Guess first
“I paid by card EMI. What do I get back?” Clause 2.3 answers it. Cut the policy into chunks of 200 characters, 1,600 characters, or its own clauses. Which way puts the whole answer in the first chunk found?
How much of clause 2.3 the first chunk found holds. 1,600-character chunks and its own clauses both put all of clause 2.3 first, but a 1,600-character chunk also hands the model 17% of the policy to read. 200-character chunks hold only 18% of it: the rule is cut in pieces.
Try the settingsCut it
Chunk size
Overlap
Ask it
No cut wins every question
The share of each question’s answering clause in the first chunk found by meaning.
| Cut | Card EMI | Days to bank | Fell ill | Paid twice |
|---|---|---|---|---|
| Fixed, 200 characters48 chunks | – | – | – | 83% |
| Fixed, 800 characters12 chunks | all of the answer | – | 53% | 83% |
| By sentence, 40029 chunks | 44% | all of the answer | 46% | all of the answer |
| By sentence, 1,6007 chunks | all of the answer | all of the answer | all of the answer | – |
| By its own clauses29 chunks | all of the answer | all of the answer | – | all of the answer |
That’s why a real assistant’s chunking is tested on a set of its own questions before it ships, and often searched by meaning and keywords together.
The document: our Refund policy (version 4.0), as plain text, 9,420 characters. By meaning: each chunk and question as 384 numbers from BAAI’s bge-small-en-v1.5 (MIT), run at full precision on 1 Nov 2026, and the chunks closest to the question by cosine similarity. By keywords: BM25, computed here in your browser.
Why the cut decides the answer.
Retrieve, then generate
The model answers from the few chunks a search hands it, not from the whole document. If the answer isn’t in those chunks, the best model can’t give it, and may make one up.
Too small, too big
Small chunks cut a rule in pieces, so no single chunk holds the answer. Big chunks hold it whole but bury it among other rules, which blurs the match and fills the model’s context. Overlap repeats the edges so a sentence cut at one edge is whole in the next chunk.
Meaning and keywords
A search by meaning compares embeddings, so “charged twice” finds “paid twice”. Keywords (BM25) find exact words like “EMI” that meaning can miss. Many systems run both and merge the results.
Sources
- Lewis and others, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 2020: the RAG paper.
- BAAI/bge-small-en-v1.5 (MIT) and Xiao and others, “C-Pack: Packaged Resources To Advance General Chinese Embedding”, 2023: the embedding model; its ONNX copy, Xenova/bge-small-en-v1.5, at revision ea104da.
- Robertson and Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, 2009: the keyword search (k1 = 1.2, b = 0.75).