I wanted to practice Retrieval-Augmented Generation (RAG) by building something simple: a study tutor for the Canadian citizenship test. The idea was to use the official 68-page Discover Canada guide to answer questions, and eventually add quizzes, flashcards, and maybe even voice interaction.
But does a language model really need the book? It probably knows enough Canadian history already. And why bother with RAG when I could just upload the PDF and ask questions?
I started with the first question.
Does the model need the book?
I asked the same model (GLM 5.3 Flash) the same question, once without the book and once with relevant passages from it:
Who received the Victoria Cross?
Without the book, the model gave a mix of recipients from different countries and periods, including some incorrect or inconsistent details.
With the book, it listed the six recipients mentioned in Discover Canada, along with their details and page references.
The model didn’t necessarily lack the knowledge. But having the book gave it a specific source to answer from, one I could verify.
You can try my RAG demo here.
How does RAG work?
Instead of asking a language model to answer using only what it already knows, we first retrieve relevant information from one or more documents (Retrieval), include it in the prompt (Augmentation), and let the model generate an answer using that context (Generation).
Before asking any questions, we split the PDF into smaller pieces called chunks. Each chunk is converted into an embedding, a numerical representation of its meaning, and stored in a vector database.
I used all-MiniLM-L6-v2, which represents each chunk with 384 numbers.
When a question arrives, we embed it using the same model, find the five closest chunks (top-k retrieval, with k = 5), and pass their text to the language model along with the question.
That’s essentially RAG: retrieve first, generate second.
Why not just give the model the entire book?
For a 68-page PDF, I probably could!
The extracted text is roughly 19,000 tokens, which fits comfortably inside the context window of many modern language models.
But I wanted to experiment with retrieval, reduce the amount of text sent with every question, and have citations pointing to specific pages. I also wanted an index I could reuse for the tutor’s other features.
What is a chunk, and how big should it be?
A chunk is the unit of retrieval. The database doesn’t return individual sentences from a chunk; it returns the whole piece.
There are several ways to split a document:
- Fixed size: Split every N characters or tokens, potentially cutting sentences in half.
- By section: Keep everything under one heading together, even if the section is very long.
- Paragraph windows: Group complete paragraphs into chunks of roughly the same size.
I experimented with the last two approaches.
Initially, I thought splitting by section made the most sense. After all, the book already has a structure. But I ran into two problems.
First, long chunks can contain several topics, making their embeddings less focused. Second, embedding models have input limits.
My embedding model accepts at most 256 tokens. Of the 145 section-based chunks I created, 19 exceeded that limit. Their embeddings represented only the beginning of the text.
For the question “What does Confederation mean?”, the relevant passage ranked fourth using section chunks but first using paragraph windows.
Smaller chunks were easier to retrieve in this case. But making chunks too small introduces another problem: the language model might not get enough context to answer.
A sentence containing “this decision” or “he later became” doesn’t tell us much without the preceding sentences.
So we have a trade-off. Retrieval benefits from focused chunks, while generation needs enough context to produce an answer.
What worked for my PDF?
I ended up using paragraph windows of approximately 800 characters, keeping paragraphs intact and including their chapter and section headings.
This produced 236 chunks, with none exceeding the embedding model’s token limit.
On my small development set of 25 answerable questions, the correct page ranked first for 24 questions using paragraph windows, compared with 22 using section chunks. Not a huge difference, but encouraging.
There was one more challenge: PDFs don’t always have a sensible reading order. Discover Canada contains two columns, captions, sidebars, and diagrams. Some captions were even assigned to the wrong sections during extraction. I had to use their positions on the page to fix that.
It made me realize that chunking isn’t just about choosing a size. It’s also about preserving the structure and meaning of the original document.
My next question is whether better retrieval actually leads to better answers. That’s what I want to explore in the next post.