Retrieval-Augmented Generation¶

The bar: you can stand up a RAG system that answers questions from your own documents — grounded in real sources, fast enough to ship, and measured well enough that you know when it's wrong.

A language model only knows what it was trained on. The moment you need it to answer questions about your handbook, your codebase, or your support tickets, you have two choices: retrain the model (expensive, slow, brittle) or hand it the right context at question time. Retrieval-Augmented Generation is the second path, and it's the one that ships.

RAG is the difference between a chatbot that hallucinates confident nonsense and a system that cites the paragraph it pulled its answer from. Get it right and you can point a model at any body of knowledge and trust the output. This module walks the whole pipeline — ingest, store, retrieve, generate, and prove it works.

19.1 RAG Architecture¶

Objective: design a retriever-generator system end to end.

Every RAG system is the same shape: take raw documents, break them into pieces, turn each piece into a vector, store those vectors, and at query time pull back the pieces most relevant to the question and paste them into the prompt. The model then answers from that context instead of from memory. That last step — context injection — is what keeps it honest.

The part that quietly decides whether your whole system works is chunking. Split too big and you bury the relevant sentence in noise the retriever can't score well; split too small and you sever the context a paragraph needed to make sense. Chunk on natural boundaries (headings, paragraphs), keep a little overlap between chunks so a thought isn't guillotined mid-sentence, and store metadata (source, section, date) alongside every chunk so you can filter and cite later.

Embedding generation is where meaning becomes math: a model maps each chunk to a point in high-dimensional space where similar meanings sit close together. The common rookie mistake is embedding your documents with one model and your queries with another — the vectors won't live in the same space and retrieval turns to garbage. Pick one embedding model and use it on both sides.

Drill:

• Build a document ingestion pipeline that reads raw files and normalizes them to clean text
• Implement a chunking strategy with overlap; compare fixed-size vs. structure-aware splits
• Generate embeddings for each chunk with a single consistent model
• Write the retrieval step: embed the query, pull top-k chunks, inject them into the prompt as context

19.2 Vector Databases¶

Objective: store vector embeddings and query them by similarity.

Once you have embeddings you need somewhere to put them that can answer "which vectors are closest to this one" fast. For a production app already on Postgres, pgvector is the pragmatic winner — it's a Postgres extension, so your vectors live in the same database as the rest of your data, inside the same backups and the same transactions. ChromaDB is a lightweight option for local development and prototyping, and Pinecone is a managed service when you need to scale past what a single Postgres box handles comfortably.

The reason you don't just brute-force compare against every vector is indexing. An IVF index buckets vectors into clusters and only searches the nearest buckets; HNSW builds a navigable graph that's faster and more accurate but uses more memory to build and hold. Both are approximate — they trade a sliver of recall for a massive speedup. Know that trade-off exists so you're not surprised when the "closest" result occasionally isn't.

Metadata filtering is the feature people forget until they need it. Storing source, author, date, or tenant_id next to each vector lets you say "search only this user's documents" or "only docs from this year" before similarity scoring. Without it, a multi-tenant RAG system will happily leak one customer's data into another's answers.

Drill:

• Enable pgvector on a Postgres database and create a table with a vector column
• Insert embeddings and run a nearest-neighbor query with the <-> distance operator
• Build the same collection in ChromaDB for local iteration
• Add an HNSW index and compare query latency against an unindexed scan
• Attach metadata to vectors and filter on it inside a similarity query

19.3 Advanced Retrieval¶

Objective: raise retrieval accuracy past naive top-k.

Pure semantic search has a blind spot: it's great at meaning, bad at exact tokens. Ask about error code E4021 and embeddings may shrug because that string carries no semantic weight. Hybrid search fixes this by combining keyword search (which nails exact matches) with semantic search (which catches paraphrases) and merging the two rankings. It's the single highest-leverage upgrade over a basic retriever.

Reranking is the second pass. Your first retrieval is fast and approximate, so pull more candidates than you need — say the top 20 — then run a cross-encoder over each query-document pair to score them precisely and keep the best 4. Cross-encoders are too slow to run over your whole corpus, which is exactly why you use them only on the shortlist.

The other two levers work on the query itself. Query expansion rewrites or enriches the question before searching — HyDE (Hypothetical Document Embeddings) generates a fake ideal answer and searches with that, on the theory that answers embed closer to answers than questions do. Multi-vector retrieval stores several embeddings per document — a small chunk for precise matching plus a parent or summary embedding for context — so you match on the sharp detail but return the fuller passage.

Drill:

• Implement hybrid search: combine keyword and semantic results and merge the rankings
• Add a cross-encoder reranker over an over-fetched candidate set
• Wire up query rewriting and a HyDE-style expansion step
• Build multi-vector retrieval with parent/summary embeddings pointing back to full passages

19.4 Evaluation¶

Objective: measure RAG quality instead of eyeballing it.

You cannot improve what you don't measure, and "it looked good in the demo" is not measurement. RAG splits cleanly into two things to evaluate: did retrieval pull the right chunks, and did generation use them faithfully.

For retrieval, borrow the information-retrieval metrics. Precision@k asks what fraction of the top-k results were actually relevant; Recall@k asks what fraction of all relevant chunks you managed to surface; MRR (Mean Reciprocal Rank) rewards putting the right answer near the top. These require a labeled set — questions paired with the chunks that should answer them — which is why building an evaluation dataset is the unglamorous work that makes everything else honest.

For generation, the two questions are faithfulness (did the answer stick to the retrieved context, or did the model invent things?) and relevance (did it actually answer what was asked?). Faithfulness is the metric that catches hallucination, and it's the whole point of RAG. Tools like LangSmith let you run these evaluations over a dataset automatically and track whether a change to your chunking or reranker helped or quietly regressed you.

Drill:

• Build an evaluation dataset of questions paired with their ground-truth source chunks
• Compute Precision@k, Recall@k, and MRR on your retriever
• Score generated answers for faithfulness and relevance
• Run an automated eval pass with LangSmith and compare two retriever configurations

Hands-On Lab: Document Q&A¶

Objective: build a production RAG system that answers questions over a real document set.