Retrieval-Augmented Generation: RAG Concepts and Boundaries

Concept

Retrieval-Augmented Generation (RAG) is a system pattern that combines information retrieval with language-model generation. It helps an application answer from private, changing, or controlled sources such as company policies, product documentation, course material, or research documents. Instead of retraining a model whenever a policy changes, the system retrieves relevant evidence at query time and supplies it as context.

RAG is not a framework, an API, or a promise of truth. It is a design that makes source-grounded answers more possible when retrieval, source quality, prompting, and evaluation all work together.

The Pipeline

INDEXING
documents → chunks → embeddings/representations → indexed chunks + metadata

QUERY TIME
question → query representation → retrieve relevant chunks
         → place evidence in model context → generate response

During preparation, documents are split into chunks so a system can retrieve focused passages rather than an entire manual. Chunks are represented with embeddings and stored with metadata such as source, date, access permissions, or section title. At query time, the question is represented similarly, and the system retrieves several candidate chunks, often called top-K results.

Retrieval and Generation Have Different Jobs

The retriever tries to find evidence useful for the question. The generator turns supplied evidence and the request into a readable answer. A capable LLM cannot reliably compensate for consistently poor retrieval. Conversely, a strong retriever does not guarantee that the generator will read, interpret, or cite its evidence correctly.

For a policy Q&A assistant, retrieval may find the current travel-expense policy, but the model could still omit an exception or combine it incorrectly with a general rule. Evaluation must therefore separate retrieval quality from answer quality.

Chunking and Similarity

Chunks that are too small can separate a rule from its exception. Chunks that are too large can be unfocused, consume context, and make relevant details harder to use. There is no universal chunk size: document structure, question style, model context, and the consequences of missing context all matter.

Semantic similarity helps retrieve wording with related learned meaning, as in Lesson 2. It does not guarantee relevance. A question about password reset may retrieve a document about account security that sounds related but lacks the requested steps. Metadata filters, source freshness, ranking, and human review can all matter.

Grounding and Citations

RAG systems may retain source metadata so a response can show supporting documents or passages. This improves auditability and lets a user inspect evidence. A citation is not proof that the claim is supported: the cited chunk may be irrelevant, outdated, or misread. Good evaluation checks claim-to-evidence support, not merely whether a citation appears.

Where RAG Fails

RAG can fail when the needed source was never indexed, wrong chunks are retrieved, documents are stale or incorrect, the query is ambiguous, sources conflict, or the model ignores supplied evidence. It can also produce an answer when the documents do not support one. RAG reduces some knowledge-grounding problems; it does not eliminate hallucinations.

When to Use It

RAG is useful for internal knowledge assistants, policy Q&A, technical-documentation help, controlled research assistants, and product-manual support. It may be unnecessary for summarizing text already supplied, exact calculations, deterministic lookups, small static rule sets, or a simple classifier. Add it because retrieval solves a real evidence problem, not because an application uses an LLM.

Best Practices

Define which sources are authoritative, preserve useful metadata, evaluate retrieval separately from generation, and make unsupported answers safe to decline or escalate. Test common, difficult, ambiguous, and unanswerable questions. Keep citations inspectable when evidence matters.

Groundedness, Correctness, and Citations Are Different Checks

An answer is grounded when its claims follow from the evidence supplied to the model. An answer is correct when it matches the real, current answer to the user's question. The two can differ. A model can faithfully summarize an outdated policy, making the response grounded in its context but wrong for the current policy. It can also state a true fact without support in the supplied context, making it correct by accident but not grounded enough for an auditable policy assistant.

A citation only identifies evidence a reader can inspect. It does not prove that the cited passage supports the claim, that the source is authoritative, or that its version is current. Evaluate each material claim against the specific passage, then evaluate the source's authority, freshness, scope, and version separately.

Context Selection: Chunking, Top-K, and Source Governance

Chunking and top-K are not just retrieval settings. They determine what the generator can see. Very small chunks may separate a policy rule from its exception; very large chunks can bury the decisive sentence. Increasing K can recover missing evidence, but more context can also add conflicting rules, distract the generator, raise cost, and make unsupported synthesis more likely.

For time-sensitive or jurisdiction-specific knowledge, metadata is part of retrieval quality. A leave-policy answer may need filters for country, employee type, effective date, policy owner, and document version before semantic ranking is useful. Prefer a current, authoritative India policy over an older India policy or a current US policy when the question is about India. Source freshness is not enough if a source is unofficial; authority is not enough if the source is superseded.

When evidence conflicts, the system should not silently choose the most fluent answer. It can retrieve the governing version, state the conflict, ask a clarifying question, or abstain and escalate when the available sources cannot establish a safe answer. Safe abstention is a successful outcome for an unanswerable or unresolved question.

Evaluate Retrieval, Answers, and the End-to-End Decision Separately

Retrieval evaluation asks whether the necessary authoritative evidence appeared in the candidate set and at what rank. Useful checks include coverage of answerable questions, recall at K, whether the correct source is ranked high enough, and whether filters exclude the wrong version or audience.

Answer evaluation asks whether the response is correct, complete, relevant, grounded claim by claim, clear about uncertainty, and appropriately cited. It should penalize invented details even when the main conclusion is right.

End-to-end evaluation asks whether the learner or user can make the intended decision safely. A system may retrieve the right policy and write a fluent answer yet still fail end to end if it omits an important exception, applies the wrong jurisdiction, or should have escalated instead of answering. Keep a fixed evaluation set with common requests, hard cases, ambiguous requests, conflicting sources, stale versions, and questions the corpus cannot answer.

Warning Signs and a Debugging Sequence

Warning signs include answers that cite a source from the wrong country or year, multiple claims supported by one vague citation, a highly ranked chunk that does not answer the question, a final answer containing details absent from every retrieved passage, a sudden quality drop after increasing K, and confident resolution of an explicitly conflicting policy set.

Use this sequence before tuning the model:

1. Reproduce the question and save the query, retrieved results, metadata, prompt context, answer, and citations.
2. Define the current authoritative answer and the evidence a correct response would need.
3. Check corpus coverage, access controls, source authority, effective dates, and version status.
4. Inspect retrieval at several K values, including the rank and surrounding context of the needed passage.
5. Inspect chunk boundaries to see whether rules, exceptions, and scope conditions were separated.
6. Compare every material answer claim with the supplied context; classify unsupported or contradicted claims as generation failures.
7. Test ambiguity and conflict handling: should the assistant ask, qualify, abstain, or escalate?
8. Add the case to a stable evaluation set and measure the affected layer again after a targeted change.

Interview Perspective

Question: What is RAG, and why use it instead of retraining?
Answer: RAG retrieves relevant source context at query time and gives it to a generator; it can reflect changing private documents without retraining for every update.
What the interviewer is testing: system-level reasoning and awareness that retrieval is not a truth guarantee.
Follow-up: How would you tell retrieval failure from generation failure?

Practice Questions

1. A relevant policy was never indexed. Which RAG stage failed?
2. Why can a high-similarity chunk still be unhelpful?
3. Is a paragraph containing a rule and its exception better split or kept together? What evidence would guide you?
4. Does a supplied citation prove an answer is grounded? Why not?
5. Should an exact database record lookup use RAG? Explain.

Quick Quiz

1. Does RAG retrain the model at every question? Answer: No.
2. What does the retriever do? Answer: Finds candidate evidence.
3. Can RAG still hallucinate? Answer: Yes.

Next Lesson

Next, evaluate generative systems honestly across correctness, safety, cost, latency, and real-world usefulness.