Glossary · Data & RAG
Retrieval-augmented generation (RAG)
Retrieval-augmented generation (RAG) is a technique that fetches relevant passages from your own documents and supplies them to a language model as grounding before it generates an answer.
Retrieval-augmented generation (RAG) fetches relevant passages from your own documents and supplies them to a language model as grounding before it answers. The retrieval stage (ingestion, chunking, indexing and ranking) determines quality; generation is rarely the failure point.
- RAG grounds a model on retrieved passages from your data instead of its training memory.
- Retrieval (not generation) is where enterprise RAG usually fails.
- It answers from private, current documents without retraining the model.
- Retrieval quality must be evaluated separately from answer quality.
- Grounding lets answers cite a source, which reduces unsupported claims.
Also known as: RAG, RAG pipeline
Retrieval-augmented generation (RAG) is a technique that fetches relevant passages from your own documents and supplies them to a language model as grounding before it generates an answer.
Instead of relying on what the model memorised in training, RAG puts the right source material in front of it at query time, so the answer reflects your current, private data, and can point back to where it came from.
How retrieval-augmented generation works
A RAG pipeline has two halves. The first, done ahead of time, is ingestion: documents are parsed, split into chunks, converted into embeddings, and stored in an index. The second, done per query, is retrieval and generation: the question is embedded, vector search (often combined with keyword search) finds the most relevant chunks, and those chunks are placed into the model's context along with the question. The model then generates an answer grounded in the retrieved text.
The quality of a RAG system is set almost entirely in the retrieval half. If the ingestion mis-parses a table, or the chunking splits a clause across two chunks, or the ranking surfaces an outdated policy version, the model never sees the passage it needed, and a fluent, wrong answer is the result. Generation is rarely the weak link.
Why retrieval-augmented generation matters for enterprise AI adoption
RAG is how a general model answers questions about your business without being retrained on it. Update the documents and the answers update; there is no training run, no model to redeploy. For an enterprise with policies, contracts and knowledge bases that change constantly, that is the difference between a useful assistant and a stale one.
It also makes answers traceable. Because each answer is grounded in retrieved passages, the system can cite its source, which is what a reviewer (and a regulator under India's governance expectations) will ask for. The catch is that RAG works flawlessly in a demo on clean text and struggles on real corpora of scanned PDFs, tables and conflicting versions. The gap between the two is an engineering problem in retrieval, and closing it is most of the work in an enterprise deployment.
Common mistakes with retrieval-augmented generation
The biggest mistake is judging the finished answer and never measuring retrieval. If you cannot say whether the right passage was fetched, you cannot tell a retrieval failure from a generation failure, and you will tune the wrong half. Evaluate recall against a labelled question set first, using an evaluation harness.
The second is naive, one-size chunking that ignores document structure, splitting tables and clauses so the retrieved fragment is meaningless. The third is ignoring versioning: when two editions of a policy both sit in the index, the system will sometimes confidently retrieve the superseded one, and the answer will be wrong in the most expensive possible way.
Related terms
- Vector search: the similarity retrieval at the core of a RAG pipeline.
- Embeddings: the numerical representations RAG retrieves over.
- Model Context Protocol: a way to expose retrieval to a model as a tool or resource.
- Evaluation harness: how retrieval and answer quality get measured separately.
How Chokmah approaches retrieval-augmented generation
We build RAG for the documents you actually have, not the clean ones in a demo. In a workflow sprint we start with ingestion (parsing scanned PDFs, tables and versioned policies properly), and we evaluate retrieval on its own, against a labelled question set, before anyone judges an answer. This is domain-wedge work for us: production retrieval over messy enterprise corpora is exactly the ground where naive pipelines break and a measured one holds.
Sources
- MIT NANDA, The GenAI Divide: State of AI in Business 2025, July 2025. https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf
A retrieval-augmented generation pipeline
Text description of this diagram
A five-stage RAG pipeline: Query, Retrieve, Augment, Generate, Evaluate, connected left to right by arrows, with a dashed feedback loop from Evaluate back to Retrieve labelled retrieval budget tuned each cycle. It shows RAG as an evaluated retrieval loop, not a one-shot product.
Related terms
- Vector searchVector search retrieves information by comparing numerical embeddings for semantic similarity, finding passages that mean the same thing as a query rather than sharing its exact keywords.
- EmbeddingsAn embedding is a list of numbers that represents the meaning of text or other data, positioning similar items close together in a high-dimensional space so software can compare them.
- Model Context Protocol (MCP)The Model Context Protocol (MCP) is an open standard that defines how AI applications connect to external tools and data through one uniform interface instead of many bespoke integrations.
- Evaluation harnessAn agent evaluation harness is a repeatable test suite that scores an AI agent's outputs against fixed, versioned cases before and after every change, so teams can tell regression from variance.
Where it shows up in our work
Frequently asked questions
Fine-tuning changes the model's weights to shift its behaviour or style; RAG leaves the model unchanged and instead supplies relevant documents at query time. For keeping answers current and grounded in private, frequently-changing data, RAG is usually the right tool because you update the documents, not the model. Fine-tuning suits fixed behaviours and formats. Many production systems use RAG for knowledge and reserve fine-tuning for tone or structure.
Because the demo used clean text and production uses scanned PDFs, tables, and versioned policies. The failure is almost always in retrieval, not generation: if the right passage is never fetched, no amount of model quality recovers the answer. Naive chunking splits tables and clauses, conflicting document versions surface the wrong one, and nobody measured retrieval separately. Fixing ingestion and retrieval fixes most RAG problems.
Measure retrieval and generation separately. For retrieval, build a labelled set of questions with the passages that should be found, then track recall (did the right passage make it into the context) before judging any answer. Only once retrieval is sound do you evaluate the generated answers for faithfulness to the retrieved text. Judging the final answer alone hides whether the problem is finding the information or using it.
It reduces unsupported answers but does not eliminate them. Grounding the model on retrieved passages gives it correct material to work from and lets answers cite a source, which cuts fabrication when retrieval succeeds. But a model can still misread a passage, blend two sources, or answer confidently when retrieval returned nothing relevant. That residual risk is why consequential RAG answers still need evaluation and, often, a human check.
Put the concept to work
We install working agentic workflows, not vocabulary. Book a free AI Reality Check.