Chapter 4: Retrieval-Augmented Generation — Grounding Agents in Knowledge They Were Never Trained On

Introduction: What the Model Doesn’t Know

A language model knows an enormous amount and none of it about you.

It has read a large fraction of the public internet, but it has never seen your company’s wiki, last night’s support tickets, the contract you signed on Tuesday, or the pricing that changed this morning. Its knowledge also stops at a training cutoff — ask about anything after that date and it will either admit ignorance or, worse, invent a confident answer. And even for facts it does hold in its weights, it can’t show you where they came from. There’s no footnote, no source, no way to check.

For an agent, those three gaps — private data, freshness, and attribution — are the difference between a party trick and a system you’d let near production. An agent that answers questions about your codebase from memory is guessing. An agent that reads the actual file first is working.

Retrieval-Augmented Generation is the technique that closes those gaps. The idea is almost embarrassingly simple: before you ask the model to answer, go find the relevant text and put it in the prompt. Don’t rely on what the model memorized during training — retrieve the facts at query time, hand them over, and ask the model to answer using them. The model supplies the language and reasoning; your data supplies the truth.

That one move — retrieve, then generate — turns out to have a lot of moving parts. You have to decide how to break documents into pieces, how to represent them so “relevant” becomes a computable thing, where to store them, how to search, how to rank what comes back, and how much of it to actually show the model. Get those wrong and RAG produces confident nonsense grounded in the wrong paragraph. Get them right and you have an agent that stays current, cites its sources, and knows things no model was ever trained on.

This chapter is a tour of that machinery, from the ground up. We’ll start with the core idea and the classic pipeline, work through the building blocks every RAG system shares, then confront the fact that real data isn’t clean prose — structured data retrieved by querying rather than embedding, messy documents full of tables and images, and even audio and video. Then we’ll look at how the cloud providers and startups package all of this as a product. Finally we’ll get to the frontier — the techniques that are turning RAG from a fixed pipeline into something an agent actively drives: agentic retrieval, visual document understanding, knowledge graphs, and self-correcting retrieval loops.

By the end you should be able to look at any RAG system and answer: how does it chunk, how does it search, how does it rank, and where would I intervene to make it better?

A note on scope: this chapter is about how retrieval works. How to measure whether it works — the faithfulness, context-relevance, and answer-relevance metrics, and frameworks like RAGAS — belongs with evaluation, and we treat it there (Chapter 6). We’ll flag the evaluation hooks as we go but won’t rebuild that toolkit here.

1. What RAG Is and Why It Exists

1.1 The Core Idea

The technique was introduced by Lewis et al. in 2020, in a paper that gave it the name. Their framing is still the clearest way to think about it. A model has parametric memory — knowledge baked into its weights during training. RAG adds non-parametric memory — an external store of text the model can look things up in. In the original paper the parametric part was a seq2seq model (BART), and the non-parametric part was a dense vector index of all of Wikipedia, searched by a neural retriever. The retriever finds passages, the generator writes the answer conditioned on them.

The result held up: combining the two beat a parametric-only model of the same size, and produced “more specific, diverse and factual” text. That last word is the one that matters. Grounding the generation in retrieved passages, rather than the model’s own recollection, is what pulls the answer toward being factual — and it’s why RAG became the default pattern for putting an LLM to work on data it wasn’t trained on.

There’s a useful mental model hiding in that split. A bare LLM is a brilliant expert with no reference materials — everything it says is from memory. RAG hands that expert an open book and says “answer from this.” The expert’s skill (language, reasoning, synthesis) is unchanged; what changes is that the answer is now anchored to a source you control and can update.

1.2 Why a Builder Reaches for It

Four concrete payoffs, each mapping to one of the gaps from the introduction:

• Grounding / fewer hallucinations. When the relevant facts are sitting in the prompt, the model doesn’t have to reconstruct them from fuzzy parametric memory. It still can go off-script, but a good RAG system makes the correct answer the path of least resistance.
• Freshness. The index is just data. Re-index when things change and the agent’s knowledge is current — no retraining, no waiting for the next model release.
• Private / proprietary data. Your documents were never in the training set and never will be. Retrieval is how the model sees them at all.
• Attribution. Because you know which chunks you retrieved, you can cite them. That single property — “here’s where this came from” — is often what makes RAG acceptable in regulated or high-stakes settings where an unsourced answer is a non-starter.

Keep those four in mind, because every design decision later in the chapter is a trade-off against them. When someone proposes a fancier retriever or a cheaper index, the question is always: does this help the agent give a grounded, fresh, sourced answer — or does it just look sophisticated?

2. The RAG Pipeline

Almost every RAG system, from a weekend prototype to a hyperscaler product, runs the same pipeline. It splits cleanly into two phases: an offline phase that prepares your data, and an online phase that runs on every query.

The stages, in the words AWS uses to define them:

1. Ingestion — load the source documents; clean and normalize them (strip boilerplate, fix encodings, pull text out of PDFs).
2. Chunking — split each document into retrievable units. This is the step that quietly decides how good your whole system will be.
3. Embedding — turn each chunk into a vector “that captures the semantic or contextual meaning” of the text.
4. Indexing — store those vectors in a database built for fast similarity search.
5. Retrieval — embed the incoming query the same way, search the index, and pull back the top-k most similar chunks.
6. Augmentation + Generation — paste the retrieved chunks into the prompt alongside the question, and let the model write the answer.

One correction to how this is often taught: the offline phase is not a one-time event. AWS’s docs describe ingestion as an upfront step, and for a demo it is. In production your data changes — documents get edited, added, deleted — and the index has to keep up. Re-embedding cost, change detection, and honoring deletions (someone exercises their right to be forgotten, and that chunk had better disappear from search) are all real operational concerns that the tidy diagram hides. If you remember one thing about the pipeline, make it this: the index is a living copy of your data, and stale retrieval is just a slower way to hallucinate.

3. The Building Blocks

The pipeline has six boxes, but four of them — embeddings, chunking, indexing, and search — are where the engineering actually lives. Let’s take them in turn.

3.1 Embeddings: Turning Meaning Into Geometry

An embedding is a list of numbers — a vector — that represents a piece of text’s meaning as a point in high-dimensional space. The trick that makes them useful: texts that mean similar things land near each other, even when they share no words. “How do I reset my password?” and “steps to recover account access” have almost no vocabulary in common, but a good embedding model places them close together.

“Close” is made precise by a similarity metric. The two you’ll meet most are cosine similarity (the angle between two vectors, ignoring their length) and dot product (angle and magnitude). Cosine is the default for text retrieval because it cares about direction — meaning — not magnitude, which tends to track incidental things like passage length. A two-sentence note and a five-paragraph essay on the same topic should count as equally relevant, and cosine treats them that way; dot product would let the longer one score higher just for being longer. Everything downstream in RAG rests on this one idea: relevance becomes distance, and finding relevant text becomes finding nearby points.

3.2 Chunking: The Underrated Decision

You can’t embed a 40-page document as a single vector and expect useful retrieval — the meaning gets averaged into mush, and you’d have to feed the whole thing to the model anyway. So you split documents into chunks. How you split them is one of the highest-leverage, least-glamorous choices in the whole system:

• Fixed-size — every N tokens, maybe with a little overlap. Dead simple, and a fine baseline. Its flaw is that it chops sentences and ideas mid-thought.
• Recursive — split on structure first (sections, then paragraphs, then sentences), falling back to smaller units only when a chunk is still too big. Respects the document’s natural boundaries. This is the sensible default for most text.
• Semantic — split where the topic shifts, detected by looking at where consecutive sentences’ embeddings diverge. More expensive to compute, but produces chunks that are each about one thing.

The tension is always the same. Chunks too small, and each one lacks the context to be understood on its own (“it supports up to 64” — 64 what?). Chunks too big, and a single chunk covers several topics, so retrieving it for one drags in irrelevant text and dilutes the signal. There’s no universal answer, but chunking is the first place to look when retrieval quality is disappointing — long before you reach for a fancier model.

3.3 Indexing: Searching Millions of Vectors Fast

Once you have a few million chunk-vectors, you need to find the nearest ones to a query vector in milliseconds. Checking every vector (exact k-nearest-neighbor) is accurate but scales linearly — fine for ten thousand vectors, hopeless for a hundred million. So production systems use Approximate Nearest Neighbor (ANN) indexes that trade a sliver of recall for enormous speed.

Two families dominate:

• HNSW (Hierarchical Navigable Small World) — connects each vector to its nearest neighbors so search becomes “start somewhere, then keep hopping to a closer neighbor until you can’t get closer.” The layers are the trick: a sparse top layer with long-range links acts like a highway that jumps you across the space, and denser lower layers are the local roads that home in on the exact neighborhood — so you reach a query’s neighbors in a handful of hops instead of scanning everything. It’s the workhorse: Elasticsearch stores dense vectors using Lucene’s HNSW, and most vector databases offer it.
• IVF (Inverted File) — clusters vectors up front, then at query time only searches the few clusters nearest the query. Cheaper memory, a little more tuning.

On top of the index sits quantization — compressing each vector (from 32-bit floats down to 8-bit, 4-bit, or even 1-bit “binary” representations) to shrink memory. Elastic’s binary quantization, for instance, claims up to 95% memory reduction (a vendor figure, but the technique is real and widely used). You give up a little precision; you get to fit far more vectors in RAM. For most builders these knobs are defaults you rarely touch — but knowing they exist explains why “the vector database” is a real category of software and not just a table with a distance function.

3.4 Dense vs. Sparse, and Why You Want Both

Everything so far describes dense retrieval — learned embeddings, semantic matching. There’s an older, complementary approach: sparse retrieval, the classic keyword search built on BM25, which scores documents by term-frequency statistics. Sparse retrieval has no idea what words mean, but it’s unbeatable at exact matches — a product SKU, an error code, a person’s name, a rare technical term.

These two catch different failures. Dense retrieval finds the paraphrase but can miss the literal string; sparse retrieval nails the literal string but misses the paraphrase. So the pragmatic answer is hybrid search: run both and merge the results. The merge is usually Reciprocal Rank Fusion (RRF), which combines the two ranked lists by position rather than by raw score:

score(d) = Σ 1 / (k + rank(d)), with k typically 60.

The elegance of RRF is that it sidesteps a real problem: BM25 scores are unbounded and cosine similarities live in [-1, 1], so you can’t just add them. Working on ranks instead of scores makes the two comparable. Hybrid search is one of the cheapest reliable wins in RAG — if you’re running dense-only, adding BM25 and RRF is usually the first upgrade worth making.

3.5 Reranking: A Second, Sharper Opinion

First-stage retrieval optimizes for speed, so it’s a little blunt — it hands back, say, the top 50 candidates. A reranker is a slower, more accurate model that re-scores those candidates and keeps the best handful. The key architectural difference: the embedding models used for retrieval are bi-encoders (they encode the query and each document separately, so document vectors can be precomputed), while rerankers are cross-encoders — they look at the query and a document together, which is far more accurate but far too expensive to run over your whole corpus. So you use the fast bi-encoder to get 50 candidates and the slow cross-encoder to pick the top 5.

Cohere’s Rerank is the best-known commercial reranker (Voyage offers another, and there are strong open cross-encoders). Adding a reranker is, along with hybrid search and better chunking, one of the three moves that reliably lift retrieval quality — which is exactly why, as we’ll see, Anthropic’s contextual-retrieval experiments stack all three.

4. Retrieving Structured Data

Everything in §3 described retrieval over unstructured text: embed a chunk, compare vectors, rank by similarity. That’s the part of RAG people picture. But it invites a fair objection — most enterprise data isn’t a wall of prose. A support ticket has a free-text description, sure, but it also has a status, a priority, an assignee, a customer ID, and a timestamp. Those are structured fields, and running embedding-based search over them is nonsense: there’s no meaningful “cosine similarity” to the value high or to a creation date. So is structured data simply out of RAG’s scope?

No — but it’s retrieved by a different mechanism. The general definition of RAG is “fetch the relevant context and put it in the prompt,” and fetching has more than one tool. Unstructured text you search; structured data you query. The key distinction to hold onto:

• Semantic search (embed + vector/hybrid) — for the free-text fields where meaning matters.
• Querying (a filter, or SQL generated from the question) — for the structured fields where exact values matter.

And because a real record like that support ticket is a mix of both, mature systems combine them in one retrieval step. “Summarize last night’s open, high-priority tickets that mention checkout failures” decomposes into a structured filter (status = open AND priority = high AND created > …) and a semantic search (mentions checkout failures) over the description. In practice this shows up as four patterns, in rough order of how “structured” the retrieval really is:

Metadata filtering is table-stakes — it’s GA in every product from §9’s vendor list. You tag each vector with structured attributes and pass a filter alongside the query: Pinecone uses MongoDB-style operators, Weaviate and Elastic combine where/filter clauses with vector search, and the clouds do it too — AWS Bedrock Knowledge Bases even offers implicit filtering, where a Claude model writes the filter from your query and a schema you provide. This is the cheapest way to make retrieval respect structure (“only this customer’s docs,” “only the last 30 days”), and there’s no excuse not to use it.

Natural-language-to-SQL is the real thing — actually querying a database as the grounding step — and it’s now a headline feature, not a research demo. AWS Bedrock Knowledge Bases has structured data retrieval built in: connect a store, and RetrieveAndGenerate converts your question to SQL (via a Redshift query engine), runs it, and answers from the result (GA since late 2024). Microsoft’s Fabric data agent generates SQL/DAX/KQL over governed OneLake data and plugs into Azure AI Foundry (GA) — notably it’s the structured-only counterpart to unstructured RAG and won’t touch a PDF. Google keeps NL-to-SQL out of Vertex AI Search and down in the databases, where BigQuery’s VECTOR_SEARCH composes with ordinary SQL WHERE clauses. In the framework world, LlamaIndex ships text-to-SQL query engines and routers that decide, per question, whether to hit the database or the vector store. And the data platforms have made this their whole pitch: Databricks Genie and Snowflake Cortex Analyst both answer natural-language questions by generating SQL against governed warehouse tables. One caution the AWS and LlamaIndex docs both stress: letting an LLM run generated SQL against a live database is a real security surface — use read-only roles and sandboxing, never a write-capable connection.

The last two patterns are worth distinguishing because vendors blur them. The RAG-native startups — Cohere, Contextual AI, Vectara — mostly handle “structured” as tables inside documents: their parsers extract a table from a PDF into Markdown or JSON so its cells become retrievable text. That’s useful (it’s how a financial table in a filing gets into RAG at all) but it is not querying a database. And enterprise search like Glean takes a fourth route: it ingests structured fields from source systems through connectors — confirmed at the field level for Salesforce, via SOQL over standard and custom objects — but represents them as searchable facets and ranking signals on a document-centric index, not a relational store you join against. When Glean needs a live structured query it delegates — its Databricks connector hands natural-language querying to Genie and runs prepared SQL from inside an agent. (A caveat its own docs admit: Glean’s Salesforce indexing doesn’t enforce field-level security, so a restricted field can surface in an answer.)

The reason this matters beyond tidiness: choosing between these is exactly the routing decision agentic RAG makes. An agent that can ask “is this a semantic question or a lookup?” and send it to the vector store, a SQL query, or both is doing structured and unstructured retrieval under one roof — which is where §10’s frontier is headed. For now, the takeaway is simply: structured data is very much in scope; you just query it instead of embedding it.

5. Getting Messy Documents In — Tables, Images, and Formats

§2’s pipeline quietly assumed your documents are clean prose you can chunk and embed. Real enterprise data isn’t. It’s PDFs full of tables and charts, PowerPoint decks, Excel workbooks, scanned invoices, and Word docs with embedded images. If you run a naive text extractor over a financial filing, the tables come out as a scrambled stream of numbers and the charts vanish entirely. Ingestion is where most production RAG systems actually fail — not in the embedding model, but in the unglamorous work of turning a messy document into retrievable text. This part is about that work.

5.1 Tables

A table’s meaning lives in its structure — which number sits in which row and column. Flatten it to a line of text and you lose exactly the thing that made it a table. So the job is table-structure recognition: detect the table, recover its grid, and emit a structured representation. The established tools are layout-aware vision models:

• Microsoft Table Transformer (TATR) — a DETR-based object-detection model trained on the PubTables-1M dataset. It does three things — detect the table, recognize its row/column/cell structure, and identify cell roles (headers vs. data) — then emits HTML or CSV. One catch worth knowing: it’s detection-only, so you supply the cell text separately from OCR or the PDF’s text layer.
• IBM TableFormer — a transformer-based structure recognizer that handles the hard cases (merged cells, multi-line rows, empty entries, missing rulings). Its reported accuracy is strong, though the headline numbers are self-reported 2022 benchmarks — treat them as “good,” not current SOTA. It’s the table engine inside IBM’s Docling (below).
• PaddleOCR PP-StructureV3 — a full document-parsing pipeline (layout + table + formula recognition, reading-order restoration) that exports Markdown, JSON, DOCX, HTML, or XLSX.

Once a table is extracted, you have the same choice from §4: keep it as text (render it to Markdown or HTML and embed that — good for “what does the table say about X”) or treat it as data (load the cells into a database and query with text-to-SQL — better for “sum the Q3 numbers”). Retrieval questions want the former; analytical questions want the latter.

5.2 Images, figures, and charts

Getting a chart or diagram into RAG comes down to two architectures, and the choice matters:

• (a) Grounding-to-text — run a vision-language model over each figure to produce a caption or description, then index that text and do ordinary text RAG. Simple, and it reuses your whole text pipeline. The cost is that captioning happens once, up front, and whatever detail the caption misses — a specific number in a chart, a spatial relationship in a diagram — is gone before retrieval ever runs.
• (b) Unified multimodal embedding — embed the image directly into the same vector space as text, so a text query can retrieve an image with no captioning step. Production options are GA: Cohere Embed v4.0 (text, images, and mixed PDFs in one space), Jina Embeddings v4, and Voyage multimodal. This preserves the visual detail for both retrieval and generation.

The tradeoff is information loss: caption-to-text discards visual, spatial, and numerical detail during preprocessing; unified embeddings keep it. One 2025 study (PwC, on financial documents) found unified embeddings beat captioning on both retrieval and answer quality — but that’s a single small preprint with a debatable baseline, so lean on the reasoning, not the number. The visual-first extreme of architecture (b) is ColPali, which embeds whole page images and skips text extraction entirely; because it’s really a retrieval-matching advance, we cover it with the frontier techniques in §10.

5.3 Document-parsing pipelines (the practical layer)

In practice you don’t wire up TATR and a captioner yourself — you reach for a parsing tool that handles layout, tables, and images together and hands you clean output. The three verified against first-party docs:

• Docling (IBM, MIT-licensed, open source) — ingests PDF, DOCX, PPTX, XLSX, HTML, EPUB, email, and images; does layout-aware understanding (it uses TableFormer for tables) and normalizes everything into a canonical DoclingDocument that exports to Markdown, HTML, or lossless JSON. Ships with chunkers that work on that structure.
• LlamaParse (LlamaIndex, commercial) — a parsing service with four quality tiers, from Agentic Plus (most accurate, slowest, priciest) down to Fast, so you dial in the accuracy-vs-cost tradeoff per job. It outputs Markdown, JSON, or tables-as-spreadsheet. The top tiers are marketed for complex tables and charts, though that’s vendor copy — competitors dispute its accuracy on scanned and handwritten inputs.
• Unstructured.io — Transform/Foundation/Pipelines; its Pipelines product is an end-to-end RAG ingestion ETL (connect → route → parse → chunk → enrich → embed → persist) into the major vector DBs.

5.4 Many formats, one representation

The unifying pattern across all of these: normalize every format to a canonical representation — usually Markdown — before chunking. Whether the input is a Word doc, a slide deck, an email, or a scanned PDF, you convert it to one clean structured form and run a single downstream pipeline. Microsoft’s open-source MarkItDown and the parsers above all embody this. A few format-specific notes:

• Word (.docx) — mostly clean text plus embedded tables and images; the easy case.
• PowerPoint (.pptx) — each slide is text plus images plus speaker notes; don’t forget the notes, which often carry the real narrative.
• Excel (.xlsx) — usually better treated as structured data (§4) than embedded as prose; a spreadsheet is a database in disguise.
• Email (.eml/.msg) — threads, quoted replies, and attachments; the .eml standard is well supported, proprietary Outlook .msg less so.

The takeaway for a builder: budget real effort for ingestion. A mediocre embedding model on well-parsed documents beats a great one on garbled text every time.

6. RAG for Sound and Vision

So far, everything has been text — even the images end up as embeddings or captions. But agents increasingly need to retrieve over audio and video: meeting recordings, support calls, podcasts, lecture videos, security footage. Is there a RAG for sound and vision? Yes — and the honest answer is that audio is largely solved, video is arriving as products, and true long-video RAG is still research.

6.1 Audio

The production default is refreshingly boring: transcribe, then do text RAG. Speech-to-text has gotten good and cheap, so the dominant pattern is to convert audio to a transcript and feed it into the pipeline you already have. Three GA services lead:

• Deepgram (Nova-3), AssemblyAI, and OpenAI’s GPT-4o Transcribe (including a gpt-4o-transcribe-diarize model) — all provide speaker diarization (who spoke) plus word-level timestamps (when).

Those two features are what make audio genuinely useful in RAG: they let you chunk a transcript by speaker turn and time, so a retrieved chunk can be attributed (“the customer said, at 14:32…”) and linked back to the exact moment in the recording. OpenAI’s Whisper remains the popular open-source baseline for transcription. Once you have transcripts, meetings, calls, and podcasts are just text — searchable, chunkable, citable.

When the meaning isn’t in the words — you’re retrieving over music, sound effects, or acoustic events with no useful transcript — you need native audio embeddings. The standard here is CLAP (Contrastive Language-Audio Pretraining), the audio analog of CLIP: it puts audio and text in a shared space so a text query (“dog barking,” “upbeat jazz”) retrieves matching audio. CLAP is open source (LAION and Microsoft lineages), not a managed API, so you self-host it. The rule of thumb: transcribe-then-RAG when meaning lives in speech; CLAP embeddings when it lives in the sound itself. (Fully transcription-free speech RAG, like the research system VoxRAG, exists but reports weak numbers — not a default yet.)

6.2 Video

Video is the hardest modality because it’s all the others at once — moving images, speech, on-screen text, and sound. Two approaches:

• The DIY fusion pattern — sample frames and caption them (or embed them with a multimodal model), transcribe the audio track, extract on-screen text, and fuse it all into text RAG. It works with tools you already have and is the common practical route today.
• Video-native retrieval — embed the video directly. The leading GA product is Twelve Labs, whose Marengo model (a multi-vector embedder decomposing video into visual, motion, on-screen-text, and speech vectors) powers any-to-any cross-modal search — text-to-video, image-to-video, and more — through GA Embed and Search APIs with pay-as-you-go pricing. Its companion Pegasus is a video-language model for summarization and generation. (“Video-native” means retrieval over the raw video without a separate frame-sample-and-caption pipeline.)

There are two adjacent options worth knowing. Long-context models (feeding a whole video straight into a model like Gemini) sidestep retrieval entirely when the video is short enough to fit — the same RAG-vs-long-context tradeoff from §7, now for video. And on the research frontier, VideoRAG (graph-based retrieval over hours-long video, with the LongerVideos benchmark) points at where long-video RAG is heading — but it’s a paper, not a product.

The practitioner summary: audio → transcribe and reuse your text stack; video → Twelve Labs if you want a managed video-native index, or a frame+transcript fusion pipeline if you’d rather assemble it yourself; long-video RAG → watch the research.

7. RAG, Long Context, or Fine-Tuning?

Newcomers often ask why RAG is needed at all now that models have million-token context windows — just paste everything in. And separately: why not fine-tune the model on your data? These are the three ways to get private knowledge into an LLM’s answer, and they solve different problems. The mistake is treating them as competitors when they’re usually collaborators.

The cleanest way to hold it in your head: fine-tune to change how the model behaves, RAG to change what it knows, long context to hold what it’s working on right now. A well-built assistant often uses all three — fine-tuned for the house voice, RAG for the knowledge base, and a long context window to hold the current document under discussion. When a corpus is genuinely large, RAG tends to win on both cost and accuracy against stuffing everything into context — a point vendors like Contextual AI make loudly, and one that holds up as your data outgrows any window.

8. RAG in the Cloud — the Hyperscaler Offerings

You can assemble all of §3 yourself. Increasingly, you don’t have to: every major cloud now sells the whole pipeline as a managed service. The through-line across all three — and the theme to watch — is a pivot from single-shot retrieval (embed the query once, fetch once, generate) toward “agentic retrieval,” where an LLM plans the search: decomposing the question, issuing multiple sub-queries, and iterating. Hold that thought; it’s the bridge to §10.

Product status moves monthly. Everything below is accurate as of this writing (mid-2026), with GA/preview flagged — but verify before you build on it.

8.1 AWS

Amazon Bedrock Knowledge Bases is AWS’s managed RAG service (GA). It grounds apps and agents in your proprietary data and manages “the vector store, embeddings and re-ranking models used during retrieval” for you. It connects to S3, SharePoint, Confluence, Google Drive, and web crawls, and if you’d rather own the vector store you can bring your own (OpenSearch Serverless, Aurora, Neptune).

Its agentic-RAG feature, Agentic Retrieval, “automatically understands user intent, identifies the most relevant data sources, and iterates through multiple retrieval steps” — decomposing a query into sub-queries routed across up to five retrievers. AWS reports a roughly 20% recall gain on multi-hop questions (their number, not independently verified).

The newest piece is Amazon S3 Vectors (launched July 2025, preview), which adds native vector storage and query directly to S3 object storage — pitched as a low-cost tier for vectors you query infrequently, integrated with Bedrock Knowledge Bases. AWS also fields Kendra (managed enterprise search) and the OpenSearch k-NN engine for teams that want more control.

8.2 Microsoft Azure

Azure AI Search (GA) is the retrieval backbone — the “R” in RAG, not a whole RAG system. It supports full-text, vector, hybrid, and multimodal queries, and the generation LLM lives in your application, consuming what the search returns. What Azure exposes is two retrieval engines, and the split is a perfect illustration of the industry shift — the difference is whether an LLM is involved in the search itself:

• Classic search — no LLM in the retrieval step. You send a query, it returns ranked results in one request, and your app hands those to the LLM for generation.
• Agentic retrieval — an LLM now sits inside retrieval: it plans the query, decomposes it into focused sub-queries, retrieves them in parallel, applies semantic reranking, and merges the results before your app generates the answer. It’s GA (via the 2026-04-01 API), with some capabilities still in preview.

The notable deprecation: Azure OpenAI “On Your Data,” the older turnkey “point the model at your Azure Search index” feature, is deprecated and retires on October 14, 2026. Microsoft is steering everyone to Foundry Agent Service with Foundry IQ. If you have anything running on “On Your Data,” that date is a migration deadline.

8.3 Google Cloud

Vertex AI RAG Engine is Google’s managed pipeline, now branded “a component of Gemini Enterprise Agent Platform.” It implements the familiar six stages and lets you plug in a range of vector backends — its own managed store, Vertex Vector Search, or third parties like Weaviate and Pinecone.

The enterprise-search product sits alongside it and comes with a warning: it has been rebranded repeatedly — Generative AI App Builder, then Enterprise Search, then Vertex AI Search and Conversation, then Agent Builder, then AI Applications, then Vertex AI Search, and now Agent Search. It offers out-of-the-box semantic search over your sites and data plus “Generate grounded answers with RAG.” Just know that Google’s own docs and console are mid-rename and internally inconsistent; if the name in the console doesn’t match the docs, you haven’t lost your mind.

8.4 The pattern across all three

Notice what happened. AWS has “Agentic Retrieval,” Azure has an “agentic retrieval” engine, Google’s product is a component of an “Agent Platform.” Three independent teams, same move: from a fixed retrieve-then-generate call to an LLM that plans retrieval — and from selling “RAG” to selling “agents.” That’s not marketing coincidence; it reflects a real technical shift we’ll unpack in §10.

9. The Vendor Landscape — Startups and Specialists

Beyond the clouds is a crowded field of specialists. Two caveats before the tour. First, most of what follows is drawn from the vendors’ own descriptions of their products — accurate as positioning, but performance numbers are self-reported unless noted, so read them as claims, not benchmarks. Second, this space moves fast and consolidates faster — one of the companies below is already winding down, which is itself the most honest signal about the market.

9.1 Enterprise “Work AI” search

Glean is the flagship enterprise-RAG company. It sells a “Work AI platform” — Enterprise Search, an AI Assistant, and AI Agents — over everything your company knows. Its real differentiator is the Enterprise Graph: a knowledge graph mapping how people, content, and systems relate, built on connectors and a permission-aware index so every result respects who’s allowed to see what. That permissions layer is the hard, unsexy problem enterprise search lives or dies on, and it’s Glean’s moat.

Perplexity, known for its consumer answer engine, offers the same capability as an API through its Sonar models — a tiered lineup (Sonar, Sonar Pro, Sonar Reasoning Pro, Sonar Deep Research) with web-search grounding built in. It’s RAG-as-a-service where the corpus is the live web. Tellingly, Perplexity is migrating Sonar toward an “Agent API” — the same RAG-to-agent rebrand we saw in the cloud.

9.2 RAG platforms and document intelligence

Vectara began as RAG-as-a-service and has repositioned as “The Enterprise Agent Platform” — retrieval is now one capability inside an agent product. The rebrand is the story.

Contextual AI takes the most distinctive technical stance, which it calls “RAG 2.0.” Instead of stitching together a frozen embedding model, a vector database, and a black-box LLM — a “Frankenstein’s monster,” in their words — it trains the retriever and the language model together, end to end. The productized result is a Contextual Language Model (CLM), which they claim beats GPT-4-based RAG baselines (a vendor claim, on benchmarks they chose). The idea is worth understanding even if you never use the product: it’s the sharpest articulation of the argument that classic RAG’s weakness is that its parts were never optimized for each other.

LlamaIndex started as the popular open-source RAG data framework and now sells LlamaCloud on top. Its standout is LlamaParse, a vision-model-powered document parser that handles complex layouts, tables, charts, handwriting, and multi-page tables — the messy PDF reality that trips up naive ingestion. It’s GA. If your pain is “our RAG is bad because our PDFs are a nightmare,” this is the category to look at.

Ragie built a slick RAG-as-a-service “context engine for agents” — hybrid indexing, agentic retrieval, “Agentic OCR” for tables and charts, connectors, an MCP server. It’s here as a cautionary data point: Ragie announced it is shutting down. The managed-RAG middle is being squeezed from above by the hyperscalers and from below by open source, and not everyone survives the squeeze. Factor platform durability into your build-vs-buy decision.

9.3 Vector databases

These are the storage-and-search engines underneath everything else:

• Pinecone — serverless, object-storage-backed vector DB. Offers dense, sparse, and full-text indexes in one place and combines them via “cascading retrieval” (dense + sparse + integrated reranking). Sells low latency at scale and enterprise compliance (SOC 2, HIPAA, ISO 27001). The “just works, fully managed” option.
• Weaviate — open-source vector DB with three deployment modes: self-hosted, managed cloud, or BYOC (runs in your own VPC). Hybrid search, built-in multimodal embeddings, and a “Query Agent” that turns intent into optimized queries. The choice when you want open source and control over where data lives.
• Elasticsearch — the search incumbent, now a serious vector database. Dense vectors on Lucene HNSW, its own ELSER sparse encoder, BM25, and RRF hybrid retrieval in a single API call. Its semantic_text field auto-handles embedding and chunking and is GA (Stack 9.0+). The natural pick if your logs and search already live in Elastic.
• Vespa — a distributed serving engine unifying retrieval, ranking, and ML inference; built for billions of items and thousands of QPS at sub-100ms latency. Open source plus a managed Vespa Cloud. The heavy-duty option when scale and ranking sophistication matter more than setup simplicity.

9.4 Retrieval components: embeddings and rerankers

Not every specialist sells a database. Cohere sells the models that make retrieval good: Rerank (a cross-encoder reranker, now at Rerank 4 as of late 2025, multilingual, deployable in your VPC) and Embed — whose embed-v4.0 is multimodal (text, images, PDFs), lets you pick the embedding size (256 up to 1536 dimensions from the same model, trading storage and speed for accuracy — a technique called Matryoshka), and handles a 128k-token context. These are the plug-in upgrades you drop into an existing pipeline — swap in a better embedder or bolt on a reranker without rebuilding anything.

The takeaway from the whole landscape: you’re rarely choosing one product; you’re assembling a stack — an embedder, an index, a reranker, and a parser — and the interesting startups each specialize in making one of those layers excellent.

10. The Agentic Frontier

Everything so far describes RAG as a pipeline — a fixed sequence you build once and run the same way every time. The frontier is about making retrieval dynamic: something an agent plans, inspects, corrects, and re-runs. This is what all those “agentic retrieval” product features are reaching for, and it’s where the research is most alive.

It helps to organize the frontier by which part of the pipeline each technique attacks. There are four levers — each one improving a different stage of the pipeline from §2:

10.1 Lever 1: Control flow — let the agent drive retrieval

Classic RAG retrieves once, blindly, whether or not the query needs it and whether or not the results are any good. Agentic RAG fixes that by putting an agent in charge of retrieval. The survey by Singh et al. (Jan 2025) defines it as embedding autonomous agents into the RAG pipeline so they apply the usual agentic patterns — reflection, planning, tool use, and multi-agent collaboration — to dynamically manage how retrieval happens. Concretely, that means: decomposing a complex question into parts, doing multi-hop retrieval (use the first result to inform the next search), treating retrieval as a tool the agent calls when it decides it needs to, and looping until it has enough. This is the research name for exactly what AWS, Azure, and Google shipped.

Several specific techniques make retrieval adaptive and self-correcting:

• Self-RAG trains a model to decide whether to retrieve at all on a given query, and then to critique what it got back using special “reflection tokens” (is this passage relevant? does it support my answer? is the answer useful?). Retrieval becomes on-demand rather than automatic, and the model grades its own sources.
• Corrective RAG (CRAG) adds a lightweight evaluator that scores the quality of what retrieval returned. If the results look good, proceed; if they look bad, take corrective action — including falling back to a web search to find better sources. It’s the “retrieve, check, and fix if wrong” loop made explicit.
• Query rewriting (“Rewrite-Retrieve-Read”) notes that the user’s raw question is often a bad search query, and inserts a step that rewrites it into something the retriever handles better. It’s the cheapest agentic trick and frequently the highest-return.
• HyDE (Hypothetical Document Embeddings) is a clever inversion: instead of embedding the question and searching for similar documents, have the LLM write a hypothetical answer first, then embed that and search — because a fake answer looks more like the real documents than the question does. Useful when you have no training data to tune a retriever.

10.2 Lever 2: Corpus structure — reorganize the data for reasoning

Some questions can’t be answered by any single chunk — “what are the main themes across these 500 documents?” has no one paragraph that contains the answer. Two techniques restructure the corpus so multi-level questions become answerable:

• RAPTOR builds a tree by recursively clustering and summarizing chunks bottom-up. Leaf nodes are the original granular chunks; higher nodes are summaries of summaries. At query time it can retrieve a fine-grained detail or a high-level synopsis, whichever the question needs.
• Microsoft GraphRAG goes further and has an LLM read the whole corpus to build a knowledge graph — extracting entities, relationships, and claims — then clusters that graph into communities and summarizes each one. This unlocks global search (“what are the overarching themes?”) that ordinary chunk-retrieval can’t touch, alongside local search for entity-specific questions. It’s the leading example of knowledge-graph-augmented retrieval. (Microsoft contrasts it with “naive” semantic search — their loaded word for standard RAG.)

10.3 Lever 3: Matching — better query-to-document scoring

The retrieval step itself can be smarter than “compress each chunk to one vector and compare.”

• ColBERT introduced late interaction: instead of one vector per chunk, keep a vector per token, and score a query against a document with MaxSim — for each query token, find its best-matching document token, and sum. It’s far more fine-grained than single-vector similarity, and because documents are still encoded independently, their token vectors can be precomputed offline. It captures nuance that a single averaged vector loses.
• ColPali is the technique that finally handles images, tables, charts, and figures properly — and it does so by not extracting text at all. This is the answer to the perennial question “how do I get tables and diagrams into RAG?” The traditional approach runs OCR and layout detection to turn a page into text, a process the ColPali authors rightly call “lengthy and brittle” — it mangles multi-column layouts, loses table structure, and throws away charts entirely. ColPali instead feeds the page image to a vision-language model and produces multi-vector embeddings directly from the pixels, matched with ColBERT-style late interaction. A chart, a table, and a paragraph are all just parts of an image the model understands. No OCR pipeline to break. For document-heavy RAG — financial filings, scientific papers, slide decks — this is one of the most practically important ideas on the frontier.

10.4 Lever 4: Chunk quality — fix the units before you retrieve them

The cheapest, most broadly applicable frontier idea attacks the chunking problem from §3. When you split a document, each chunk loses the context of the whole — a chunk that says “the margin improved to 31%” doesn’t say whose margin, or when. Anthropic’s Contextual Retrieval fixes this by using a cheap model to write a one-line description of what each chunk is about in the context of its document, and prepending that to the chunk before both embedding it and indexing it for BM25 (they call the two halves “Contextual Embeddings” and “Contextual BM25”). The original technique used Claude 3 Haiku for that step; today you’d reach for a current small model like Claude Haiku 4.5, and prompt caching makes generating context for every chunk cheap.

The reported results are a clean illustration of stacking the §3 building blocks. Anthropic reports the top-20 retrieval failure rate dropped by 35% from contextual embeddings alone, 49% when adding contextual BM25 (there’s hybrid search again), and 67% when adding a reranker on top (and there’s reranking). Those are Anthropic’s own internal numbers, so treat the exact figures as directional — but the lesson is solid, and it’s the same lesson from §3 made concrete: fix your chunks, add hybrid search, add a reranker. In that order.

11. Putting It Together

That’s a lot of techniques. The good news is they compose, and there’s a sane order to reach for them. Map them back to the four levers:

And a cheapest-first playbook, because reaching for GraphRAG when your real problem is bad chunking is a common and expensive mistake:

1. Get the basics right first. Sensible recursive chunking, a decent embedding model, and measure retrieval quality (this is where evaluation — Chapter 6 — earns its keep; you can’t improve what you don’t measure).
2. Fix chunks and ranking. Add contextual retrieval and a reranker. Add hybrid search (BM25 + vector + RRF). This trio resolves a large share of “our RAG is bad” complaints.
3. Make retrieval smarter. Add query rewriting, then agentic/iterative retrieval, when single-shot retrieval demonstrably isn’t finding the right things.
4. Reach for the heavy tools only when the problem demands them. GraphRAG and RAPTOR when questions genuinely require cross-document synthesis; ColPali when your value is locked in tables, charts, and scanned pages.

The meta-point, and the reason this chapter lives next to memory and the agent harness rather than off in an information-retrieval appendix: RAG has stopped being a fixed pipeline and become a set of retrieval policies an agent invokes at run time. The hyperscalers renamed their products “agentic retrieval” for a reason. Once an agent can decide whether to retrieve, how to phrase the search, whether the results are good enough, and whether to try again, retrieval is no longer a preprocessing step bolted in front of the model. It’s one of the agent’s tools — and grounding the agent in real, current, sourced knowledge is what that tool is for.

Summary: What to Internalize

• RAG closes three gaps a bare model can’t: private data, freshness, and attribution. Before answering, retrieve the relevant text and put it in the prompt. The model supplies language and reasoning; your data supplies the truth.
• The pipeline is six stages — ingest, chunk, embed, index, retrieve, augment+generate — split into an offline indexing phase and an online query phase. The index is a living copy of your data; stale retrieval is a slow hallucination.
• Four building blocks carry the weight: embeddings (meaning as geometry), chunking (the underrated, highest-leverage knob), indexing (HNSW/IVF ANN search), and search (dense + sparse hybrid, then a cross-encoder reranker).
• Structured data is in scope — you query it, not embed it. Metadata filtering is table-stakes; natural-language-to-SQL (AWS Bedrock, Azure Fabric, Databricks Genie, LlamaIndex) grounds answers on database rows; and deciding between them is the routing job agentic RAG does.
• Ingestion is where RAG usually fails, not the embedder. Real data is tables, charts, PowerPoint, and scanned PDFs — use layout-aware parsers (Docling, LlamaParse, Unstructured) and normalize everything to Markdown before chunking. A great embedder on garbled text loses to a plain one on clean text.
• Sound and vision are here — with caveats. Audio is solved by transcribe-then-RAG (diarization + word timestamps; CLAP for non-speech sound). Video is arriving as products (Twelve Labs’ video-native search) or a frame+transcript fusion pipeline; true long-video RAG is still research.
• RAG, long context, and fine-tuning are collaborators, not rivals. Fine-tune to change behavior, RAG to change knowledge, long context to hold the current working set.
• All three hyperscalers ship managed RAG and are pivoting to “agentic retrieval.” Watch the statuses (Azure OpenAI “On Your Data” retires Oct 14 2026; S3 Vectors is preview) and expect the branding to keep sliding from “RAG” to “agents.”
• You assemble a stack, rarely a single product — an embedder, an index, a reranker, a parser. The startup field is strong at each layer and consolidating fast (Ragie’s shutdown is the tell).
• The frontier makes retrieval dynamic, along four levers: chunk quality (Contextual Retrieval), matching (ColBERT, and ColPali for visual documents with no OCR), control flow (Agentic RAG, Self-RAG, CRAG, query rewriting), and corpus structure (RAPTOR, GraphRAG).
• Fix things cheapest-first: chunking and reranking before agentic loops, agentic loops before GraphRAG. Reaching for the exotic tool when a plain one would do is the most common way RAG projects waste effort.

The next chapter turns from what an agent retrieves to how it connects to the outside world at all: the Model Context Protocol, the emerging standard for wiring agents to tools, data sources, and services.