Large language models have demonstrated remarkable capabilities in generating fluent, coherent text. Yet they suffer from a fundamental limitation: their knowledge is frozen at training time, and they cannot reliably access or cite specific sources. Retrieval-Augmented Generation (RAG) addresses this by combining the generative power of neural networks with the precision of information retrieval systems.

not run yet

RAG Retrieval Simulation

Visualize how semantic similarity finds relevant documents

import numpy as np
import matplotlib.pyplot as plt
 
# Simulate document embeddings in 2D for visualization
np.random.seed(42)
 
# Create document clusters (different topics)
topics = {
    'Machine Learning': np.random.randn(5, 2) + [2, 3],
    'Web Development': np.random.randn(5, 2) + [-2, 2],
    'Database Systems': np.random.randn(5, 2) + [0, -2],
    'Cloud Computing': np.random.randn(5, 2) + [3, -1],
}
 
# Query embedding (looking for ML content)
query = np.array([2.5, 3.5])
 
fig, ax = plt.subplots(figsize=(10, 8))
 
colors = ['C0', 'C1', 'C2', 'C3']
for i, (topic, docs) in enumerate(topics.items()):
    ax.scatter(docs[:, 0], docs[:, 1], c=colors[i], label=topic, s=100, alpha=0.7)
 
# Plot query
ax.scatter(query[0], query[1], c=plt.rcParams['text.color'], marker='*', s=300,
           edgecolors=plt.rcParams['axes.facecolor'], linewidths=2, label='Query', zorder=5)
 
# Draw lines to nearest documents
all_docs = np.vstack(list(topics.values()))
distances = np.linalg.norm(all_docs - query, axis=1)
top_k = 3
nearest_indices = np.argsort(distances)[:top_k]
 
for idx in nearest_indices:
    ax.plot([query[0], all_docs[idx, 0]], [query[1], all_docs[idx, 1]],
            'w--', alpha=0.5, linewidth=2)
    ax.scatter(all_docs[idx, 0], all_docs[idx, 1], c='white', s=150,
               facecolors='none', linewidths=2)
 
ax.set_title('RAG Document Retrieval: Finding Relevant Chunks')
ax.set_xlabel('Embedding Dimension 1')
ax.set_ylabel('Embedding Dimension 2')
ax.legend(loc='upper left')
ax.grid(True, alpha=0.3)
 
plt.tight_layout()
plt.show()
 
print(f"Query retrieved {top_k} most similar documents (shown with circles)")
print("In practice, these embeddings have 768+ dimensions!")

python loads on first run, about 18 mb, cached by your browser

The Knowledge Problem in Language Models

Traditional language models encode knowledge implicitly in their parameters during training. This creates several challenges:

Challenge Description Impact
Knowledge Cutoff Parameters reflect training data date Cannot answer about recent events
Hallucination Model generates plausible but false information Unreliable for factual queries
Opacity No clear source attribution Difficult to verify claims
Update Cost Retraining required for new knowledge Expensive and time-consuming

The foundational RAG paper by Lewis et al. (2020) introduced an elegant solution: rather than storing all knowledge in model parameters, retrieve relevant documents at inference time and condition generation on this retrieved context.

"RAG models combine the best of both worlds: the parametric memory of pre-trained seq2seq models and the non-parametric memory of retrieval-based models, accessing knowledge from an external corpus." - Lewis et al., 2020

RAG Architecture Overview

The RAG architecture consists of three primary components working in concert:

RAG system architecture

  1. User query
  2. Query encoderA bi-encoder, E_q, turns the query into a dense vector
  3. Dense retrievalMaximum inner product search (MIPS)
    Document indexFAISS, ScaNN, Pinecone, etc.
  4. Fusion moduleCombines the query with the retrieved docs
  5. GeneratorAn LLM decoder: a seq2seq model such as BART, T5, or GPT
  6. Generated answer

The Retrieval Score

The core retrieval mechanism uses dense vector similarity. Given a query qq and document dd, the retrieval score is computed as the inner product of their embeddings:

s(q,d)=Eq(q)TEd(d)s(q, d) = E_q(q)^T E_d(d)

Where:

  • EqE_q is the query encoder
  • EdE_d is the document encoder
  • Both encoders map text to dense vectors in the same embedding space

This formulation enables efficient Maximum Inner Product Search (MIPS) over millions of documents using approximate nearest neighbor algorithms.

Dense Retrieval: The Bi-Encoder Architecture

How Bi-Encoders Work

The bi-encoder architecture processes queries and documents independently through separate (or shared) transformer encoders:

Bi-encoder architecture

Query

  1. Query text“What is the capital of France?”
  2. Query encoderE_q, a BERT-based transformer
  3. Query embedding768-dim dense vector
    q = [0.2, ...]

Document

  1. Document text“Paris is the capital and largest city of France...”
  2. Document encoderE_d, a BERT-based transformer
  3. Doc embedding768-dim dense vector
    d = [0.3, ...]
Similarity
s(q, d) = q^T d = 0.87
Query and document are encoded independently; their vectors meet only in the similarity score.

Training Dense Retrievers

Dense retrievers are typically trained with contrastive learning. For each query qq, we have:

  • One positive document d+d^+ (relevant)
  • Multiple negative documents d1−,d2−,...,dn−d^-_1, d^-_2, ..., d^-_n (irrelevant)

The training objective maximizes the score for positive pairs while minimizing it for negatives. The loss function uses the negative log-likelihood of the positive document:

L=−log⁡exp⁡(s(q,d+))exp⁡(s(q,d+))+∑i=1nexp⁡(s(q,di−))\mathcal{L} = -\log \frac{\exp(s(q, d^+))}{\exp(s(q, d^+)) + \sum_{i=1}^{n} \exp(s(q, d^-_i))}

This is equivalent to the cross-entropy loss over a softmax distribution of document scores.

Hard Negative Mining

The quality of negative examples significantly impacts retrieval performance. Strategies include:

Strategy Description Effectiveness
Random Negatives Sample random documents from corpus Baseline
BM25 Negatives Top BM25 results that aren't relevant Good
In-Batch Negatives Other queries' positives in same batch Efficient
Hard Negatives High-scoring but incorrect documents Best

Document Chunking Strategies

Real-world documents are often too long for transformer context windows. Effective chunking is critical for RAG performance.

Common Chunking Approaches

Document chunking strategies

  1. Original document10,000 tokens
  2. Fixed-sizeSimple, but may break semantic units
    • Chunk 1: tokens [0:512]
    • Chunk 2: tokens [512:1024]
    • Chunk 3: tokens [1024:1536]
    Sentence-basedPreserves sentence boundaries
    • Chunk 1: sentences 1 to 5 (~400 tokens)
    • Chunk 2: sentences 6 to 12 (~500 tokens)
    SemanticPreserves document structure
    • Chunk 1: “Introduction” section
    • Chunk 2: “Methods” section
    • Chunk 3: “Results” section
    RecursiveHierarchical, respects structure
    • Split on paragraphs
    • If too large, split on sentences
    • If still too large, split on tokens

Chunk Size Trade-offs

Chunk Size Advantages Disadvantages
Small (100-200 tokens) Precise retrieval, fine-grained May lose context
Medium (300-500 tokens) Balanced General purpose
Large (500-1000 tokens) More context per chunk Less precise retrieval

Overlapping Chunks

To prevent information loss at chunk boundaries, many systems use overlapping windows:

Overlapping chunk strategy

Document
Chunk 1
Chunk 2
Chunk 3
Chunk 4

Each row is one chunk of the document. Dark cells are overlaps, typically 50 to 100 tokens, shared by two neighboring chunks.

The overlap ensures that information spanning chunk boundaries is captured in at least one chunk.

The RAG Probability Distribution

The mathematical formulation of RAG defines a marginalized probability over retrieved documents. For input xx (query) and output yy (generated answer):

p(y∣x)=∑z∈top-kp(z∣x)p(y∣x,z)p(y|x) = \sum_{z \in \text{top-k}} p(z|x) p(y|x,z)

Where:

  • zz represents a retrieved document
  • p(z∣x)p(z|x) is the retrieval probability (from dense retrieval scores)
  • p(y∣x,z)p(y|x,z) is the generation probability conditioned on query and document

RAG-Sequence vs RAG-Token

The Lewis et al. paper introduced two variants:

RAG-Sequence: Uses the same document for generating the entire sequence

pRAG-Sequence(y∣x)=∑z∈top-kp(z∣x)∏i=1Np(yi∣x,z,y1:i−1)p_{\text{RAG-Sequence}}(y|x) = \sum_{z \in \text{top-k}} p(z|x) \prod_{i=1}^{N} p(y_i|x, z, y_{1:i-1})

RAG-Token: Can use different documents for each output token

pRAG-Token(y∣x)=∏i=1N∑z∈top-kp(z∣x)p(yi∣x,z,y1:i−1)p_{\text{RAG-Token}}(y|x) = \prod_{i=1}^{N} \sum_{z \in \text{top-k}} p(z|x) p(y_i|x, z, y_{1:i-1})

RAG-Sequence vs RAG-Token

RAG-Sequence

  1. Query
  2. RetrieveDoc A, Doc B, Doc C
  3. Doc A“Paris is the capital...”
    Doc B“The capital of France...”
    Doc C“France's capital city...”
    One complete answer per document
  4. MarginalizeWeighted sum of complete sequences

RAG-Token

  1. Query
  2. RetrieveDoc A, Doc B, Doc C
  3. Token 1“The”
  4. Token 2“capital”
  5. Token 3“is”
RAG-Sequence marginalizes over complete sequences. RAG-Token marginalizes at every token, so each token can draw from a different document.

Fusion Mechanisms

Fusion mechanisms determine how retrieved documents are combined with the query for generation.

Concatenation-Based Fusion

The simplest approach concatenates retrieved documents with the query:

Concatenation fusion

  1. Query
    Doc 1
    Doc 2
    Doc 3
  2. Input to generator
    [QUERY] What is photosynthesis?
[DOC 1] Photosynthesis is a process used by plants to convert light energy into chemical energy...
[DOC 2] The light-dependent reactions occur in the thylakoid membranes and require sunlight...
[DOC 3] Carbon fixation occurs in the Calvin cycle, where CO2 is converted into glucose...
  3. GeneratorProduces an answer conditioned on the full context

Cross-Attention Fusion

More sophisticated approaches use cross-attention between query and documents:

Cross-attention fusion

  1. Query tokens
  2. Self-attention
  3. Cross-attentionQuery attends to documents
    Document representations
  4. Feed-forward
  5. Output tokens

Fusion-in-Decoder (FiD)

The Fusion-in-Decoder approach, introduced for open-domain QA, processes each document independently with the query, then fuses representations in the decoder:

Fusion-in-Decoder architecture

  1. Query + Doc 1
  2. Encoder
  3. Representation 1
  1. Query + Doc 2
  2. Encoder
  3. Representation 2
  1. Query + Doc k
  2. Encoder
  3. Representation k
DecoderAttends to all k representations and generates the answer
Each document is encoded together with the query on its own; the representations are fused only in the decoder.

This allows the model to process many documents efficiently while still attending to all of them during generation.

Training RAG Systems

End-to-End Training

The original RAG approach trains the retriever and generator jointly. The total loss is:

LRAG=−log⁡p(y∣x)=−log⁡∑z∈top-kp(z∣x)p(y∣x,z)\mathcal{L}_{\text{RAG}} = -\log p(y|x) = -\log \sum_{z \in \text{top-k}} p(z|x) p(y|x,z)

However, this is computationally expensive because:

  1. Retrieval requires searching the entire corpus
  2. Gradients must flow through the retrieval operation
  3. Document embeddings must be periodically updated

Practical Training Strategies

Strategy Description Pros Cons
Frozen Retriever Train only generator Fast, stable Suboptimal retrieval
Separate Training Train retriever, then generator Modular No joint optimization
Periodic Updates Update doc embeddings periodically Balanced Complexity
End-to-End Joint training Optimal Expensive

The Comprehensive Survey Perspective

According to the 2023 survey on RAG for Large Language Models, the field has evolved significantly since the original formulation:

"RAG has evolved from a simple retrieve-then-read pipeline to sophisticated systems incorporating query rewriting, iterative retrieval, and self-reflection mechanisms."

The survey identifies three paradigms:

  1. Naive RAG: Simple retrieve-then-generate pipeline
  2. Advanced RAG: Incorporates pre-retrieval optimization (query rewriting) and post-retrieval processing (re-ranking)
  3. Modular RAG: Flexible composition of retrieval, generation, and reasoning modules

Advanced RAG Techniques

Query Expansion and Rewriting

Poor queries lead to poor retrieval. Query expansion improves retrieval by reformulating the original query:

Query rewriting pipeline

  1. Original query
    Why is the sky blue?
  2. Query rewriterLLM-based
  3. Expanded queries
    • “Rayleigh scattering atmosphere blue light”
    • “Physics of sky color wavelength”
    • “Why does the sky appear blue during daytime”
  4. Retrieve with all queries
  5. Merge and re-rank results

Hypothetical Document Embeddings (HyDE)

HyDE generates a hypothetical answer first, then uses it for retrieval:

HyDE process

  1. Query
    What causes aurora borealis?
  2. LLMGenerates a hypothetical answer
  3. Hypothetical answer
    The aurora borealis is caused by charged particles from the sun interacting with gases in Earth's atmosphere, creating colorful light displays...
  4. Embed the hypothetical document
  5. Retrieve similar real documents

Iterative Retrieval

Complex queries may require multiple retrieval steps:

Iterative retrieval

  1. Query
    How did Einstein's work influence quantum computing?
  2. Round 1Retrieve docs about Einstein's contributions
    • Learn about the photoelectric effect and the EPR paradox
  3. Round 2Retrieve docs about quantum computing foundations
    • Learn about qubits and entanglement
  4. Round 3Retrieve docs connecting these concepts
    • Find connections between EPR and quantum gates
  5. SynthesizeCombine information from all rounds

Evaluation Metrics

Retrieval Metrics

Metric Formula Description
Recall@k Relevant in top-k / Total relevant Coverage of relevant docs
Precision@k Relevant in top-k / k Accuracy of top-k
MRR Mean of 1/rank of first relevant Rank of first hit
NDCG Normalized discounted cumulative gain Rank-weighted relevance

End-to-End Metrics

Metric Application Notes
Exact Match QA tasks Strict string matching
F1 Score QA tasks Token overlap
BLEU/ROUGE Generation N-gram overlap
Faithfulness Attribution Does answer match sources?

Practical Implementation Considerations

Vector Database Selection

Database Scalability Speed Features
FAISS Billions Very fast Open source, GPU
Pinecone Billions Fast Managed, easy API
Weaviate Millions Fast Hybrid search
Milvus Billions Fast Open source
Chroma Millions Moderate Simple, local-first
Qdrant Billions Fast Rust-based, filtering

Latency Breakdown

A typical RAG query involves multiple steps:

RAG latency analysis

Query encoding~10 to 50 ms, transformer forward pass50 ms
Vector search~10 to 100 ms, depends on index size100 ms
Document fetch~5 to 50 ms, database read50 ms
Context assembly~1 to 5 ms, string concatenation5 ms
LLM generation~500 to 2000 ms, depends on model and length2,000 ms
Bars show the upper end of each range. A full query totals about 550 to 2200 ms.

Optimization targets:

  • Caching frequent queries
  • Smaller embedding models
  • Approximate search algorithms
  • Streaming generation

Challenges and Future Directions

Current Limitations

  1. Attribution Accuracy: Retrieved documents may not actually support generated claims
  2. Retrieval Failures: Dense retrieval struggles with certain query types
  3. Context Length: Limited ability to use many retrieved documents
  4. Latency: Additional retrieval step adds latency
  5. Maintenance: Document indices require updates as knowledge changes

Emerging Solutions

The 2023 RAG survey highlights several promising directions:

  • Self-RAG: Models that decide when retrieval is needed
  • Corrective RAG: Self-correction mechanisms for retrieval errors
  • Graph RAG: Using knowledge graphs alongside dense retrieval
  • Long-Context RAG: Leveraging extended context windows (100k+ tokens)

Conclusion

Retrieval-Augmented Generation represents a fundamental shift in how we build knowledge-intensive AI systems. By combining the fluency of neural text generation with the precision of information retrieval, RAG systems can provide accurate, attributable, and up-to-date responses.

The architecture's elegance lies in its modularity: the retriever and generator can be independently improved, and the knowledge base can be updated without retraining the model. As the comprehensive 2023 survey notes, RAG has evolved from a research technique to a foundational pattern for production AI systems.

The mathematical framework - from the bi-encoder retrieval score s(q,d)=Eq(q)TEd(d)s(q, d) = E_q(q)^T E_d(d) to the marginalized generation probability p(y∣x)=∑zp(z∣x)p(y∣x,z)p(y|x) = \sum_{z} p(z|x) p(y|x,z) - provides a principled foundation for continued innovation. As language models grow more capable and retrieval systems more efficient, the synergy between these components will only deepen.

Check your understanding1 of 5
What is the main purpose of RAG?

This article draws on peer-reviewed research including the foundational RAG paper by Lewis et al. (2020) and the comprehensive 2023 survey on RAG systems. For complete technical details, consult the original publications.

Sources2 linked references
  1. 1.foundational RAG paper by Lewis et al. (2020)
  2. 2.2023 survey on RAG for Large Language Models

Found this useful? Pass it on.