1. Introduction to Retrieval Augmented Generation (RAG) [0]

thumbnail RAG (Retrieval Augmented Generation) combines custom data with Large Language Models (LLMs) to provide answers grounded in specific information.

1.1. The Motivation for RAG [0]

  1. LLMs are trained on public data, but most of the world's data is private. LLMs are trained on vast amounts of publicly available information, but this doesn't include private or proprietary data. [6]
  2. Context windows are increasing, making it feasible to include private data. LLMs are becoming capable of processing much larger amounts of text (context windows), allowing for the inclusion of private data that the LLM has never seen before. [8]
    • Context windows have grown from around 4,000-8,000 tokens (a dozen pages) a year ago to up to a million tokens (thousands of pages) today. [9]
  3. RAG enables LLMs to access and process private data. This private data can be personal, corporate, or any other information not native to the LLM's training set. [12]
  4. RAG is crucial for LLMs acting as a new operating system. It's essential to feed external, private information into LLMs for processing. [15]

1.2. The Three General Steps of RAG [17]

  1. Indexing external data: This involves creating a searchable database from your custom data. [18]
    • This can include various database types like SQL, relational databases, or vector stores. [21]
  2. Retrieval: Relevant documents are retrieved based on a user's input, like a question. [22]
  3. Generation: The retrieved documents are passed to an LLM, which then produces answers based on that information. [22]
    • RAG powerfully combines the knowledge and processing capabilities of LLMs with large-scale private data sources. [23]

2. The RAG Landscape: Key Techniques and Components [24]

thumbnail The RAG process involves several stages, from understanding the user's query to generating a final answer, with advanced techniques to improve each step.

2.1. Query Translation [27]

  1. Query Translation: This stage modifies a user's question to make it more suitable for retrieval. [27]
    • Query Writing: Rephrasing the question. [29]
    • Query Decomposition: Breaking a question into smaller sub-questions. [29]
  2. Routing: Directs the translated question to the correct data source. [30]
    • This is important when dealing with multiple data sources like vector stores, relational databases, or graph databases. [31]
  3. Query Construction: Converts natural language into a Domain-Specific Language (DSL) for a data source. [33]
    • Examples include text-to-SQL for relational databases, text-to-Cypher for graph databases, and text-to-metadata filters for vector databases. [34]

2.2. Indexing and Retrieval [36]

  1. Indexing: Processes documents to make them easily retrievable. [36]
    • This involves different embedding methods and indexing strategies. [37]
  2. Retrieval: After indexing, documents are retrieved. [39]
    • Techniques exist to rerank or filter these retrieved documents. [39]
  3. Active RAG: An advanced method involving feedback loops in the retrieval or generation stages. [40]
    • This includes grading documents and answers for relevance and faithfulness (checking for hallucinations). [41]
    • If grading fails, the system can retrieve again or regenerate the answer. [43]

2.3. Overall Course Outline [46]

  1. Basics: The course will start with the fundamental concepts of indexing, retrieval, and generation. [47]
  2. Advanced Techniques: It will then cover more complex methods like query translation, routing, and query construction. [47]

3. Indexing: Creating Searchable Data [48]

thumbnail

3.1. The Purpose of Indexing [49]

  1. Loading and Retrieving Documents: The first step in indexing is loading external documents that will be used by a retriever. [52]
  2. Establishing Relevance: The retriever's goal is to find documents related to a given question. [53]
  3. Numerical Representation: Relevance is typically established using numerical representations of documents, as comparing numbers (vectors) is easier than comparing free-form text. [54]
  4. Compression Methods: Various methods compress text documents into numerical representations for easy searching. [56]

3.2. Types of Numerical Representations [58]

  1. Sparse Vectors:
    • Developed by Google and others, these methods use word frequency to build sparse vectors. [58]
    • The vector locations correspond to a large vocabulary of possible words, and each value represents the count of that word. [58]
    • These vectors are "sparse" because most entries are zero, given the large vocabulary relative to the document's content. [59]
    • Effective search methods exist for this type of representation. [60]
  2. Embeddings (Machine Learned):
    • More recent methods use machine learning to create compressed, fixed-length vectors for documents. [61]
    • Documents are typically split into smaller chunks because embedding models have limited context windows (e.g., 512 to 8,000 tokens). [62]
    • Each chunk is compressed into a vector that captures its semantic meaning. [63]
    • These vectors are then indexed for efficient searching. [64]

3.3. How Embeddings Facilitate Retrieval [65]

  1. Embedding Questions: Questions can be embedded using the same method as documents. [65]
  2. Similarity Search: Numerical comparisons (similarity searches) are performed on these vectors to find relevant documents. [65]

3.4. Code Walkthrough: Computing Embeddings [67]

  1. Setup: The example uses a notebook with installed packages and API keys for LangSmith (useful for tracing). [68]
  2. Token Count: The number of tokens in a question is computed to understand its size, as LLMs and embedding models operate on tokens. [72]
  3. OpenAI Embeddings:
    • An OpenAI embedding model is specified. [77]
    • The embed_query function is used to embed a question and a document. [77]
    • The output is a fixed-length vector (e.g., length 1536) that encodes the text's semantics. [78]
  4. Cosine Similarity: This can be used to compare the embeddings of the question and documents. [81]
  5. Indexing Process:
    • Documents are loaded, split, and embedded. [82]
    • Each split is embedded into a vector and stored in a vector store along with a link to the original document. [84]

4. Retrieval: Finding Relevant Information [86]

thumbnail

4.1. The Retrieval Process [87]

  1. Overview: Retrieval is the process of finding documents relevant to a question after indexing. [87]
  2. Indexing Recap: Documents are split, embedded into searchable numerical representations (vectors), and stored in an index. [89]
  3. Similarity Search: When a question is embedded, the index performs a similarity search to return relevant document splits. [91]
  4. Semantic Meaning in Space:
    • Documents are embedded into a high-dimensional space where their location reflects their semantic meaning. [92]
    • Documents in similar locations in this space contain similar semantic information. [95]
    • This principle is fundamental to modern vector store search methods. [96]

4.2. How Similarity Search Works [97]

  1. Embedding and Projection: Documents are embedded into a high-dimensional space (e.g., a toy 3D space), and questions are embedded into the same space. [97]
  2. Neighborhood Search: A search is performed around the question's vector in this space to find nearby documents. [99]
  3. Retrieval of Neighbors: These nearby documents are retrieved because they have similar semantics to the question. [101]
  4. Selecting Documents: A specific number (K) of nearby documents can be chosen for retrieval. [104]

4.3. LangChain Integrations [105]

  1. Rich Ecosystem: LangChain offers many integrations for embedding models, indexes, document loaders, and splitters, allowing for flexible experimentation. [106]

4.4. Code Walkthrough: Implementing Retrieval [108]

  1. Setup: The notebook environment is set up with necessary packages and environment variables for LangSmith. [109]
  2. Reloading Documents: Documents are reloaded and resplit. [113]
  3. Building the Index: The index is built, embedding each split and storing it. [115]
  4. The 'K' Parameter: The parameter 'K' determines the number of nearest neighbors to fetch during retrieval. [117]
  5. Performing Retrieval:
    • A question is asked (e.g., "What is task decomposition?"). [122]
    • The get_relevant_documents function is called, returning K documents (in this case, 1 document because K=1). [122]
  6. LangSmith Trace: The LangSmith trace shows the question and the retrieved document, confirming its relevance to task decomposition. [124]
  7. KNN Implementation: This demonstrates how to easily implement K-Nearest Neighbors (KNN) search with a few lines of code. [127]

5. Generation: Creating Answers [129]

thumbnail

5.1. The Generation Process [130]

  1. Basic RAG Flow: The process involves indexing, retrieval, and then generation of an answer based on retrieved documents. [133]
  2. Context Window Stuffing: Retrieved documents are placed into the LLM's context window for answer generation. [135]
  3. Embedding and Searching: Documents are split, embedded, and stored in a vector store. Questions are also embedded. Similarity search (like KNN) finds relevant documents based on their proximity in the embedding space. [136]
  4. Prompting: Relevant document splits are packed into the context window, and an answer is produced. [140]

5.2. Prompts and Chains [142]

  1. Prompt Templates: These act as placeholders for context and questions, allowing retrieved documents and the user's question to be slotted in. [143]
  2. Building a Prompt: Retrieved documents and the question are used to populate a prompt template. [147]
  3. LLM Invocation: The populated prompt is passed to an LLM (e.g., GPT-3.5) to generate chat messages, which are then parsed into a string answer. [148]
  4. Code Walkthrough:
    • A retriever is built by loading, splitting, embedding, and indexing documents. [155]
    • A prompt template is defined with context and question variables. [164]
    • An LLM (GPT-3.5) is defined. [171]
    • LangChain Expression Language (LCEL) is used to compose the prompt and LLM into a chain. [174]
    • The chain is invoked with a dictionary containing the context and question. [178]
    • The output is the generated answer. [180]
  5. LangSmith Trace: The trace shows the prompt, context, question, and the final answer. [181]
  6. RAG Chain: A pre-built RAG chain can automate the retrieval process, taking a retriever and a question as input. [191]
    • This chain automatically triggers the retriever, passes documents to the context, and then uses the prompt and LLM to generate the answer. [194]
  7. Trace of RAG Chain: The trace confirms the retriever was run, documents were retrieved, and the LLM generated the final answer. [199]

5.3. Future Topics [201]

  1. Addressing Limitations: Follow-up videos will delve into more complex themes that address limitations in this basic RAG pipeline. [202]

6. Query Translation: Improving Retrieval [203]

thumbnail

6.1. The Need for Query Translation [204]

  1. Purpose: Query translation modifies a user's input question to improve retrieval accuracy. [206]
  2. Problem Statement: User queries can be ambiguous or poorly written, leading to incorrect document retrieval due to semantic similarity search. [207]

6.2. Approaches to Query Translation [209]

  1. Query Rewriting: Reframing the question from a different perspective. [211]
    • Multi-Query: Generating multiple versions of the question. [212]
    • RAG Fusion: A technique that combines multiple queries with ranking. [212]
  2. Decomposition into Sub-questions: Breaking down a complex question into simpler, solvable sub-questions. [214]
    • Example: "Least to Most" prompting. [215]
  3. Making Questions More Abstract (Step-Back Prompting): Asking a higher-level, more general question as a prerequisite to answering the user's specific question. [216]

6.3. Multi-Query Approach [219]

  1. Intuition: Rewording a question from different perspectives increases the likelihood that at least one version will align well with a relevant document in the embedding space. [219]
  2. Shotgun Approach: Fanning out the question into multiple perspectives aims to increase retrieval reliability. [222]
  3. Combined Retrieval: Multiple generated questions can be used for retrieval, and their results can be combined. [225]

6.4. Code Walkthrough: Multi-Query [228]

  1. Setup: Notebook with package installations and LangSmith API keys. [229]
  2. Indexing: A blog post on agents is loaded, split, and indexed in a local Chroma vector store. [232]
  3. Multi-Query Prompt: A prompt is defined to instruct the LLM to reframe a question into multiple sub-questions. [239]
  4. Query Generation Chain: The prompt is passed to an LLM, and the output is parsed into a list of questions. [240]
  5. Applying Multi-Query:
    • A sample input question is provided. [244]
    • The generate_queries chain creates a list of rephrased questions. [245]
    • Each rephrased question is used to retrieve documents independently. [247]
    • A function collects the unique union of documents from all retrievals. [248]
  6. LangSmith Trace:
    • The trace shows the initial prompt and the generated set of rephrased questions. [254]
    • It also shows the independent retrieval performed for each rephrased question. [255]
  7. End-to-End RAG Chain: The retrieved documents are passed to a final RAG prompt, along with the original question, to generate an answer. [259]
  8. Final Output: The process generates an answer based on the combined retrieved documents from the multiple queries. [275]

6.5. RAG Fusion [279]

  1. Concept: RAG Fusion is similar to multi-query but adds a reciprocal rank fusion step to rank retrieved documents. [285]
  2. Process:
    • A question is broken into multiple differently worded questions. [289]
    • Retrieval is performed for each question. [289]
    • Reciprocal Rank Fusion ranks the combined results from these retrievals. [289]
  3. Code Walkthrough:
    • The prompt for generating multiple search queries is similar to multi-query. [296]
    • A query generation chain is created. [300]
    • Retrieval is performed for each generated question, resulting in a list of lists of documents. [306]
    • Reciprocal Rank Fusion is applied to consolidate and rank these documents. [311]
  4. LangSmith Trace:
    • The trace shows the generated search queries and the individual retrievals. [316]
    • It then shows the ranked list of unique documents after fusion. [320]
  5. Full RAG Chain: The ranked documents are passed to a RAG prompt and LLM to generate the final answer. [322]
  6. Use Cases: Useful for querying across different vector stores or when using many differently worded questions. [327]

6.6. Decomposition [332]

  1. Concept: Decomposing an input question into a set of sub-questions that can be solved individually. [333]
  2. Sequential Problem Solving:
    • Example: "Think machine learning" is decomposed into "think," "think machine," and "think machine learning." [339]
    • Each sub-problem is solved sequentially, building upon previous answers. [341]
  3. Interleaved Retrieval with Chain of Thought (IRC): Combines retrieval with Chain of Thought reasoning to solve sub-problems dynamically. [347]
  4. Code Walkthrough:
    • A retriever is already defined. [352]
    • A prompt is created to decompose an input question into sub-problems. [354]
    • A sample question about LM-powered autonomous agents is used. [357]
    • The decomposed questions are: "What is LM technology?", "How does it work?", "What are the components?", and "How do the components interact?". [358]
    • A chain is defined to answer each question individually, using retrieval and prior answers as context. [359]
  5. LangSmith Trace:
    • The trace shows how each question is answered sequentially, incorporating answers from previous questions as context. [368]
    • For question two, the answer and context from question one are included in the prompt. [372]
    • For question three, answers from questions one and two are included. [376]
  6. Parallel Answering: An alternative approach answers sub-questions in parallel and concatenates the results, suitable when answers don't depend on each other. [380]

6.7. Step-Back Prompting [389]

  1. Concept: This approach asks a more abstract, higher-level question related to the original query to improve retrieval. [391]
  2. Method: Uses few-shot prompting with examples to generate a "step-back" question. [399]
    • Example: Original question about a specific country's region leads to a more abstract question about the individual's personal history. [406]
  3. Code Walkthrough:
    • A prompt is formulated using few-shot examples from a paper. [415]
    • The prompt instructs the LLM to generate a more generic, step-back question. [418]
    • Example: "What is task composition for LLM agents?" becomes "What is the process of task composition in general?". [422]
  4. Combined Retrieval: The original question and the step-back question are both used for retrieval. [427]
  5. LangSmith Trace:
    • The trace shows the generated step-back question ("What is the process of task composition?"). [436]
    • Retrieval is performed for both the original and step-back questions. [438]
    • The retrieved context from both is combined and used to answer the original question. [441]
  6. Application: Useful in domains with conceptual knowledge underpinning user questions, potentially improving retrieval by formulating a higher-level query. [444]

6.8. HiDE (Hypothetical Document Embeddings) [449]

  1. Concept: Translates raw questions into hypothetical documents, mapping them into "document space" to improve retrieval. [451]
  2. Intuition: Hypothetical documents generated by an LLM might be closer in the embedding space to desired retrieval documents than the original, short, potentially ill-worded question. [459]
  3. Code Walkthrough:
    • A prompt is defined to generate a "paper passage" (hypothetical document) to answer a given question. [465]
    • An LLM (OpenAI's GPT) generates a hypothetical document based on the question. [467]
    • This hypothetical document is then embedded and used to retrieve relevant chunks from the vector store. [471]
    • The retrieved documents are passed to a RAG chain to generate the final answer. [475]
  4. LangSmith Trace: The trace shows the generated hypothetical document and the subsequent retrieval of related documents. [478]
  5. Benefits: Can improve performance in certain domains and allows for tuning the document generation prompt for specific needs. [489]

7. Routing: Directing Queries to the Right Source [496]

thumbnail

7.1. The Role of Routing [497]

  1. Purpose: Routing directs a potentially modified question to the appropriate data source. [501]
  2. Multiple Data Sources: This is crucial when dealing with diverse data sources like vector stores, relational databases, and graph databases. [502]
  3. Content-Based Routing: The question's content determines which data source is best suited to answer it. [504]

7.2. Routing Methods [505]

  1. Logical Routing:
    • An LLM is given knowledge of available data sources and reasons about which one to use for a given question. [507]
    • This involves the LLM applying logic to select the appropriate data source. [508]
  2. Semantic Routing:
    • A question is embedded, and prompts representing different data sources are also embedded. [509]
    • The similarity between the question embedding and prompt embeddings determines which prompt (and thus data source) is chosen. [509]

7.3. Logical Routing with Function Calling [511]

  1. Example Scenario: Routing a question to Python docs, JS docs, or Go docs. [514]
  2. Structured Output:
    • A data model is defined to constrain the LLM's output to one of the specified data sources (e.g., DataSource with options like vector_store or sql_database). [518]
    • This data model is converted into a function schema (e.g., for OpenAI). [523]
    • The schema is bound to the LLM, enabling it to output a structured object adhering to the schema. [524]
    • A parser converts the LLM's JSON output into a structured object (e.g., a Pydantic object). [527]
  3. Code Implementation:
    • A prompt instructs the LLM to act as a router based on programming language. [537]
    • The LLM is bound to the structured output schema. [530]
    • When asked a question involving Python code, the LLM correctly outputs a RouteQuery object specifying "Python docs." [535]
  4. Building the Chain: The router's output can be used to dynamically select different chains (e.g., a retriever chain for Python docs, another for JS docs). [539]

7.4. Semantic Routing [545]

  1. Example Scenario: Routing between a physics prompt and a math prompt. [545]
  2. Process:
    • Prompts for each potential route are embedded. [548]
    • An input question is embedded. [550]
    • A runnable Lambda function computes similarity between the question embedding and prompt embeddings. [551]
    • The prompt with the highest similarity is chosen. [553]
  3. Demonstration: Asking "What is a black hole?" results in selecting the physics prompt due to semantic similarity. [554]

8. Query Construction: Translating Natural Language to DSL [564]

thumbnail

8.1. The Goal of Query Construction [565]

  1. Purpose: To convert natural language questions into a specific domain-specific language (DSL) required by data sources. [569]
  2. Previous Steps: This follows query translation (optimizing the question) and routing (selecting the data source). [565]
  3. Focus: This section specifically covers constructing mediated filters for vector stores. [570]

8.2. Mediated Filters for Vector Stores [571]

  1. Problem: Converting natural language queries with metadata filters into structured queries for vector stores. [571]
    • Example: "Find videos on chat LangChain published after 2024." [572]
  2. Function Calling: This process uses function calling (e.g., with OpenAI) to achieve the conversion. [576]
  3. Process:
    • Metadata fields available in the vector store are provided to the LLM. [578]
    • The LLM generates queries that adhere to the provided schema. [579]
    • These queries are parsed into structured objects (like Pydantic objects) for use in searches. [579]

8.3. Code Walkthrough: Query Analyzer [581]

  1. Metadata Example: Examining metadata from a YouTube video transcript (description, URL, publish date, length). [582]
  2. Index Filters: Imagining an index with filters for view count, publication date, video length, and unstructured search on content/title. [586]
  3. TutorialSearch Object: Encapsulates information about available filters and searches. [589]
  4. Prompt: A simple prompt instructs the LLM to convert natural language into database queries, referencing the TutorialSearch object. [595]
  5. Structured Output with Function Calling:
    • The TutorialSearch Pydantic object is bound to the LLM. [597]
    • This converts the object into a function schema for the LLM. [599]
    • The LLM takes a natural language question and returns a structured query object. [599]
  6. Query Analyzer Chain: A chain is set up to perform this query analysis. [601]
  7. Testing:
    • A purely semantic input ("RAG from scratch") results in content and title searches. [603]
    • A question with a date filter ("videos on chat LangChain published after 2024") correctly generates semantic search and a publish date filter. [607]
  8. General Strategy: This approach converts unstructured natural language input into a structured query object based on a provided schema, applicable to various querying needs and vector databases. [614]

9. Advanced Indexing Techniques [620]

thumbnail

9.1. Multi-Representation Indexing [621]

  1. Concept: Decouples raw documents from the unit used for retrieval, often using an LLM to create a "proposition" (a distilled summary) of document splits. [621]
  2. Goal: The proposition is optimized for retrieval, containing keywords or key ideas that help match embedded questions. [640]
  3. Implementation:
    • A summary of the document is created and embedded for retrieval. [640]
    • The raw, full document is stored independently in a separate document store. [643]
    • When a summary is retrieved, the full document is returned to the LLM for generation. [644]
  4. Benefits:
    • Allows LLMs to handle the entire document at generation time, especially useful with long context windows. [648]
    • Ensures the LLM has full context for answering questions. [650]
  5. Code Walkthrough:
    • Two blog posts are loaded. [658]
    • Summaries are created for each document. [660]
    • A vector store indexes the summaries, and a document store holds the full documents. [667]
    • A MultiVectorRetriever combines these stores. [671]
    • A query ("memory and agents") retrieves the summary related to agents. [679]
    • Using the retrieved summary and its associated ID, the full document (web page) is fetched from the document store. [681]

9.2. Raptor: Hierarchical Indexing [694]

  1. Concept: A technique for hierarchical indexing that builds a tree of document summaries. [695]
  2. Problem Addressed: Standard retrieval (like KNN) might miss information if a question requires consolidating data across many chunks or documents, exceeding the typical K parameter. [700]
  3. Process:
    • Documents are clustered, and each cluster is summarized. [707]
    • This process is repeated recursively, creating higher-level summaries until a single summary for the entire corpus is formed. [709]
    • All summaries (from leaves to the root) are indexed together, creating an abstraction hierarchy. [710]
  4. Benefit: Higher-level questions retrieve higher-level summaries, while lower-level questions retrieve more detailed chunks, providing better semantic coverage across abstraction levels. [713]
  5. Code Walkthrough (High-Level):
    • Applies the Raptor technique to a large set of LangChain Expression Language documents (around 30 documents). [726]
    • Documents are embedded, clustered, and summarized recursively. [744]
    • The raw documents and all generated summaries are indexed together in the vector store. [761]
  6. Long Context Application: Works well with long context LLMs, allowing embedding of full documents without splitting, creating summarization trees effectively. [768]

9.3. Co-bear: Token-Level Embeddings [776]

  1. Concept: An advanced embedding approach that represents each token in a document with a vector, rather than compressing the entire document into a single vector. [779]
  2. Intuition: Compressing a full document with nuances into a single vector might be overly restrictive. Co-bear aims for a more granular representation. [800]
  3. Process:
    • Documents are tokenized, and an embedding/vector is produced for each token, with positional weighting. [805]
    • The same is done for the question. [810]
    • For each token in the question, similarity is computed against all tokens in the document. The maximum similarity is taken. [811]
    • The final score is the sum of these maximum similarities across all question tokens. [814]
  4. Performance: Reports strong performance but raises questions about latency and production readiness due to its computational intensity. [816]
  5. Code Walkthrough (using Rouille library):
    • The Rouille library simplifies playing with Co-bear. [819]
    • A pre-trained model is used to mediate the process. [821]
    • A Wikipedia page about Miyazaki is fetched as a document. [835]
    • An index is created using Co-bear's indexing process. [838]
    • A rag_search function is used to retrieve results. [850]
    • The Co-bear index can be wrapped as a LangChain retriever for seamless integration. [855]
  6. Considerations: Potentially beneficial for longer documents due to its token-level approach, but requires investigating context limits. [864]

10. Building Sophisticated RAG Flows with LangGraph [868]

thumbnail

10.1. The Evolution of RAG Flows [869]

  1. Basic RAG: A linear flow starting with a question, retrieving documents, and generating an answer grounded in those documents. [870]
  2. Limitations of Basic RAG: Practical questions arise regarding when to retrieve, the quality of retrieved documents, and how to retry retrieval with improved questions. [872]
  3. Active RAG: An approach where an LLM decides when and where to retrieve based on existing retrievals or generations. [874]

10.2. State Machines and LangGraph [876]

  1. Levels of Control:
    • Base Case: LLM chooses a single step's output (e.g., generation after retrieval). [877]
    • Routing: LLM chooses between different steps (e.g., vector store vs. graph DB). [879]
    • State Machine: LLM chooses between different steps with specified transitions, enabling more sophisticated logical flows. [881]
  2. LangGraph: A framework for building state machines as graphs, allowing for diverse and complex RAG flows. [882]
    • This promotes "flow engineering," where workflows are designed and then implemented. [883]

10.3. Corrective RAG (C-RAG) Example [884]

  1. Workflow:
    • Retrieve documents. [886]
    • Grade documents for relevance. [887]
    • If at least one document is relevant, proceed to generation. [888]
    • If documents are ambiguous or incorrect, retrieve from an external source (e.g., web search). [890]
    • Pass the external source results as context for answer generation. [891]
  2. Key Idea: Reasoning about retrieved documents and using feedback loops (like web search fallback) to improve the RAG process. [892]
  3. Implementation with LangGraph: The C-RAG workflow is implemented as a state machine using LangGraph. [899]
    • Simplifications include deciding relevance, using web search as a supplement, and query writing for web search optimization. [900]

10.4. Implementing the Graph with LangGraph [902]

  1. State Definition: A graph_state dictionary holds key information passed between nodes (e.g., question, documents, generation). [911]
  2. Flow Engineering: The process involves designing the flow diagram and then implementing each node and edge as functions. [919]
    • Nodes: Functions that take the state and modify it (e.g., retrieve, grade_documents, generate). [941]
    • Edges: Conditional logic that determines the next node based on the state (e.g., route_question, decide_to_generate). [951]
  3. Key Components:
    • Router: Uses a structured LLM to direct questions to the appropriate tool (vector store, web search, or fallback). [1112]
    • Graders: Use structured output to provide binary relevance, hallucination, or answer relevance scores. [1134]
    • Retrieval: Fetches documents from a vector store. [1108]
    • Web Search: Uses a tool for external search. [1132]
    • Generation: Performs standard RAG using a prompt and LLM. [1168]
    • Fallback: A simple LLM chain for when no tools are used or other paths fail. [1175]
  4. Graph Construction: Nodes and edges are defined and connected to form the graph structure. [1245]
  5. Execution and Tracing:
    • The graph is executed with test questions. [1277]
    • Output logs show the flow through the graph (e.g., routing, grading, web search, generation). [1280]
    • LangSmith Traces: Provide detailed visibility into each node's execution, including grader outputs, query transformations, web search results, and the final prompt/answer. [1301]
  6. LangGraph vs. Agents:
    • LangGraph: Best for well-defined flows with predictable steps, offering reliability and the ability to use smaller, faster models. It acts like "guardrails" for agents. [1336]
    • Agents: More suitable for flexible workflows requiring open-ended reasoning and decision-making, often using larger models but potentially with lower reliability. [1349]
  7. Cohere Command R: A suitable model for this workflow due to its tool use, routing capabilities, small size, speed, and large context window. [1076]

10.5. Adaptive RAG and Flow Engineering [1036]

  1. Core Ideas: Combines query analysis (modifying questions for retrieval) with online flow engineering (using tests/feedback loops for corrections). [1038]
  2. Query Analysis: Includes techniques like decomposition, step-back prompting, and routing to different sources (vector store, web search, LLM fallback). [1041]
  3. Online Flow Engineering: Involves performing checks within the RAG pipeline (e.g., relevance of retrieved documents, hallucination checks) and making corrections, such as retrying retrieval with an improved question or using web search as a fallback. [1053]
  4. Example Flow:
    • Start with a question. [1065]
    • Route to vector store, web search, or LLM fallback. [1066]
    • Retrieve documents. [1069]
    • Check document relevance; if not relevant, fall back to web search. [1070]
    • Generate an answer. [1072]
    • Check for hallucinations and answer relevance. [1073]
    • Return the answer. [1075]

11. Is RAG Dead? The Future of Retrieval in Long Context LLMs [1387]

11.1. The Impact of Long Context Windows [1388]

  1. Increasing Context Lengths: LLMs now have significantly larger context windows, from thousands to millions of tokens, capable of processing hundreds or thousands of pages. [1390]
  2. The Question: With such large context windows, is RAG still necessary, or can LLMs simply "context stuff" and retrieve information directly? [1398]

11.2. Testing Long Context Retrieval [1399]

  1. Methodology: A "multi-needle" test was conducted, placing multiple pieces of information (ingredients for a pizza) at different intervals within a large context window (120,000 tokens for GPT-4). [1400]
  2. Key Findings:
    • Retrieval Difficulty Increases with More "Needles": Performance (percentage of needles retrieved) drops as more facts are placed in the context. [1427]
    • Reasoning Adds Difficulty: Asking the LLM to perform reasoning on the retrieved information (e.g., return the first letter of each ingredient) further degrades performance compared to simple retrieval. [1429]
    • Recency Bias: Needles placed towards the beginning of the context window are harder to retrieve than those placed towards the end. This effect is observed even with large context windows and is potentially due to LLMs' tendency to attend more to recent tokens. [1446]
  3. Skepticism Towards Needle-in-a-Haystack Tests: These tests might be misleadingly easy because:
    • They often involve single needles with no reasoning. [1470]
    • The "needles" are often distinct from the "haystack" context, making them easier to spot. [1470]
    • Subtle facts are harder to retrieve. [1471]

11.3. The Changing Role of RAG [1476]

  1. RAG Today: Focuses on precise retrieval of relevant document chunks, often using arbitrary chunk sizes, specific embedding models, and KNN search with a fixed 'K' parameter. [1477]
  2. Challenges with Long Context:
    • No Retrieval Guarantees: Context stuffing doesn't guarantee accurate retrieval, especially for multiple facts or reasoning tasks. [1478]
    • Cost and Latency: High token usage and cost ($1 per 100k tokens for GPT-4). [1494]
    • Auditability and Security: Difficult to audit retrieval or manage security/authentication for different users. [1496]
  3. The Optimal Regime: Likely a middle ground between precise chunking and full context stuffing. [1499]
  4. Document-Centric RAG:
    • No Chunking: Operate on full documents rather than arbitrary chunks. [1500]
    • Query Analysis Remains Relevant: Techniques like routing, query rewriting, and query construction are still crucial for directing questions to the right document or data source. [1505]
    • Multi-Representation Indexing:
      • Create a summary or "proposition" of a document for retrieval. [1516]
      • Index the summary and store the full document separately. [1518]
      • Use the summary for retrieval and the full document for generation, leveraging long context LLMs. [1520]
      • Variants like "parent document retriever" also exist. [1524]
    • Raptor (Hierarchical Indexing):
      • Clusters documents, summarizes clusters, and repeats recursively to build an abstraction hierarchy of summaries. [1527]
      • Indexes all summaries, allowing retrieval based on question abstraction level. [1532]
      • Works well across full documents using long context embedding models. [1535]
  5. Reasoning and Feedback Loops:
    • Self-RAG / Corrective RAG: Move beyond single-shot RAG to cyclic flows with grading at retrieval and generation stages. [1544]
    • Grading: Assess document relevance, check for hallucinations, and evaluate answer relevance. [1548]
    • Feedback: If errors occur (e.g., irrelevant documents, hallucinations), rewrite questions, retrieve again, or use fallbacks like web search. [1549]
    • Latency Management: Use fast LLMs for grading and limit the number of feedback turns. [1558]
  6. Overall Picture:
    • Query analysis, routing, and query construction remain vital. [1577]
    • Less emphasis on arbitrary chunking; focus shifts to working with full documents. [1579]
    • Indexing techniques like multi-representation and hierarchical indexing are important for document-centric approaches. [1581]
    • Reasoning and feedback loops (grading, rewriting, fallbacks) are increasingly integrated into RAG systems, moving towards more cyclic and self-reflective flows. [1587]