Why Your RAG Pipeline Is Broken, and How to Fix It with Jason Liu - 709

Embedding Choices

Jason Liu highlights the complexity of choosing between off-the-shelf and custom embeddings in retrieval-augmented generation (RAG) systems. He emphasizes the importance of testing over guessing, suggesting that companies should invest in robust datasets to evaluate their embedding strategies effectively 1. Jason notes that many companies mistakenly rely on external embedding models, assuming they will perfectly match their specific needs, which often leads to suboptimal results 2.

It's really a big assumption to think that we know what is and is not similar in this embedding space.

--- Jason Liu

He argues that understanding the nuances of embeddings is crucial for improving retrieval systems and avoiding common pitfalls.

   

Chunking Impact

The effectiveness of chunking strategies in RAG systems is another critical area 1. Jason Liu explains that chunking can significantly impact retrieval performance, especially when dealing with structured data like tables within PDFs 3. He shares that while some chunking strategies may only offer marginal improvements, others can lead to substantial gains, highlighting the importance of experimentation and testing.

There are times when using hybrid search with embeddings and BM25 is like 3% better.

--- Jason Liu

Jason stresses the need for companies to tailor their chunking approaches to their specific datasets and use cases.

   

Long Context Use

Long context models offer unique advantages in RAG systems by allowing more comprehensive data processing. Jason Liu describes how these models can be used to handle complex tasks, such as generating personalized pricing from lengthy transcripts and documents 4. He explains that while long context doesn't guarantee a one-shot answer, it facilitates a structured approach to problem-solving by breaking down information into manageable parts 5.

Long context can be a convenience for you.

--- Jason Liu

This method enables more sophisticated reasoning and decision-making processes, enhancing the overall effectiveness of RAG systems.