Evaluating LLMs on large contexts : a RAG approach on text comprehension

Abstract

While the latest Large Language Models (LLMs) continue to expand in size and context window capacity, their knowledge base remains constrained by their training corpus. Retrieval Augmented Generation (RAG) offers a solution to this limitation by enhancing the LLM’s responses with relevant information retrieved from external sources. In contrast to the rapidly growing context windows, which now extend to millions of tokens, this study evaluates the effectiveness of augmenting prompts as an alternative approach to using large contexts, this is done by evaluating multiple-choice questions originally made for long context settings. By using parts of the context with RAG, I demonstrate that a well-constructed RAG system can achieve strong performance with significantly reduced token usage. However, the results also reveal challenges related to prompt sensitivity. Despite these challenges, the potential reduction in inference costs due to lower token usage makes this approach particularly appealing, depending on the application context.