RAG vs RAG + agentic planning

We take 10 hard questions from the FRAMES benchmark.
These questions test the AI system’s ability to handle complex reasoning tasks, and to handle multi-hop retrieval.
The knowledge base is Wikipedia articles.

We see that a simple RAG pipeline with Claude 3.5 Sonnet achieves 40% accuracy.
On the same retrieval stack, PromptQL achieves 100% accuracy.

The setup

• Naive RAG: A simple RAG pipeline with vector search that retrieves the top 5 articles that match the question and then provides that to the LLM
• Agentic RAG: A vector search tool is provided to the LLM. The tool retrieves the top 5 articles that match a given query. The LLM uses the tool to create search queries on its own or as guided by the user.
• PromptQL: Same retrieval function as used in the setups above, but instead of retrieving data and processing in context, PromptQL uses the LLM creates a plan and then executes it programmatically.

The LLM used in all the 3 setups above is Claude 3.5 Sonnet.

PromptQL achieves this accuracy by providing a planning engine on top of the underlying retrieval and LLM primitives,
and by ensuring a high degree of plan following.

Benchmark questions

Discussion: RAG failure modes

Extraction errors in processing

The following question failed with Agentic RAG because of errors in processing the data in context:

If my future wife has the same first name as the 15th first lady of the United States’ mother and her surname is the same as the second assassinated president’s mother’s maiden name, what is my future wife’s name?

Correct answer: Jane Ballou

Agentic RAG final response:

Let me process all this information:

1. The 15th First Lady was Mary Todd Lincoln, and I need to find her mother’s first name
2. The second assassinated president was James Garfield, and I need to find his mother’s maiden name

From searching through the results:

1. Mary Todd Lincoln’s mother’s name Eliza Parker
2. James Garfield’s mother was Eliza Ballou

Therefore, if your future wife has the first name “Eliza” (like Mary Todd Lincoln’s mother) and the surname “Ballou” (like James Garfield’s mother’s maiden name), her name would be Eliza Ballou.

So your future wife’s name would be Eliza Ballou.

The problem here is the wrong extraction of the 15th First Lady from the Wikipedia article. Mary Todd Lincoln is actually the 16th First Lady, while Harriet Lane is the 15th First Lady. This is a common problem with errors in processing when dealing with large data in-context.

Context size limitations

The following question failed on Agentic RAG due to context size limits:

I have an element in mind and would like you to identify the person it was named after. Here’