RAG Is Mostly a Retrieval Problem

When a retrieval-augmented system gives a bad answer, people blame the model. Nine times out of ten the model never got the right context in the first place.

Fix retrieval before you swap models

If the correct passage was not pulled in, no model can answer well. So the work is in retrieval: how you split documents, how you rank results, and how fresh the index is. That is where most of the quality comes from.

What we tune first

• Chunking: too big and you drown the model, too small and you lose meaning.
• Ranking: the right passage has to land in the top few, not somewhere on page three.
• Freshness: an answer from last quarter's document is a wrong answer.

The takeaway

Spend your time making sure the right context reaches the model. Once it does, even a mid-sized model gives good answers. Skip that and the biggest model will still sound confident and be wrong.