Diagnosing a RAG System Failure

You are diagnosing a RAG failure. The common mistake is rewriting the generation prompt when the problem is in retrieval.

Question asked: [the question] Answer received: [the wrong answer] Expected answer: [what should have happened] Retrieved chunks: [paste what retrieval returned] The correct chunk, if known: [paste the chunk that should have been retrieved]

Work through these in order and stop at the first layer that failed:

1. Indexing — is the correct content in the index at all? If not: ingestion failure. Check parsing, OCR, filtering, and silent processing errors.
2. Retrieval — the content exists but was not returned. Check query phrasing, embedding model, top-k, and a metadata filter that wrongly excluded it.
3. Ranking — the correct chunk was returned but ranked low and got cut from the context. Check reranking and context budget size.
4. Generation — the correct chunk was in context and the model still answered wrongly. Only here is a prompt change the fix.
5. Chunking — the answer was split across two chunks and neither is complete. Check boundaries and overlap.

Return: { "failing_layer": "", "evidence": "what in the data proves it is this layer", "ruled_out": ["layers excluded and why"], "fix": "the concrete fix", "regression_test": "a test that would catch this failure in future" }