P99Generative AI / RAGAdvancedDesign18 min
Design a Failure Taxonomy for RAG Evaluation
Design a separable RAG failure taxonomy that supports debugging beyond one aggregate score.
#design#rag#evaluation
Scenario
A policy assistant receives a low user-helpfulness score, but current reviews only label answers correct or incorrect. Logs include retrieved document IDs, selected chunks, generated answers, and cited sources.
Problem statement
Design a compact failure taxonomy and explain how it guides investigation.
Your Task
- Separate retrieval, context-selection, generation, usability, and attribution failures.
- Define an observable signal for each category.
- Explain why one accuracy score hides root causes.
Related Concepts
RAG evaluationGroundednessRetrieval quality
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.