Faithfulness Judge for RAG Answers
A judge prompt that decomposes an answer into atomic claims and verifies each against the retrieved chunks, returning a faithfulness score plus the list of unsupported claims.
Prompt
You are a quality reviewer for a legal RAG system's answers. Your job is to find claims not supported by the sources.
Retrieved chunks: <retrieved> [Paste chunks with ids] </retrieved>
Answer under review: <answer> [Paste the answer] </answer>
Process:
- Decompose the answer into atomic claims. An atomic claim is a single independently verifiable sentence. Split compound sentences.
- Assign each claim one verdict:
- SUPPORTED — directly supported by a chunk. Quote the exact supporting text.
- PARTIAL — partially supported, but the answer extends, generalises, or sharpens beyond the source. Explain the gap.
- UNSUPPORTED — no basis in the chunks. This is a hallucination.
- CONTRADICTED — the chunks say the opposite. Quote it.
- Pay particular attention to the common failure sites: clause numbers, dates, amounts, party names, quantifiers ("always", "in every case", "solely"), and modal verbs ("must" vs "may").
Return JSON: { "claims": [{"claim": "", "verdict": "", "evidence": "", "note": ""}], "faithfulness_score": "count of SUPPORTED over total claims", "unsupported_claims": [], "verdict": "PASS if no UNSUPPORTED and no CONTRADICTED, else FAIL", "most_dangerous_error": "the claim most dangerous to rely on legally, or null" }
Do not judge writing quality or completeness. Faithfulness only.
Why this converts well
- Ready to copy and use immediately
- Tailored for legal and public-sector work
- Easy to adapt to internal workflows
Tags
Related Prompts
Strict Source-Grounded Answering over a Legal Corpus
System prompt that forbids parametric knowledge, requires a cited source span for every claim, and defines explicit behaviour when the retrieved context does not cover the question.
Chunking Strategy Design for Legal Documents
Analyses a legal document type and produces a justified chunking strategy — boundaries, overlap, metadata, and edge cases — instead of arbitrary fixed-size splitting.
Building an Evaluation Set for a Legal Retrieval System
Generates an evaluation set from a given corpus, including questions that should fail, to measure retrieval and faithfulness rather than only happy-path answering.