Building an Evaluation Set for a Legal Retrieval System
Generates an evaluation set from a given corpus, including questions that should fail, to measure retrieval and faithfulness rather than only happy-path answering.
Prompt
You are building an evaluation set for a legal RAG system.
Corpus: [describe the documents — type, volume, legal domain] Sample chunks: <corpus_sample> [Paste 3-5 representative chunks] </corpus_sample>
Generate 20 evaluation items with this distribution:
- 6 direct factual questions — answer sits in a single chunk.
- 4 multi-hop questions — answer requires combining two or more chunks.
- 3 comparison questions — across clauses, documents, or versions.
- 4 unanswerable questions — the corpus does not contain the answer and the system must say so. These matter most.
- 3 trap questions — phrased to presuppose a false fact absent from the corpus, testing whether the model corrects the premise or plays along.
For each item return JSON with: { "id": "", "question_he": "", "question_en": "", "type": "single_hop | multi_hop | comparison | unanswerable | trap", "expected_behavior": "precise description of what counts as correct", "required_chunks": ["chunk ids that must appear in retrieval"], "failure_mode": "what failure looks like for this item" }
Do not generate questions whose answer is trivially inferable from the phrasing. Every question must require genuine retrieval.
Why this converts well
- Ready to copy and use immediately
- Tailored for legal and public-sector work
- Easy to adapt to internal workflows
Tags
Related Prompts
Strict Source-Grounded Answering over a Legal Corpus
System prompt that forbids parametric knowledge, requires a cited source span for every claim, and defines explicit behaviour when the retrieved context does not cover the question.
Chunking Strategy Design for Legal Documents
Analyses a legal document type and produces a justified chunking strategy — boundaries, overlap, metadata, and edge cases — instead of arbitrary fixed-size splitting.
Faithfulness Judge for RAG Answers
A judge prompt that decomposes an answer into atomic claims and verifies each against the retrieved chunks, returning a faithfulness score plus the list of unsupported claims.