AI and machine learning · Advanced · 28 min
How to Evaluate and Improve a RAG App with a Test Set
By Padding · Updated 15 Nov 2026
Python 3.15 · Ubuntu 26.04.1 LTS · PostgreSQL 18 · pgvector 0.8 · anthropic SDK 2.8.0
You will write a test set of questions with known answers, measure retrieval with hit rate and mean reciprocal rank and the full pipeline with pass or fail checks, then use the results to add a distance cutoff that makes the app decline questions it cannot answer.
On this page
A RAG app can look fine on the three questions you tried, and fail on the fourth. You cannot tell whether a change to the chunk size, the number of chunks or the prompt helped, unless you measure the same thing before and after. This tutorial builds that measurement: a small test set, a script that scores retrieval, a script that scores whole answers, and a loop for improving the app based on what fails.
The app you built answers questions about a made-up hosting company's help pages. The same method works for any RAG app. A RAG app has two parts that fail in different ways. Retrieval can miss the chunk that holds the answer, and generation can ignore it, garble it, or answer a question the documents do not cover. Measure the two separately, because the fixes differ.
Each answer check calls the embedding service and Claude once per question, so a run costs a few cents. Claude's thinking tokens are billed as output tokens and do not show in the answer, so the cost can be higher than the visible text suggests. The scripts print every failure, so you can read exactly what went wrong.
Step 1 · Write the test set
A test set is a list of questions where you already know the right answer. Fifty questions drawn from real users would be better than anything you can invent, so use real ones when you have them, such as support tickets, search logs or questions colleagues ask. Cover the cases that matter: questions in words that differ from the documents, questions whose answer spans two sections, and questions the documents cannot answer, where the right behaviour is to decline.
Each line of the file holds the question, whether the documents can answer it, the headings of the chunks that hold the answer, and a short fact that a correct answer must contain. Gold answers refer to headings, not chunk IDs, so the test set stays valid when you change how the documents are split:
cd ~/rag-app
mkdir -p eval
cat > eval/questions.jsonl <<'JSONL'
{"question": "When does Fernwood bill me each month?", "answerable": true, "gold": ["Billing and refunds > When you are charged"], "must_include": "signed up"}
{"question": "Can I get my money back after 30 days?", "answerable": true, "gold": ["Billing and refunds > Refunds"], "must_include": "30"}
{"question": "How long do refunds take to arrive?", "answerable": true, "gold": ["Billing and refunds > Refunds"], "must_include": "business days"}
{"question": "What happens if my card payment fails?", "answerable": true, "gold": ["Billing and refunds > Failed payments"], "must_include": "suspended"}
{"question": "How do I point my own domain at my site?", "answerable": true, "gold": ["Domains and DNS > Point a domain at your site"], "must_include": "A record"}
{"question": "Why has my DNS change not worked yet?", "answerable": true, "gold": ["Domains and DNS > DNS changes take time"], "must_include": "48 hours"}
{"question": "Can I transfer a domain I own to Fernwood?", "answerable": true, "gold": ["Domains and DNS > Transfer a domain in"], "must_include": "60 days"}
{"question": "How many backups do you keep?", "answerable": true, "gold": ["Backups and restores > Automatic backups"], "must_include": "seven"}
{"question": "How do I restore my site to an earlier version?", "answerable": true, "gold": ["Backups and restores > Restore from a backup"], "must_include": "database"}
{"question": "How long can I download a backup for?", "answerable": true, "gold": ["Backups and restores > Download a copy"], "must_include": "24 hours"}
{"question": "What is the weather in Paris?", "answerable": false, "gold": [], "must_include": null}
{"question": "Who won the football match yesterday?", "answerable": false, "gold": [], "must_include": null}
JSONL
wc -l eval/questions.jsonlCheck it worked
The output is 12 eval/questions.jsonl. Ten questions can be answered from the documents and two cannot. Keep a test set honest: write the gold answers from the documents before you look at what the app returns, and do not tune the app only until these twelve pass. When your set grows, set some questions aside and look at them only for a final check, as you would with any test data.
Step 2 · Measure retrieval
Retrieval comes first, because the model cannot use a chunk it was never given. Two numbers describe it. Hit rate at k is the share of questions where a correct chunk is among the top k results. Mean reciprocal rank (MRR) averages 1 / rank of the first correct chunk, so a correct chunk in first place scores 1, in second place 0.5, and missing scores 0. Create eval_retrieval.py:
cat > eval_retrieval.py <<'PY'
import json
from rag import retrieve
KS = (1, 3, 4)
rows = [json.loads(line) for line in open("eval/questions.jsonl")]
answerable = [r for r in rows if r["answerable"]]
ranks = []
for row in answerable:
found = retrieve(row["question"], k=max(KS))
rank = next((i for i, c in enumerate(found, 1) if c["heading"] in row["gold"]), None)
ranks.append(rank)
if rank != 1:
print(f"rank {rank}: {row['question']}")
for k in KS:
hits = sum(1 for r in ranks if r is not None and r <= k)
print(f"hit@{k}: {hits}/{len(ranks)}")
print(f"MRR: {sum(1 / r for r in ranks if r) / len(ranks):.3f}")
PY
uv run eval_retrieval.pyThe script skips the unanswerable questions, since they have no correct chunk. It prints a line for every question whose correct chunk is not in first place, giving the rank or None if it is missing from the top four, and then the summary.
Check it worked
The last four lines are hit@1, hit@3 and hit@4 as fractions of 10, and MRR between 0 and 1. For example, hit@1: 8/10, hit@3: 10/10 and MRR: 0.900 would mean that the right chunk is usually first and always in the top three. Read the lines above them: each is a retrieval miss you can inspect.
What the numbers tell you:
- If
hit@4is much higher thanhit@1, the right chunk is retrieved but ranked low. Sending more chunks to the model, or adding a reranker model, helps, at the cost of a longer prompt. - If even
hit@4is low, the right chunk is not found at all. Look at the failing questions: split chunks that cut an answer in two, an embedding model that does not know your vocabulary, and questions phrased very differently from the documents are the usual causes. Change the chunk size, add the heading to the embedded text (the ingest script already does), or try another embedding model, and run this script again after each change.
Troubleshooting
If every question prints rank None, the chunks table has no vectors or the vectors came from a different model than the one that embeds the question. Run psql -d ragdemo -c 'SELECT count(embedding) FROM chunks' and, if you changed models, empty the column and run the ingest script again.
Step 3 · Measure whole answers
Now score the full pipeline. For each question, call ask and apply three checks. The app must agree about whether the question is answerable. For an answerable question, one of its cited sources must be a gold heading, and the answer must contain the required fact. This is crude on purpose: a plain string check is cheap, repeatable and exact. Create eval_answers.py:
cat > eval_answers.py <<'PY'
import json
import sys
import anthropic
if "--guarded" in sys.argv:
from guarded import ask
else:
from rag import ask
rows = [json.loads(line) for line in open("eval/questions.jsonl")]
passed = 0
answered_ok = declined_ok = 0
for row in rows:
try:
out = ask(row["question"])
except (ValueError, anthropic.APIError) as err:
print("FAIL:", row["question"], "->", type(err).__name__, str(err)[:80])
continue
problems = []
if out["answerable"] != row["answerable"]:
problems.append(f"answerable={out['answerable']}, expected {row['answerable']}")
if row["answerable"]:
if not any(s["heading"] in row["gold"] for s in out["sources"]):
problems.append("no gold source cited")
if row["must_include"] and row["must_include"].lower() not in out["answer"].lower():
problems.append(f"answer lacks {row['must_include']!r}")
if problems:
print("FAIL:", row["question"], "->", "; ".join(problems))
else:
passed += 1
if row["answerable"]:
answered_ok += 1
else:
declined_ok += 1
print(f"passed {passed}/{len(rows)}")
print(f"answerable questions answered correctly: {answered_ok}/{sum(r['answerable'] for r in rows)}")
print(f"unanswerable questions declined: {declined_ok}/{sum(not r['answerable'] for r in rows)}")
PY
uv run eval_answers.pyCheck it worked
The output has one FAIL: line for each question that failed, naming the checks, and then three summary lines: how many questions passed, how many answerable ones were answered correctly, and how many unanswerable ones were declined. Because the model writes a fresh answer each time, run the script two or three times before you trust a small difference between two versions of your app. A gap of one question is noise.
A failure falls into one of these groups, and each has its own fix:
- The right chunk was not retrieved: fix retrieval, as above.
- The right chunk was retrieved but the answer misses the fact or cites the wrong chunk: look at the prompt and at how much text each chunk gives the model.
- An unanswerable question was answered anyway: this is the dangerous one, because a confident wrong answer is worse than no answer. The next step addresses it.
You can also grade answers with a second model call that reads the question, the gold fact and the answer, and returns a verdict. That helps when correct answers are worded in many ways and a substring check is too strict. A model grader makes mistakes of its own and has to be checked against your own reading of a sample of its verdicts, so use it as an addition to the string checks, not a replacement.
Step 4 · Choose a cutoff for questions the documents cannot answer
Retrieval always returns the nearest chunks, even when none of them is close. A question about the weather in Paris still gets four chunks about billing, and a model that is given them may try to be helpful. The distance of the best chunk is a signal you can use: if even the nearest chunk is far away, the question is probably off-topic. Look at the distances in your test set. Create eval_distances.py:
cat > eval_distances.py <<'PY'
import json
from rag import retrieve
rows = [json.loads(line) for line in open("eval/questions.jsonl")]
distances = {True: [], False: []}
for row in rows:
best = retrieve(row["question"], k=1)[0]["distance"]
distances[row["answerable"]].append(best)
print(f"{best:.3f} answerable={row['answerable']!s:<5} {row['question']}")
print(f"largest distance among answerable questions: {max(distances[True]):.3f}")
print(f"smallest distance among unanswerable questions: {min(distances[False]):.3f}")
PY
uv run eval_distances.pyCheck it worked
The output lists the best-chunk distance for each question, then the two summary lines. If the largest distance among answerable questions is smaller than the smallest among unanswerable ones, a cutoff between them separates your test set perfectly. With real embeddings the two ranges often overlap a little. Then you choose which mistake you can live with. A low cutoff refuses some answerable questions, and a high one lets some off-topic questions through. For a help centre, a wrong refusal costs a support ticket, and a wrong answer can cost a customer, so lean towards the refusal side, but decide on your own data. Twelve questions is too few to set a cutoff for a real app, so grow the set first.
Store your choice in the environment file, replacing <value> with a number between the two distances:
echo 'MAX_DISTANCE=<value>' >> ~/.config/rag-demo/envReload the file and add the guarded version of ask. It retrieves once, declines when the best chunk is too far away, and otherwise hands the same chunks to the generation step, so it costs no extra embedding call:
set -a
. ~/.config/rag-demo/env
set +a
cat > guarded.py <<'PY'
import os
import rag
MAX_DISTANCE = float(os.environ["MAX_DISTANCE"])
def ask(question):
chunks = rag.retrieve(question)
if not chunks or chunks[0]["distance"] > MAX_DISTANCE:
return {"answer": "The help centre does not cover that.", "answerable": False, "sources": []}
return rag.answer_from(question, chunks)
PY
uv run eval_answers.py --guardedTo use it in the API, import guarded in app.py and call guarded.ask where the app calls rag.ask.
Check it worked
Compare with the run before the cutoff. The line for unanswerable questions declined should now read 2/2, and no unanswerable question should appear in a FAIL: line. Look at the answerable questions too: if any now fail with answerable=False, expected True, your cutoff is too low, and you should raise it.
Step 5 · Keep the evaluation as a habit
The point of a test set is to compare versions, so use it every time you change something that affects answers: the chunk size, the overlap, the embedding model, TOP_K, the prompt, the model, the cutoff. Change one thing at a time, run both scripts, and keep the results next to the change. A note such as "chunk size 800 to 1,200: hit@1 7/10 to 9/10, passed 8/12 to 10/12" is the record that tells you, months later, why the app is set up as it is.
A chunk-size experiment follows the steps of this series. Change MAX_CHARS and OVERLAP_CHARS in the chunker and run it again. Load the new chunks into a separate table, embed them with the ingest script and the new table name, and point retrieve at it. Then run eval_retrieval.py for each version. Because gold answers name headings, not chunk IDs, the same test set works for every chunk size.
Also keep a log of real questions and the answers users rated badly, and add them to the test set. A test set that grows from real failures is the one that catches the next one.
Step 6 · Clean up the whole series
This is the last tutorial of the series, so this step removes everything the four tutorials created. The first command deletes the ragdemo database with all the chunks and vectors, the database role named after your login, the rag-app folder, the rag-demo folder with your sample documents, and the key file. Run it only if you do not need any of them any more. If an API server from the previous tutorial is still running, stop it with Ctrl+C first:
cd ~
rm -rf ~/rag-app ~/.config/rag-demo ~/rag-demo
sudo -u postgres dropdb ragdemo
sudo -u postgres dropuser "$(whoami)"If dropuser fails because the role still owns objects in another database, drop or reassign those first. The role only owned the objects in ragdemo here. PostgreSQL 18 and pgvector stay installed. sudo apt purge postgresql-18 postgresql-18-pgvector removes the packages, and it leaves the data directory in /var/lib/postgresql, which you can delete if the server held nothing else. The Python tools stay too. Also revoke the Anthropic and Voyage AI API keys in their consoles if you created them only for this series, and, if you finished the first LLM tutorial, delete its key folder: rm -rf ~/.config/llm-demo.
Next steps
- Protect the app from hostile text in questions and in documents: defend an LLM app against prompt injection.
- Turn the scripts into a proper test suite that runs on every change: write tests with pytest.
- Back up a database like this one before you delete it next time: back up and restore PostgreSQL with pg_dump.
Related tutorials
- How to Write and Run Tests in Python with pytestYou will install pytest 9.2 in a uv project, write tests for a small price-parsing module, run many cases from one test with parametrize, read a failure and fix the code, and use a fixture that gives each test its own temporary folder.
- How to Find and Fix Slow PostgreSQL Queries with EXPLAIN ANALYZEYou will read a PostgreSQL 18 query plan, add an index that turns a 40 ms table scan into a 0.2 ms lookup, and use pg_stat_statements to find which queries deserve that effort.
Need this done for you?
Padding does this work for growing companies, from the first diagnosis to a system that keeps working.
Get new tutorials