The baseline, unflattering
10
Recordings evaluated
k=3 · baseline run
0.23
Avg hit-rate@3
target after tuning ≥ 0.60
0.19
Avg MRR
first relevant chunk often below rank 3
0.56
Avg coverage
key-term recall in retrieved chunks
A hit-rate of 0.23 against a 0.60 target is not a good result, and publishing it is the point. A harness that only ever reports success is not measuring anything; the number exists so that a change to chunk size or embedding model can be judged rather than argued about.
This is a v1, not a verdict. The value of the first number is that the second one can be compared to it.
What the harness measures
For each recording the harness builds test questions from known transcript content, retrieves the top-k chunks from that recording's index, and scores three things:
| Metric | What it asks | What a low score tells you |
|---|---|---|
| hit-rate@k | Did a relevant chunk appear in the top k at all? | The embedding model or the chunk size is misaligned with how questions are actually phrased. |
| MRR | How high did the first relevant chunk rank? Rank 1 scores 1.0; rank 3 scores 0.33. | Retrieval finds the right material but ranks it poorly — a reranking or chunking problem, not a coverage one. |
| Coverage | What fraction of expected key terms appear in the retrieved chunks? | The context handed to the model does not contain the terms the question is about. No prompt fixes this. |
Separating these matters because they fail differently and are fixed differently. Coverage is a retrieval-recall problem, MRR is a ranking problem, and only once both are healthy does prompt work pay off. Measuring them together produces a single number that tells you something is wrong and nothing about what.
Fig. 07 — one retrieval, two different questions
The improvement loop
Scores are written to the gold layer alongside the episode metrics, which means retrieval quality sits in the same place as everything else about a recording and can be tracked over time rather than sampled in a notebook and forgotten. Tuning then has somewhere to aim: chunk size, embedding model, prompt, and k — changed one at a time, scored the same way each time.
Based on House of Lords proceedings (UK Parliament). Not affiliated with, or endorsed by, the UK Parliament.