G4 - retrieval evaluation

an evaluation dataset is needed to measure if the retrieval performance has actually improved
$r e c a l l @ k$ and $p r e c i s i o n @ k$ are good evaluation metrics
the ranking is also important, as two retrievers may retrieve the same chunks but one may rank irrelevant chunks higher
if the first relevant result appears at rank $r$ , thereciprocal rank is:

R R = \frac{1}{r}

across many queries, the mean gives Mean Reciprocal Rank (MRR)
RR rewards putting useful evidence near the top
a retrieval evaluation set can be built using queries and relevant sources
then different configurations can be tested against the same evaluation set for a controlled comparison
multi-metric evaluation is needed
reranking must also be evaluated, and any issues can be decomposed into candidate retrieval and ranking
actual user experience is crucial, so end-to-end evaluation is needed
it is analogous to evaluating both individual pipeline components and the complete ML system
regression testing checks if changes break existing functionality, similar to CI
golden or reference dataset includes representative inputs and expected/reference evidence/outcomes
it become a repeatable benchmark for app changes
production feedback combined with systematic offline evaluation is a great evaluation strategy