G7 - advanced RAG

similarity thresholds

• require a result to satisfy some minimum relevance/similarity criterion before accepting it
• top-$k$ controls quality, whereas threshold controls minimum acceptance quality
• it can help remove terrible matches, eg: when a restaurant menu is asked to an app designed for a science lab
• threshold is tuned through evaluation to optimise relevance and grounded-ness
• to improve trust-worthiness, LLMs can have a controlled fallback if no retrieved chunk meets the required threshold

embedding fine-tuning

• with suitable training examples, an embedding model can potentially be adapted so domain-relevant items are represented more usefully
• always diagnose first before execution
• remember that new embedding model means new vectors

RAG A/B testing

• offline evaluation can A/B test against a golden evaluation set

• if both configurations are viable candidates, production traffic can be split in online A/B testing

• complete RAG optimisation loop

CURRENT RAG
    ↓
offline evaluation
    ↓
identify bottleneck
    │
    ├── chunking?
    ├── embeddings?
    ├── search?
    ├── threshold?
    ├── reranking?
    └── generation?
    ↓
candidate RAG
    ↓
regression evaluation
    ↓
quality gates
    ↓
shadow / A-B where appropriate
    ↓
production monitoring
    ↓
promote / rollback