G6 - GenAI evaluation

metrics

groundedness

reponse's claims supported by the supplied context

relevance

response answers what was asked

coherenche

response is logically organised and internally understandable

fluency

response's linguistic quality, such as grammar, natural phrasing, readability, linguistic correctness

evaluators

GenAI evaluation often needs more sophisticated evaluators
automated evaluators takes query, response, context, reference answer and produces an assessment
some qualitative criteria can be assessed using an AI model acting as an evaluator/judge
this can be described as LLM-as-a-judge/model-based evaluation
evaluator output is itself an assessment, not infallible ground truth
its quality should itself be validated for the use case
Foundry provides evaluation capabilities for common dimensions, called built-in evaluators
a custom evaluator can be created to meet specific domain requirements
deterministic evaluation can also be used for some evaluators
eg: output must be valid JSON with specific fields
an example of a strong GenAI evaluation:

├── schema validity
├── expected fields
└── exact business rules

RETRIEVAL
├── Recall@k
├── Precision@k
└── ranking

QUALITATIVE
├── groundedness
├── relevance
├── coherence
└── fluency

OPERATIONAL
├── latency
├── tokens
└── cost

SAFETY
└── relevant safety evaluations