B3 - model lifecycle

champion/challenger

• absolute quality gates are the minimum requirements that need to be satisfied

• eg: $F_1\ge 0.88$

• the new model should also perform better, not just meet the minimum gate, ie. relative evaluation

• eg: $F_{1,challenger}\ge F_{1,champion}+\delta$, where $\delta$ is a required improvement

• it should also provide a meaningful benefit

• eg: a challenger may have a better metric, but might:

  • require twice the RAM
  • take $40\mathrm{\%}$ longer
  • introduce compatibility risk

• statistical uncertainty also matters, so need:

  • confidence intervals
  • repeated evaluation
  • sufficiently large representative datasets
  • significance/practical-effect analysis

• model evaluation can involve:

  • model quality: F1, recall, precision
  • operational quality: latency, throughput, memory, cost, reliability
  • compatibility: signature/schema, pre-processing, downstream integration

promotion

• after the challenger passes everything, it is promoted to become the new champion

• aliases are useful here, as versions should be immutable for traceability

• a canary/progressive rollout pattern can be employed, which involves gradual increase in traffic to the new version while monitoring

• alternatively, A/B testing splits the traffic to the old and the new champion, and compares downstream outcome

• shadow deployment is when a request is secretly sent to the challenger for evaluation, while the old champion is used for responses

rollback

• if there are problems with the newly promoted champion, we may need to roll back to the previous one
• simply rolling back the model version may not be enough
• older version may be using different environment, schema, etc.