10 - production ML and the model lifecycle

DATA
  ↓
validation
  ↓
TRAIN
  ↓
evaluate
  ↓
good enough?
  ↓
MODEL
  ↓
register
  ↓
deploy
  ↓
inference
  ↓
monitor
  ↓
performance degraded?
  ↓
retrain
  │
  └───────────────↺

registry

a centralised store for managing versioned assets like models or datasets

a registry lets one manage model versions and their associated metadata/lineage
eg: suppose models are trained over several months
model.pkl
model_final.pkl
model_final2.pkl
model_actually_final.pkl

model_final_USE_THIS_ONE.pkl


- rather than saving them as above, a mode registry is maintained:

DetectorFailureModel
│
├── Version 1
│ ├── F1 = 0.89
│ ├── training run 143
│ └── created June 3
│
├── Version 2
│ ├── F1 = 0.92
│ ├── training run 217
│ └── created July 14
│
└── Version 3
├── F1 = 0.94
├── training run 391
└── created August 20
```

modern registries link a model version back to the training run that produced it

lineage

the tracked history showing the exact data, code, and parameters used to create them

Production model v17
	│
	├── training run 8342
	│
	├── Git commit a8f21...
	│
	├── dataset v42
	│
	├── environment v8
	│
	├── hyperparameters
	│
	└── evaluation metrics

deployment

making a validated model available in an environment where it can actually perform its intended job

Registered model v17
	↓
deployment
	↓
production environment
	↓
real data
	↓
predictions

inference

the execution phase where the trained, frozen model applies training knowledge to generate real-time predictions on new data

\begin{aligned} training: & D_{t r a i n} \to θ \\ inference: & x_{n e w} \to f (x, θ) \to \hat{y} \end{aligned}

real-time inference is when an answer is needed immediately
the application sends a request and waits for a response, often exposed through an endpoint/API
the requests are individual and small
latency matters
batch inference is when large batches are provided, and an answer is not needed immediately
an always-running, low-latency API is not needed

data drift

the change in the statistical properties or distribution ( $P (X)$ ) of the input data over time after deployment

concept drift

the change in the statistical relationship ( $(P (X | Y)$ ) between the input features and the darget variable over time

eg: consider a failure detection model for a detector, where $X =$ temperature, and $Y =$ failure
- originally, $T = 15 - 25 ° C$ but now, $T = 30 - 40 ° C$ , which is data drift
- if $T$ remains the same, but a component was replaced, and originally $T = 25 ° C ⟹ P (failure) = 0.7$ but now $T = 25 ° C ⟹ P (failure) = 0.1$ , it is concept drift
drift does not automatically mean failure; it is a warning signal
there are two broad monitoring categories
system monitoring: CPU/GPU, memory, latency, throughput, request failures, availability, cost
ML monitoring: feature distributions, missing values, data drift, prediction distributions, accuracy, precision/recall/F1, RMSE, concept drift, model quality

ground-truth feedback

the process of comparing machine learning or AI model predictions against verified, real-world data labels to measure accuracy and guide model improvements

retraining

there can be different triggers for retraining:
- scheduled
- new data
- performance based
- drift based
retraining should not automatically mean deployment as the retrained model might be worse
there should be a quality gate
it may be the case that the retrained model performs better in evaluation but worse after deployment; it needs to be rolled back
it may be a beneficial to first only send a fraction of users/requests to the new model for observation, and gradually increase the share if everything looks healthy, which is a canary/progressive rollout