B2 - models, signatures and registration

import pickle

with open("model.pkl", "wb") as f:
    pickle.dump(model, f)

• model.pkl is a model artifact that contains the serialized learned model

• it is not ready for distribution as it lacks information such as:

  • framwork
  • environment
  • inputs expecred
  • function to call
  • outputs produces
  • training run ID

• MLflow provides a standard way to package models with additional metadata

MLflow Model
├── model artifact
├── MLmodel metadata
├── model flavour(s)
├── signature
├── dependencies/environment information
└── other model metadata

eg: logging a scikit-learn model:

mlflow.sklearn.log_model(
    sk_model=model,
    name="model"
)

model flavours

• different ML frameworks represent models differently, eg: scikit-learn, PyTorch, TensorFlow, XGBoost

• model flavours describe ways a model can be interpreted/loaded

• consider a logged scikit-learn model

• it may have a framework-specific flavour: sklearn,which understands how to load/use it as a scikit-learn model

• it also has a useful generic inference interface called pyfunc, which is python function flavour

• without a common interface, applications may need framework-specific code

• MLflow's Python-function interface gives a more standardized abstraction

• a model can have more than one flavour

model signatures

• an MLflow model signature describes the model's expected inputs and output

• useful for:

  • input validation
  • deployment
  • reproducibility
  • documentation
  • detecting schema mismatches

• MLflow can infer signatures from example training inputs and predictions

signature = infer_signature(
    X_train,
    model.predict(X_train)
)

• an example input can also be stored

MLflow Model
├── model
├── signature
└── input example

• if a model signature is changed, one must mind:
  • interface compatibility
  • latency
  • resource requirements
  • security
  • operational behaviour

model resgistry

• a model registry gives a central managed collection of model versions

• each version corresponds to a particular registered model artifact

• the version number can be used instead of a run hash

• logging might be done for every run, but only good candidates are registered

• registration means that a model version is now managed/tracked in the model registry

• deployment is a later action

• it also doesn't mean that a model is approved

• registry can hold models that are:

  • candidate
  • validated
  • production
  • archived

• model registries/workflows can use an alias representing a meaningful reference

• it can move without pretending the underlying immutable model version changed

• tags can also be attached to describe a registered model

• a lineage:

Registered Model v23
        │
        ▼
MLflow model
        │
        ▼
MLflow Run 83921
        ├── Git commit abc...
        ├── Data asset:v12
        ├── Environment:v5
        ├── Hyperparameters
        ├── Metrics
        └── Artifacts

• note: Azure container registries are different from model registries as they store container images

• promotion pipeline:

Training Job
     ↓
MLflow Run
     ↓
log parameters
log metrics
log model
     ↓
Candidate
     ↓
Quality Gates
     │
     ├── F1 ≥ threshold?
     ├── Recall ≥ threshold?
     ├── Signature valid?
     └── other checks?
     ↓
REGISTER
     ↓
Registered Model:vN
     ↓
staging/testing
     ↓
DEPLOY