B1 - experiment tracking

• Azure ML jobs can use MLflow APIs to record experiment information
• an experiment groups related runs
• a run represents one execution/trial

parameters, metrics and artifacts

• parameters describe the configuration of a run, and typically don't change continuously during a run

• MLflow's API uses the word parameter more generally for configuration values logged

• eg: mlflow.log_param("max_depth", 10) where max_depth is a hyperparameter

• metrics are numerical measurements of performance or behaviour

mlflow.log_metric("f1", f1)
mlflow.log_metric("recall", recall)

• since metrics can change over time unlike parameters
• the same metric can be logged repeatedly:

mlflow.log_metric("loss", 0.90, step=1)
mlflow.log_metric("loss", 0.61, step=2)
...

• MLflow can track loss vs training step

• metrics can form a time series

• artifacts are files/outputs associated with the run, eg:

artifacts/
├── confusion_matrix.png
├── roc_curve.png
├── model.pkl
├── feature_importance.csv
└── evaluation_report.json

• logs are not the same as MLflow metrics:

# human-readable log
print(f"F1 = {f1}")

# structured metric
mlflow.log_metric("f1", f1)

• conceptually, training code could look like:

import mlflow

with mlflow.start_run():

    mlflow.log_param("max_depth", max_depth)
    mlflow.log_param("learning_rate", learning_rate)

    model.fit(X_train, y_train)

    predictions = model.predict(X_val)

    f1 = ...
    recall = ...

    mlflow.log_metric("f1", f1)
    mlflow.log_metric("recall", recall)

    mlflow.log_artifact("confusion_matrix.png")

• when training code runs as an Azure ML job, Azure ML can integrate with MLflow tracking:

Azure ML Job
      │
      ▼
train.py
      │
      ├── mlflow.log_param(...)
      ├── mlflow.log_metric(...)
      └── mlflow.log_artifact(...)
             │
             ▼
        MLflow tracking
             │
             ▼
      visible through Azure ML

misc

• within Azure ML scenarios, the Azure ML workspace provides integration with an MLflow-compatible tracking endpoint

• MLflow can be used locally too, where local tracking is needed

• run IDs are unique identifier for each MLflow run, which is useful for lineage

• tags are descriptive and searchable metadata useful for organising/describing runs

autologging

mlflow.sklearn.autolog()

• MLflow can automatically capture supported information such as:
  • model parameters
  • metrics
  • model artifacts
  • framework-specific information
• the exact things captured depend on the framework/integration
• custom metrics/artifacts still require manual logging

artifact cs MLflow model

mlflow.sklearn.log_model(...)
mlflow.log_artifact("model.pkl")

• logging as a model saves the model with MLflow model metadata/flavour information

• it is useful l for downstream model lifecycle tooling

• saving as an artifact simply saves the file

model signature

• an MLflow model signature describes expected model inputs/outputs
• eg:

MODEL
│
├── Input schema
│   ├── temperature
│   ├── voltage
│   └── noise
│
└── Output schema
    └── failure_probability

• so deployment can detect if data with the wrong shape/type is sent