C3 - building a complete pipeline

• repo contains:

src/
├── preprocess.py
├── train.py
└── evaluate.py

• the pre-processing component:

PREPROCESS COMPONENT

Input:
    raw_data

Output:
    clean_data

preprocess_component = command(
    name="preprocess",
    code="./src",
    command="""
        python preprocess.py
        --raw-data ${{inputs.raw_data}}
        --clean-data ${{outputs.clean_data}}
    """,
    inputs={
        "raw_data": Input(...)
    },
    outputs={
        "clean_data": Output(...)
    },
    environment="azureml:preprocess-env:3"
)

• the train component:

TRAIN COMPONENT

Inputs:
    clean_data
    max_depth
    learning_rate
    n_estimators

Output:
    model

• the evaluate component:

EVALUATE COMPONENT

Inputs:
    model
    test_data

Outputs:
    metrics

• the pipeline can be defined as:

@pipeline()
def detector_training_pipeline(raw_data):

    prep = preprocess_component(
        raw_data=raw_data
    )

    train = training_component(
        clean_data=prep.outputs.clean_data
    )

    evaluate = evaluation_component(
        model=train.outputs.model
    )

• this is constructing a pipeline graph that Azure ML will later execute remotely

• if monthly retraining is needed, the required inputs can be pipeline-level inputs:

PIPELINE

Inputs:
├── raw_training_data
├── test_data
├── min_f1
└── min_recall

• these values flow into appropriate components:

raw_training_data
       │
       ▼
   PREPROCESS

test_data ───────────────┐
	                      EVALUATE

min_f1  ────────────────┐
min_recall ──────────────┤
                    QUALITY GATE

• training can be replaced with a sweep
• clean_data is the same for every trial
  
• then the primary metric and the goal can be specified
• that metric must be logged:

mlflow.log_metric("f1", f1)

• the candidate must go into a separate evaluation stage
• hyperparameter selection and final evaluation should be conceptually separated
• if hyperparameters are repeatedly chosen based on the same evaluation dataset, the model is effectively optimized toward it

TRAINING DATA
     ↓
SWEEP
     │
uses validation information
     ↓
BEST CONFIGURATION
     ↓
FINAL EVALUATION
     │
uses held-out test data
     ↓
production decision

• train for fit parameters​

• validation for choose hyperparameters/model

• test for final unbiased-ish evaluation​
  

• evaluation might produce:

  • F1
  • recall
  • precision
  • ROC-AUC
  • latency results
  • evaluation report

• some values can be logged through MLflow

• an evaluation report can also be an artifact, eg: evaluation.json

• quality gates should be separated from train.py

• components can have appropriate compute rather than running the whole workflow on one oversized machine

• separate environments can reduce unnecessary dependencies and improve modularity, but environments should not be fragmented unecessarily either

• pipelines need to be submitted like a command job:

pipeline_job = detector_training_pipeline(...)

ml_client.jobs.create_or_update(pipeline_job)

• a pipelines run is itself a job with child jobs/steps inside it

PIPELINE JOB 872
│
├── preprocess job
├── sweep job
│   ├── trial job 1
│   ├── trial job 2
│   └── ...
├── evaluation job
└── registration step

• pipeline definitions can be expressed through Azure ML's supported programmatic/configuration mechanisms

• GitHub fits into this:

GitHub Actions
      ↓
authenticate with Entra
      ↓
Azure CLI / SDK
      ↓
submit pipeline
      ↓
Azure ML
      ↓
PREPROCESS
      ↓
SWEEP
      ↓
EVALUATE
      ↓
QUALITY GATE
      ↓
REGISTER

• GitHub Actions does not need to execute the actual training loops itself, but can simply submit/orchestrate the pipeline
• use the CI runner to coordinate, and ML compute to execute jobs