C1 - pipelines & components

this is the idea of a dataflow graph

components define how the step works, like function definitions

jobs are actual executions, like function calls

ML pipelines are naturally represented as a Directed Acyclic Graph (DAG)

directed because each edge has a direction: $A\to B$

acyclic because there are no cycles inside: $A\to B\nrightarrow A$

graph because nodes and edges represent the workflow

consider monitor > retrain > deploy, and then monitoring later triggers another retraining even

that is a system-level lifecycle loop

the individual pipeline should still be a DAG

the lifecycle can repeat, but each individual pipeline remains acyclic

components make reuse possible

eg: preprocess:v3 can be reuse it in multiple pipelines: A preprocess:v3 > sklearn training, B preprocess:v3 > XGBoost training, C preprocess:v3 > neural network training

component reuse reduces duplication and inconsistency

they can also be versioned

pipelines can have inputs

therefore, they can be reused later with different configurations without needing to recreate the pipeline

a hyperparameter can be exposed as a pipeline input, but not every pipeline input is a hyperparameter

eg: data_asset_version

component outputs become downstream inputs

eg: preprocess.outputs.clean_data can be wired into train.inputs.data

this makes the data dependencies explicit

pipeline data dependencies can determine execution ordering

so, Azure ML can infer the order, eg: pre-processing before training

thus, pipelines are naturally represented as DAGs

independent branches can run in parallel

eg: when training two models, if they don't depend on each other, they can potentially run concurrently

pipeline caching/reuse mechanisms can be used for eligible steps/configurations

eg: pre-processing takes 40 mins, and produces exactly the same output for a given input/configuration

component interface design is needed to explicitly state all the inputs and outputs

adding conditional logic for metrics creates an automated quality gate

thus pipelines can encode controlled promotion logic