E1 - CI, CD for ML systems

ordinary software

consider a basic software workflow:

Developer
   ↓
Pull Request
   ↓
CI
├── lint
├── unit tests
├── security checks
└── build
   ↓
merge
   ↓
CD
   ↓
deploy application

CI - continuous integration

checks if the proposed change is safe enough to integrate into the shared codebase

CD - continuous deployment/delivery

deals with moving an accepted change toward an environment such as staging or production

change \to CI \to validated change \to CD \to deployed change

CI might run on every pull request
production CD might run only:
- after merge to main
- after approval
- after a release
- when a model passes promotion gates

ML

ML involves much more components with lifecycles, such as: code, data, env, infrastructure, pipeline, model, deployment
a commit might change only one, which should not trigger the same workflow
different directories have different responsibilities in a repo:

detector-ml/
│
├── src/
│   ├── preprocess.py
│   ├── train.py
│   ├── evaluate.py
│   └── score.py
│
├── tests/
│   ├── test_preprocess.py
│   └── test_score.py
│
├── components/
│   ├── preprocess.yml
│   ├── train.yml
│   └── evaluate.yml
│
├── pipelines/
│   └── training.yml
│
├── environments/
│   └── training.yml
│
├── infra/
│   └── main.bicep
│
└── .github/
    └── workflows/

pull-request CI

consider changes are made to preprocess.py, a PR is opened, and pre-merging CI involves:

PR
 ↓
checkout repository
 ↓
install test dependencies
 ↓
lint/static checks
 ↓
unit tests
 ↓
pipeline/component validation
 ↓
security checks
 ↓
PASS / FAIL

this is a quality gate on the codebase
it can also be unit tested

test cheap things cheaply

unit tests

tests a small isolated piece

integration test

tests whether multiple pieces work together

smoke test

tests if the basic system is working at all

CI shouldn't necessarily train the full production model as that can be expensive, slow, frustrating and wasteful
PR CI might use small representative sample for a tiny training smoke test to see if the pipeline executes
full training happens later under a deliberate workflow

infrastructure CI/CD

Bicep change
   ↓
syntax/lint validation
   ↓
ARM validation
   ↓
WHAT-IF
   ↓
review proposed changes

note: What-If shows what the deployment would change
when approved:

main.bicep
   ↓
GitHub Actions
   ↓
authenticate
   ↓
Azure deployment
   ↓
ARM
   ↓
desired infrastructure state

ML pipeline CD

consider changes to: src/train.py, components/train.yml and environments/training.yml
workflow might:

merge
 ↓
authenticate to Azure
 ↓
register/update required ML assets
 ↓
submit Azure ML training pipeline

then:

GitHub Actions
      ↓
Azure ML
      ↓
actual training compute

GitHub coordinates, Azure ML executes

model deployment CD

have a separate deployment workflow:

approved model:v29
       ↓
deployment workflow
       ↓
create green
       ↓
smoke test
       ↓
canary
       ↓
monitor
       ↓
promote traffic

this separation reduces coupling and blast radius

workflow separation

a giant workflow means that any changes trigger the entire thing
it also makes diagnosis and debugging difficult
a workflow separation might have:

.github/workflows/

ci.yml  # for PR tests and validation
infra-deploy.yml  # for  aprooved infra change andBicep deployment
train.yml  # for scheduled/approved trigger and Azure ML training pipeline
model-deploy.yml  # for approved candidate and safe endpoint rollout

the trigger should match the responsibility
conceptually configure workflows to respond only to relevant paths

environments

production shouldn't necessarily be the first Azure environment that sees a change
commonly: dev > staging > production for another layer of risk reduction
this does not mean having: main-dev.bicep, main-staging.bicep, main-prod.bicep
use parameters to express the differences
if a model is validated in staging, production should ideally receive the same immutable model artifact, which is called artifact promotion

secrets

workload identity federation is:

GitHub Actions
      ↓
GitHub-issued OIDC token
      ↓
Microsoft Entra trusts
specified GitHub identity
      ↓
Azure access token
      ↓
Azure operations

Azure needs an identity representing the GitHub workload, such as an application/service principal or suitable managed identity configuration with a federated credential
apply least privilege for permissions
different workflows may reasonably use different identities/permissions
environment can be protected by requiring reviewer approval to proceed
automation should remove repetitive manual work, not eliminate human judgement completely