A10 - architecture design

question

• Imagine you're working on this project: Predict imminent detector failure from telemetry.
• You have 400 GB of historical telemetry stored as Parquet files. Every week, new labelled telemetry arrives.
• Your Python training program: src/train.py uses:
  • Python 3.11
  • pandas
  • scikit-learn
  • MLflow
• The model is a relatively lightweight gradient-boosted classifier, so CPU training is sufficient.
• During development, you want interactive notebooks.
• In production, training happens automatically once per week and normally takes about 40 minutes. You want production training compute to cost essentially nothing while idle.
• The current training dataset must be versioned so six months later you can identify exactly which logical dataset was used for a model.
• The trained model produces: model.pkl and metrics: precision, recall, F1
• Eventually GitHub Actions will submit the production training job.

solution

ARM / resource layer

rg-detector-ml
├── Azure Machine Learning Workspace
├── Azure Storage Account
├── Azure Key Vault
├── Azure Container Registry      [if required]
└── monitoring/support resources  [as required]

• for reproducibility: main.bicep
• keep infrastructure provisioning separate from weekly training: Bicep creates/configures the platform; Azure ML jobs subsequently use that platform

storage

• actual bytes in Azure Storage:

Azure Storage
└── telemetry/
    ├── historical/
    ├── week-001/
    ├── week-002/
    └── ...

• AML has named reference: Datastore: telementry_store
• also logical versioned datasets, ie. data assets:

detector-training-data
├── v1
├── v2
├── v3
└── ...

• the scenario specifies many Parquet files, consider either uri_folder if the job simply needs a folder of files, or MLTable if Azure ML's structured tabular abstraction is needed
• next week's telementry should lead to a new logical data version rather than silently mutating an older version reference
• investigate read-only mount as 400 Gb is quite large, and benchmark against the actual access path

compute

• for development: compute instance because need:
  • notebooks
  • debugging
  • interactive exploration
  • persistent development environment
• for production: CPU compute cluster because training is automated and want explicit reusable compute configuration, with min_nodes = 0 so it can scale down completely when idle
• the cold start delay is acceptable for a weekly workload
• set max_nodes appropriately based on concurrency, workload design, quota, and budget

environment

• since requirements are straightforward investigate an appropriate curated environment
• else, define a custom environment using an appropriate base image plus pinned Conda/pip dependencies

detector-training-env:v1
├── suitable base runtime
├── Python 3.11
├── pandas
├── scikit-learn
└── MLflow

production training job

CODE
./src
└── train.py

COMMAND
python train.py
    --data ${{inputs.training_data}}
    [other configurable parameters]

DATA
detector-training-data:vN

ENVIRONMENT
detector-training-env:vN

COMPUTE
cpu-training-cluster

outputs and metrics

• the model artifact/output is persisted
• metrics describe model performance and should be logged as structured metrics, for example through MLflow:
  • precision
  • recall
  • F1
• producing model.pkl absolutely does not mean it should automatically become the production mode

Train
  ↓
Candidate model
  ↓
Evaluate
  ↓
Quality gates
  ├── FAIL → reject
  └── PASS
        ↓
     register
        ↓
consider promotion/deployment

Actions ID

• use workload identity federation with Microsoft Entra

GitHub Actions
      ↓
GitHub identity token
      ↓
Microsoft Entra federation
      ↓
Azure access token
      ↓
RBAC authorization
      ↓
Azure CLI
      ↓
submit Azure ML job

• workload identity should receive only the Azure permissions required at an appropriately narrow scope

reproducibility and lineage

• want at least:

MODEL v17
├── Azure ML / MLflow run ID
├── source repository
├── Git commit
├── training code
├── training data asset + version
├── underlying immutable/versioned data
├── environment + version
├── relevant dependency/container versions
├── compute configuration
├── hyperparameters
├── random seed where relevant
├── evaluation metrics
│   ├── precision
│   ├── recall
│   └── F1
└── model artifact

architecture

detector-ml/
├── src/
│   ├── train.py
│   └── evaluate.py
├── tests/
│   └── ...
├── jobs/
│   └── train.yml
├── environments/
│   └── conda.yml
├── infra/
│   └── main.bicep
└── .github/
    └── workflows/
        └── train.yml

                         GIT REPOSITORY
                               │
                               ▼
                        GitHub Actions
                               │
                        workload federation
                               │
                               ▼
                        Microsoft Entra ID
                               │
                         RBAC authorization
                               │
                               ▼
                           Azure ML
                               │
                               │ submit
                               ▼
                         COMMAND JOB
                               │
		┌────────────────┼────────────────┐
		 ▼                                               ▼                                               ▼
		CODE                                    DATA                           ENVIRONMENT
	    train.py                      training-data:vN                          env:vN
                               │
                               ▼
                         CPU CLUSTER
                         min_nodes=0
                               │
                               ▼
                            TRAIN
                               │
			  ┌────────────┴────────────┐
			  ▼                                                                          ▼
		   METRICS                                                         ARTIFACT
	 precision/recall/F1                                                     model.pkl
			  │                                                                         │
			  └────────────┬────────────┘
                               ▼
                           EVALUATE
                               │
                         QUALITY GATE
                          ┌────┴────┐
                          ▼                           ▼
                        FAIL                 PASS
                          │                          │
                        reject                     ▼
                                   REGISTER