A8 - ML jobs

from azure.ai.ml import command, Input

job = command(
    code="./src",   # directory containing the source code
    command="python train.py --data ${{inputs.data}} --max-depth 10",  # to execute after job env is running
    inputs={
        "data": Input(
            type="uri_folder",
            path="azureml:detector-data:5"
        )
    },
    environment="azureml:sklearn-env:3",
    compute="cpu-cluster"
)

• ${{inputs.data}} resolves that placeholder to the path/URI through which the job can access the data
• this allows the data to be changed as a parameter rather than hardcoding the URL into the code
• hyperparameters can be configured similarly

inputs={
    "max_depth": 10
}

command="python train.py --max-depth ${{inputs.max_depth}}"

• this creates a job definition/object that needs to be submitted:

ml_client.jobs.create_or_update(job)

• submitted jobs may be queued as Azure may need to:

  • provision a node
  • prepare the environment
  • satisfy quota/capacity
  • schedule resources

• job outputs/artifacts need to be persisted to storage

• outputs can be defined similarly to inputs:

outputs={
    "model_output": ...
}
# use ${{outputs.model_output}} in command

• metrics are different from artifacts

• logs are also produced by jobs that are retained by Azure for inspection

• .amlignore works similarly to .gitignore, to exclude files from code uploads

• jobs can also be defined in YAML and submitted using CLI:

$schema: ...
type: command

code: ./src

command: >
  python train.py
  --data ${{inputs.data}}

environment: azureml:sklearn-env:3
compute: azureml:cpu-cluster

inputs:
  data:
    type: uri_folder
    path: azureml:detector-data:5

• SDK is useful when job construction is part of python logic

• YAML + CLI is useful for configuration-driven workflows and automation

• there can be issues in either of the four: code, data, environment, or compute

• Azure does not fix bugs in train.py, the job simply executes remotely and fails

• one needs to inspect logs, fix the code, and submit another job

• there may also be a ModuleNotFoundError, which is an environment problem

• if the processing needs 64 GB RAM, but the compute node only has 8, the job could fail from resource exhaustion, which is a compute problem

• Job outputs help prepare for pipelines

• eg: job A (pre-process > clean data) > job B (train > model) > job C (evaluate > metrics)