A6 - datastores & data assets

Azure storage

the physical storage where the bytes actually live

Azure Blob Storage
└── detector-data/
    ├── run_0001.csv
    ├── run_0002.csv
    └── ...

datastore

an Azure ML object representing a connection/reference to an underlying storage service

so, the storage URL need not be used, and every job/team does not require account, container, path, authentication mechanism, and connection details
it does not contain the data itself, but is a connection abstraction

data asset

a named/versioned reference to data that Azure ML workflows can consume

data versioning supports reproducibility and lineage
these versions can be references to different paths/locations

PHYSICAL LAYER
Azure Storage
└── detector/
    ├── 2024/
    ├── 2025/
    └── 2026/

CONNECTION LAYER
Azure ML Datastore
"detector_store"
        │
        ▼
points to underlying storage

ML ASSET LAYER
detector-training-data
├── version 1 → relevant path/data
└── version 2 → updated path/data

three common data asset types:
- uri_file: data is essentially a single file
- uri_folder: data consists of a directory/folder of files
- mltable: a more structured/tabular data abstraction that can describe operations/schema-related loading behaviour and provide a convenient tabular interface
there are multiple input modes in which Azure ML makes the data accessible to the training process
download mode is when ML makes data available by downloading it to the job's compute
used when:
- dataset fits locally
- job reads much/all of it repeatedly
- local access is beneficial
in mount-style mode, the remote data is exposed to the job through a filesystem-like path without requiring the entire dataset to be eagerly copied to local disk first
the application can often interact with it as though it were available through the filesystem
used when:
- dataset is very large
- copying everything would be expensive
- workload doesn't necessarily need every byte locally
remote access is not free, so these must be considered when choosing mount over download:
- dataset size
- access pattern
- network throughput
- local disk
- start up time
- repeated reads
- cost
direct access mode is when the job receives the URI directly rather than Azure ML downloading/mounting it
useful for libraries/services capable of directly consuming remote URIs
data should not be in the container
container = code + software environment; data should be in an external storage/data asset
Azure ML connects them when the job runs