X2 - getting and preparing the data

identifying the following:

• the data source, where it is stored
• the current data format (tabular/structured/semi-structured/unstructured)
• the desired data format to train the model

eg: building a time-series forecasting model for predictive maintenance using an IoT sensor data

• extracting temperature readings from machines using JSON machines
• these JSON files are converted into a tabular format and transformed to calculate the average temperature per machine per minute
• this final dataset is the input for training

choose how to serve data to ML workflows:

• separate compute from storage
• store data for model training workloads

designing a data ingestion solution involves building a pipeline that reliably feeds the model with fresh data automatically and its scale

create a data ingestion pipeline using Azure Synapse Analytics, Azure Databricks, or Azure Machine Learning and design a solution:

• extract raw data from its source (like a CRM system or IoT device)
• copy and transform the data with Synapse Analytics
• store the prepared data in a Blob Storage
• train the model with Machine Learning

ingest early

separate extraction, transformation, storage and training

flexible, repeatable and scalable solution

eg: predicting equipment failures

• sensor data continuously extracted
• transformed nightly and cleaned into aggregate data readings
• stored as training data ready sets
• train and retrain models as new data arrives

this architecture must be planned and designed upfront