D1 - online vs batch endpoints

(X, y) \to learn model

x_{n e w} \to trained model \to \hat{y}

endpoint

app \overset{request}{\to} endpoint \to deployment \to model

endpoint is analogous to a stable front door, behind which there may be multiple deployments, eg: v17, v18

deployment

the chief benefit of this separation is that when a new deployment is added, client does not need a completely new endpoint

online endpoint

Azure ML managed online endpoints provide scalable HTTPS/REST endpoints for real-time inference
Azure manages much of the underlying serving infrastructure, security, scaling and monitoring
the prediction is returned as part of the request/response interaction, so it is synchronous
online endpoints are suited to scenarios with relatively short request-processing times and low-latency requirements

ONLINE DEPLOYMENT
├── model
├── environment
├── inference code/config
└── compute

Microsoft's current comparison notes that online-deployment compute is provisioned at deployment time and does not scale to zero, whereas batch compute can scale to zero
so online serving has an availability/cost trade-off

Azure ML custom online deployments can use a scoring script
Microsoft's current online-endpoint guidance uses init() to initialize the deployment/model and run() to process scoring requests

def init():
   ...
   
def run(data):
   ...

this way the model needs to be loaded only once, and can be ran multiple times
for supported MLflow deployment scenarios, Azure ML can deploy MLflow models without requiring custom scoring script
custom scoring can still be supplied when needed
therefore, proper logging gives a richer model package that serving tooling understands instead of just model.pkl

Azure ML batch endpoints are designed for long-running asynchronous inference over large datasets and can parallelize work across compute
they are asynchronous
batch endpoints are specifically recommended when inputs are large, distributed across multiple files, don't need low latency, and can benefit from parallelization
the compute can scale to zero
Azure ML batch endpoints can operationalize not only models but also pipeline components for long-running asynchronous workflows
this can be useful when inference itself is a multi-step process

online endpoints support authentication mechanisms, including Microsoft Entra-based authentication for managed endpoints
Microsoft's current guidance recommends Entra token authentication for production managed-online workloads because it is identity/RBAC based

control plane supports operations to manage endpoint such as:
- create endpoint
- update deployment
- delete deployment
- change configuration
data plane supports operations use endpoint for inference, such as:
- send telemetry > receive production
they are explicitly distinguished by Microsoft's endpoint authentication docs

Azure ML online endpoints support multiple deployments and traffic splitting, which Microsoft uses for safe blue/green rollout patterns
the endpoint itself remains stable, the deployment changes
this decouples the client contract from the model lifecycle
for batch, there can also be multiple endpoints, with one serving s the default
batch routing isn't the same real-time percentage traffic-splitting mechanism used for online endpoints
Azure's current comparison describes online endpoints as supporting traffic splitting while batch endpoints switch a default deployment
there might ultimately be both endpoint types for the same model family
eg: real-time monitoring with live telemetry, and historical scoring with large historical telemetry serving different workloads
training and inference compute can also use different hardware depending on the workload
model signature becomes crucial for clients to send compatible data