D2 - managed online deployments

• the complete request path:

CLIENT
│
│ HTTPS request
▼
AUTHENTICATION
│
▼
ONLINE ENDPOINT
│
│ traffic routing
▼
DEPLOYMENT
│
▼
SERVING INSTANCE
│
▼
SCORING RUNTIME
│
▼
run(data)
│
▼
MODEL
│
▼
prediction
│
▼
HTTP RESPONSE

## authentication
- for production managed online endpoints, Microsoft Entra authentication can be used so an application's identity obtains a token and invokes the endpoint under Azure authorization controls

Application
↓
identity
↓
Microsoft Entra
↓
access token
↓
endpoint

- an authorization failure points  toward access permissions rather than model accuracy
## endpoint
- the endpoint routes requests according to the configured deployment traffic allocation
- the application still calls the same endpoint
- the deployment needs **serving instances**
- requests can be handled by the available healthy serving instances
- for **instance sizing**, serving compute should be selected based on:
	- latency
	- throughput
	- memory
	- cost
- **instance count** refers to the number of healthy serving instances

- machines can be scaled:
	- **horizontally:** make each machine larger, eg: 2 CPU to 4 CPU
	- **vertically:** add more machines, eg: 2 instances to 4 instances
- online serving frequently uses horizontal scaling to handle changing request volumes

- Microsoft documents autoscaling for managed online endpoints through Azure Monitor auto-scale
- the deployment can therefore adapt serving capacity to demand

- **latency** is the time taken by a request
- **throughput** is the amount of work a service can handle per unit time
## serving  instance
- when a deployment starts, Azure needs the serving runtime to become operational:

Compute instance
↓
serving container/runtime starts
↓
environment available
↓
scoring code starts
↓
init()
↓
model loads
↓
ready

- several things can go wrong here, such as environment failure, model loading failure
- **health probes** help the platform know whether a serving instance is actually usable, and they are:
	- **liveness** because if the container/process has crashed or become irrecoverably unhealthy, the platform may need to restart/recover it
	- **readiness** asks whether the model is and ready to receive traffic
- health configuration must reflect real model start-up behaviour such as:
	- initial delay
	- period/frequency
	- timeout
	- failure threshold
- otherwise, there might be loops where a model takes 90 secs to load, but the config deems it as broken if not ready n 5 secs

- there might also be issues when just one request takes too long even though the deployment is healthy, ie. **request timeout**
- therefore, it is important to distinguish between start-up/probe timeout and inference request timeout

- the request may fail because of invalid input/schema mismatch, where the **model signature** and validation become useful
## Failure framwork
- a useful failure-layer framework:

1. CLIENT
   malformed request?

2. AUTH
   valid identity/token/permission?

3. ENDPOINT
   routing/configuration?

4. DEPLOYMENT
   healthy instances?

5. ENVIRONMENT
   dependencies available?

6. INIT
   model loads?

7. RUN
   scoring code works?

8. MODEL
   prediction itself valid?

9. CAPACITY
   enough CPU/RAM/instances?

10. NETWORK
    connectivity/configuration?

## HTTP-style failures
- **4xx-ish problems** often points toward the request/caller side:
	- authentication
	- authorization
	- malformed request
	- bad input
- **5xx-ish problems** often suggests the service/deployment couldn't successfully process the request:
	- container failure
	- scoring exception
	- dependency failure
	- resource problem
## monitoring
- once serving is healthy, operational telemetry is needed such as:
	- request count
	- latency
	- error rate
	- CPU
	- memory
	- instance utilisation
- and later model-specific monitoring:
	- input distributions
	- prediction distributions
	- data drift
	- model quality

- service health does not imply model health
## actual end-to-end request

1. Client constructs valid payload
   ↓
2. Client authenticates
   ↓
3. HTTPS request reaches endpoint
   ↓
4. Endpoint routes to deployment
   ↓
5. Healthy/ready serving instance receives request
   ↓
6. run(data) processes payload
   ↓
7. model performs inference
   ↓
8. response formatted
   ↓
9. prediction returned
   ↓
10. latency/errors/telemetry recorded