D3 - batch endpoints and batch inference

CLIENT / PIPELINE
      │
      │ invoke
      ▼
BATCH ENDPOINT
      │
      ▼
BATCH DEPLOYMENT
      │
      ▼
BATCH JOB
      │
      ├── input data
      ├── model
      ├── environment
      └── compute cluster
      │
      ▼
parallel processing
      │
      ▼
persisted predictions

• the key difference from online being:

submit inference workload
        ↓
batch job executes asynchronously
        ↓
results persisted

compute

• batch inference can exploit parallel compute because many inference operations are independent
• group work into mini-batches, so each worker receives manageable chunks of input
• one model.predict(...) can often use CPU/vectorized operations much more efficiently
• mini-batch sizing involves memory, throughput, I/O and model behaviour, and should be benchmarked
• file-based mini-batching is when work can conceptually be divided by input files:

Worker 1
├── file001
├── file002
└── file003

Worker 2
├── file004
├── file005
└── file006

• the exact batching behaviour depends on the deployment/input configuration, but the core idea is partition the input into independently processable units

• batch scoring has a similar initialization idea to online scoring:

def init():
    # load model once per worker

def run(mini_batch):
    # process this chunk
    # return predictions

• max_nodes is just one capacity limit

• there may also be configuration controlling how many scoring processes run per node

• therefore the total parallelism is nodes $\times$ processes per node subject to CPU, RAM and workload constraints

• even when nodes are scaled-out, storage/network/coordination may become the bottleneck, so parallel scaling has diminishing returns

• batch sizing has a trade-off between throughput and cost

• scaling down to zero is attractive as start-up delay is usually acceptable

outputs

• batch inference generally needs to persist results, eg: predictions/results.csv
• the client shouldn't need to hold an HTTP connection open for the time scoring takes (hours)

invoke
  ↓
job ID/status
  ↓
processing
  ↓
results persisted

failures

• batch systems can support retry/failure-handling policies around mini-batches and failed work

• retries are useful for transient failures such as network error

• failure classification is important before choosing remediation

• acceptable partial failure must be defined

• tolerant failure policies still need:

  • failure reporting
  • auditability
  • appropriate thresholds
  • retry strategy
  • downstream awareness

• per-mini-batch timeout prevents a single work item from hanging indefinitely

• poor partitioning creates a straggler problem

• better partitioning tries to distribute work more evenly

• storage might be the bottleneck

• batch performance depends on the whole pipeline

rollout and monitoring

• instead of traffic splitting between two models, explicitly test one model on a selected dataset, compare results, then make the appropriate deployment the default/selected deployment for future invocations
• useful operational metrics:
  • job duration
  • records processed
  • mini-batches completed
  • mini-batches failed
  • retry count
  • throughput
  • CPU
  • memory
• and model level outputs:
  • prediction distribution
  • quality metrics when labels arrive
  • drift

                 MONTHLY SCHEDULE
                        │
                        ▼
                 BATCH ENDPOINT
                        │
                        ▼
                  deployment-v28
                        │
                        ▼
                CPU COMPUTE CLUSTER
                  min_nodes = 0
                        │
                scale up on job
                        │
                        ▼
                  2 TB INPUT DATA
                        │
                 partition/minibatch
	 ┌──────────────┼──────────────┐
	 ▼                                          ▼                                          ▼
   Worker 1                            Worker 2                            Worker N
	 │                                         │                                         │
   init()                                       init()                                  init()
	 │                                         │                                         │
  batch A                                   batch B                                   batch C
	 │                                         │                                         │
	 └──────────────┼──────────────┘
						▼
					PREDICTIONS
						│
						▼
					Azure Storage
						│
						▼
					job completes
						│
						▼
				    scale back to 0