E3 - promotion across environments

• promotion:

• central principle:

roles

• development:
  • experimentation
  • debugging
  • rapid iteration
  • integration work
• staging: mock production to test if it will work
  • not another training stage
  • testing the candidate artifact and serving system
• production: the live system

environment and infrastructure

• configurations should differ

• eg: instance count for staging = 1, and production = 5; storage for staging = detector-staging, and production = detector-prod

• for ease, do not hard-code environment details

• use common definitions with environment-specific parameters

• eg: infra/main.bicep define:

	Azure ML workspace
	Storage
	Key Vault
	monitoring resources
	...

• deploy the same infrastructure definition with different parameter sets:

	main.bicep
	├── dev params
	├── staging params
	└── prod params

• this reduces configuration drift, where the declared and actual state differ

• manual portal changes are dangerous

• prefer: change code/config → review → automated deployment over ad-hoc production modification

staging validation

• a model is deployed, need to test:

readiness
     ↓
smoke test
     ↓
schema compatibility
     ↓
integration test
     ↓
latency benchmark
     ↓
security/access checks

• potentially also model-specific checks on appropriate validation data

• model evaluation checks if the model predictivity is acceptable, eg: absolute gates

• smoke test checks if the basics work (valid request gives valid response)

• integration test checks if the different pieces work together (app + endpoint + model + downstream service)

• approval gates can be:

  • fully automated
  • human
  • both

• approval is not rebuilding, but attachment to the elevated artifact/configuration

• environments can be protected, requiring reviewer approval before deployment

• environmental identities must be separated for safety and smaller blast radius

• combine with least privilege

production deployment

• release metadata can contain:

Release 2026.09.17
│
├── model:v29
├── image digest ABC123
├── Git commit 82ab...
├── environment config
├── deployment config
└── pipeline/run provenance

• this helps in proper rollback if needed