F2 - models and deployment

• model, deployment and application are distinct things
• model is the underlying pretrained AI model, might include learned parameters, capabilities, context limits and modalities
• deployment is the configured way for applications to invoke a particular model
• application is the system being built

APPLICATION
│
├── instructions
├── conversation logic
├── RAG
├── tools
└── evaluation
       │
       ▼
MODEL DEPLOYMENT
       │
       ▼
FOUNDATION MODEL

• Foundry gives access to a model catalogue containing models from Microsoft/OpenAI and other model providers

• for model selection, consider quality, capability, latency and cost

• input/output modality must match the model capabilities

• eg: app needs to understand text and images, then a text-only model would not suffice

• this is a hard requirement

• context window is the maximum amount of information available to the model for a request

• RAG exists partly so relevant information can be retrieved rather than providing the entire corpus

evaluation

• GenAI optimisation often involves model choice, input tokens and output tokens rather than simply compute-instance sizing
• eg: request A requires 1000 input tokens, B requires 100000, so even with the same model, B requires substantially more processing
• candidate models are evaluated on a representative evaluation dataset as a simple benchmark is not enough

generation parameters

• temperature controls the amount of variation in the responses, where low gives more deterministic, and high gives more variation

• it does not give the model new knowledge

• output limits control the amount of output a model may generate

• it can help with latency, cost and predictable app behaviour

• GenAI optimization is partly token-budget engineering

• rate limits and capacity are the throughput and capacity constraints

• so, consider expected request rate, token throughput, latency, quotas and retries/backoff when appropriate

upgrade

• the exact Foundry lifecycle differs from custom Azure ML model registry workflows, but the engineering principle survives that a newer model is not automatically preferable
• changing model can change behaviours such as:
  • formatting
  • tool calling
  • reasoning
  • latency
  • token consumption
  • safety behaviour

candidate foundation model
       ↓
task-quality evaluation
       ↓
groundedness / relevance etc.
       ↓
latency
       ↓
token cost
       ↓
production decision