F1 - architecture

• for GenAI, unlike classical ML, start with a pre-trained foundational model

Foundry model catalog
        ↓
select suitable model
        ↓
deploy/access model
        ↓
build application around it

• choose according to workload requirements

• deployment gives the application a way to invoke the selected model

• the application then receives generated output after sending things such as:

  • system instructions
  • user prompt
  • context
  • generation configuration

• generally, GenAI request involves:

SYSTEM INSTRUCTIONS
        +
USER MESSAGE
        +
RETRIEVED CONTEXT
        +
CONVERSATION CONTEXT
        ↓
       LLM
        ↓
generated tokens

• LLM is just one component of the application, the other components may be:

  • RAG
  • prompt
  • evaluation
  • safety
  • monitoring

• a Foundry project provides a working context for building the AI application

FOUNDRY PROJECT
│
├── model deployments
├── application/agent development
├── evaluations
├── connections/resources
└── observability

• the application often needs other services such as model deployment, search/index, etc, and connections/configurations allow the application to work with those resources

• in the Microsoft stack, Azure AI Search is an important service for retrieval/indexing scenarios

• common architecture:

• some applications need more than simply answering the prompt, hence multiple actions/tools may be needed

• an agent combines model reasoning with instructions, tools and state/orchestration to accomplish task

• evaluation differs from MLOps, and is harder

• may care about:

  • grounded-ness
  • relevance
  • response quality
  • task completion
  • safety
  • retrieval quality
  • latency
  • cost