G8 - the complete GenAIOps lifecycle

• the key difference between ML and GenAIOps lifecycle is that the release unit is the AI application, not just model weights

assistant:v12
│
├── foundation model deployment
├── system prompt:v8
├── prompt template:v5
├── chunking config:v4
├── embedding model:v3
├── search config:v7
├── index/corpus:v21
├── top-k = 5
├── reranker:v2
└── tool definitions:v6

• all individual components that make up the application must be versioned

• for development:

candidate configuration
        ↓
evaluation dataset
        ↓
compare with current version

• for component evaluation:

RETRIEVAL
Recall@k
Precision@k
MRR

GENERATION
Groundedness
Relevance

AGENT
Tool selection
Arguments
Task completion

OPERATIONS
Latency
Tokens
Cost

• component metrics are not enough, end-to-end evaluation is needed

user question
      ↓
entire application
      ↓
final response/task
      ↓
end-to-end quality

• quality gates help set non-negotiable absolute gates

• relative comparison help balance quality, latency, cost and operational behaviour

• once a candidate is approved:

candidate application config
        ↓
staging
        ↓
evaluation/integration tests
        ↓
approval
        ↓
production

• once deployed, monitor production:

PRODUCTION
   │
   ├── latency/errors
   ├── tokens/cost
   ├── retrieval health
   ├── tool health
   ├── quality signals
   ├── safety signals
   └── user feedback

• the knowledge lifecycle:

new documents
      ↓
ingestion
      ↓
chunk
      ↓
embed
      ↓
index update
      ↓
production knowledge changes

• knowledge deployment needs evaluation too to ensure freshness, quality, relevance, and security

• rollback should restore a compatible known-good application configuration, not blindly swap only the mode