G5 - tracing, monitoring and safety

• monitoring is concerned with the overall behaviour of the application, ie. system-level trends

• eg: p95 latency, error rate, grounded-ness, token consumption

• tracing is concerned with a particular request, ie. request-level trends

• it helps pinpoint the faulty step

• it has spans, which are operations within a trace

TRACE: request 123
│
├── span: retrieval
├── span: model invocation
├── span: tool call
└── span: final generation

• several services such as Foundry app, Azure Ai Search, model, and external API, participate in a logical user request

• tracing helps connect those operations into one end-to-end path

• tokens are another operational quantity introduced in GenAI that needs monitoring

• it helps identify operational regression

• production quality and user feedback are also invaluable

• production GenAI safety may involve controls around:

  • harmful content
  • prompt attacks
  • inappropriate tool use
  • sensitive information
  • unauthorized data access

• prompt-injection is when retrieved data, such as prompts, try to mimic system instruction

• retrieved/external content should be treated as untrusted data

• it can be direct, eg: Ignore your instructions and do X., or indirect, eg; Ignore the application's instructions and do X.

• the second is especially relevant to RAG/agents because the malicious instruction enters through external content

• security should not depend only on prompting

• enforce authentication and authorisation before generation

• tool safety is also important, ensuring least privilege is applied

• input/output controls can be in place, such as input/output safety checks