G2 - chunking, embeddings and indexing

chunking

• chunking balances specificity and context
• if chunks are too large, lots of irrelevant material is sent to the LLM causing problems such as:
  • retrieval less precise
  • more tokens
  • greater latency/cost
  • irrelevant context
• if chunks are too small, related information may get separated
• documents often already contain useful boundaries
• chunking can respect these logical structures rather than cutting blindly every arbitrary number of characters
• chunk overlap can help when neighbouring chunks share important information in the boundary
• overlap is another tuning parameter that must balance context preservation with duplication

embedding

• each chunk is then embedded into a vector

• queries are also embedded into vectors, then compared with the corpus

• semantically close chunks should rank highly

• similarity can be measured in different ways, such as cosine similarity, Euclidean distance, etc

• embedding model is part of configuration, and GenAIOps lineage should include:

  • index version
  • chunking configuration
  • embedding model/version
  • retrieval configuration

• if embedding models are changed, existing documents may need to be re-embedded using the new model, and a new compatible index may need to be rebuilt

• metadata storage in embedded chunks can help with filtering, citation, provenance and debugging

• new documents may not necessarily overwrite old ones

• knowledge ingestion itself is a production pipeline