12 - embeddings, vector search and RAG

pipeline

chunk > embed > index > retrieve > augment > generate

embedding

a high-dimensional vector that capture the meaning of words, sentences, or chunks of texts so a computer can process them mathematically

text \overset{embedding}{\to} v \in R^{n}

semantically related texts are placed near each other in the vector space
cf. a token ID is an integer identifier
consider two vectors: $a, b \in R^{n}$ , similarity is measured as the cosine:

\cos θ = \frac{a \cdot b}{| a | | b |}

for normalised embeddings, similar direction imply a high cosine similarity, hence semantical similarity
dot product and euclidean distances are other measures of similarity in different systems
eg: passages have been embedded as: $D_{i} \to v_{i}$ , and the user asks a question: $Q \to q$
- the similarity is calculated: $S_{i} = \cos (q, v_{i})$
- vector/semantic search retrieves passages $D_{i}$ with the highest similarities
consider a large PDF with sections
it can be represented as a single vector, but it is more useful to store the sections, hence the PDF is divided into chunks
each chunk gets its own embedding: $C_{i} \to v_{i}$
retrieval operates at the chunk level
there is a trade-off between chunk size and context

chunk overlap

Chunk 1:
A B C D E

Chunk 2:
      D E F G H

Chunk 3:
            G H I J K

chunk overlap preserves context around boundaries
excessive overlap creates more chunks, embeddings, storage, duplicated retrieval, and higher processing cost
finding the vectors closest to the query vector is essentially a nearest-neghbour search problem
at large scale, systems commonly use approximate nearest-neighbour search techniques
meta data often contains important information such as version, date, etc, and can become vital for narrowing retrieval and maintaining provenance
keyword search is still useful for exact identifiers such as serial numbers, error codes, IDs, exact names, unusual acronyms, etc
hybrid search combines both vector and keyword search
retrieval simply returns relevant information, not generate an answer

retrieval-augmented generation (RAG)

retrieval: find relevant information
augmented: add that information to the model's context
generation have the LLM produce and answer using that context

RAG has an offline and an online side
the offline phase, or indexing, is done when preparing the knowledge source
the online phase, or querying, is when the user asks something

 OFFLINE

Documents
   ↓
Chunk
   ↓
Embed
   ↓
Index
   │
   │
   │             ONLINE
   │
   │        User question
   │              ↓
   │            Embed
   │              ↓
   └───► Search
		  ↓
	relevant chunks
		  ↓
    question + chunks
		  ↓
		 LLM
		  ↓
		answer

grounded answers are those supported by he provided/retrieved context
eg: retrieve: Recalibration must occur after replacement of the high-voltage supply. but the LLM answers: The detector should be recalibrated after replacing the high-voltage supply.
RAG doesn't necessarily make hallucinations disappear
failure can occur at multiple stages
retrieval failure happens when correct information exists, but the wrong chunks are retrieved
generation failure happens when the correct information is retrieved, but the LLM misinterprets or ignores it
RAG quality cares about the quality the retrieval as well as the generation
retrieval recall questions whether the relevant material was found
retrieval precision checks how much of what was retrieved was actually useful
thus, there is another trade-off
top-k retrieval retrieves the top $k$ results, so $k$ is a RAG configuration/hyperparameter
similarity thresholds also help, but this has another trade-off
reranking is when an initial model retrieves a number of chunks, and then a more sophisticated model examines and ranks them more carefully
if fresh knowledge arrives, it can be broken into chunks, embedded, and the index updated, without requiring the LLM to be retrained