RAG (Retrieval-Augmented Generation)

Embeddings and cosine similarity

In a RAG system, every document fragment is turned into an embedding, and so is the user's question. Cosine similarity measures the angle between two vectors: the closer to 1, the more alike their meanings. That is how you find the fragments most relevant to a question, even when they share no words.

The absolute similarity value is misleading: it depends on the model, the language and how the texts were written. What is useful is comparing the gap between good and bad results for real questions. That is why similarity thresholds should be calibrated with data, not set by eye.

Vector databases

A vector database stores embeddings and lets you quickly find the ones closest to a given vector, even among millions of records, thanks to approximate indexes (ANN). Some examples are pgvector (a PostgreSQL extension), Qdrant, Pinecone, Weaviate and Chroma.

You don't always need a dedicated one. With a few thousand fragments, computing the similarity against all of them (an exhaustive search) is fast enough and saves infrastructure. A vector database starts to pay off when the volume grows or when you need filters and low latency at scale.

Chunking

Chunking means splitting documents into fragments (chunks) before generating their embeddings. A whole document mixes too many topics into a single vector, and a fragment that is too small loses its context. The typical size ranges from a few hundred to a couple of thousand characters, with some overlap between consecutive fragments so that no idea is cut in half.

The best strategy depends on the content: by paragraph or section for documentation, and by record for structured data (a ticket, an order). When you index records, it pays to turn them into readable summaries ("Order 1042 from Maria, shipped on 3 March, pending return") rather than dumping their columns: an embedding captures meaning, not structure.

Hybrid search

Hybrid search combines semantic search (using embeddings) with traditional keyword search (such as BM25). Each one covers the other's weak spots: semantic search understands synonyms and paraphrases but fails with exact codes, references and proper names; lexical search works the other way round.

In a customer support application, "my parcel hasn't arrived" is resolved well by meaning, but "order 1042" or a specific error code needs an exact match. The results of both searches are merged, for example with Reciprocal Rank Fusion, which combines the positions from each list.

Reranking

Reranking is a second step that reorders the retrieved candidates with a more accurate model (often a cross-encoder), which evaluates each question-fragment pair together instead of comparing precomputed vectors.

The typical pattern is to retrieve many candidates cheaply (say, 50) and have the reranker pick the best ones (say, 5), which are the ones that go into the context. It improves precision but adds latency and cost, and it only makes sense once you have checked that the problem is the order of the results and not that the information isn't there.

Knowledge graphs

A knowledge graph represents information as entities (customers, products, orders) and the relationships between them ("customer X bought product Y", "product Y has issue Z"). In the context of RAG (GraphRAG), it lets you answer questions that require following relationships, not just finding similar fragments.

It is useful when the question is relational ("which customers who bought this product have opened tickets this month?"), where similarity search falls short. The price is high, because building and maintaining the graph is complex. In most cases it is wiser to start with vector RAG and answer structured questions with SQL queries.