Chunking, Embeddings and Vector Search

Retrieval quality decides RAG quality. Three choices matter most: how you chunk, which embedding model you use and how you search.

Parsing and cleaning

Garbage in, garbage out. Extract text properly from PDFs and DOCX files, remove headers, footers and page numbers, and keep headings, because they carry meaning. Tables and scanned pages need extra care.

Chunking

• Chunk size: too small loses context, too big dilutes the match and wastes tokens. A few hundred words is a common starting point, then tune it on your own questions.
• Overlap: repeat a little text between neighbouring chunks so ideas are not cut in half.
• Split on natural boundaries, such as headings and paragraphs, when you can.
• Store metadata: source, title, section, date, language, access rights.

Embeddings

An embedding model turns text into a vector. Texts with similar meaning land near each other. Use the same embedding model for documents and questions, and remember that changing the model means re-embedding everything.

Similarity search

To retrieve, embed the question and find the closest chunk vectors, usually by cosine similarity. Return the top-k chunks (for example the 3 to 8 best).

function cosine(a, b) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

const ranked = chunks
  .map((c) => ({ ...c, score: cosine(queryVector, c.vector) }))
  .sort((x, y) => y.score - x.score)
  .slice(0, 5);

A plain loop like this is fine for a few thousand chunks. For more, use a vector database with an index.

Build the prompt

Answer the question using only the sources below. If the sources do not contain the answer, say so.
Cite the source number like [1].

<sources>
[1] {{chunk 1}}
[2] {{chunk 2}}
</sources>

Question: {{question}}