A RAG system answers questions based on your own documents. To do so, documents are searched not as wholes but in sections. How those sections are cut is an inconspicuous decision with large effect: it determines what search can find at all.
Why split at all
An embedding condenses a text into one vector. The longer the text, the blurrier that summary, and a precise question no longer finds the matching passage. At the same time the model’s context window is limited, and every inserted token costs. Chunking looks for the point where sections are small enough for precise search and large enough for understandable context.
Common strategies
Fixed size: sections of a set number of tokens, often with overlap so that sentences at the boundary are not torn apart. Simple, but blind to structure. Structure-based: cutting along headings, paragraphs, tables or sections, which works much better for well-structured documents such as contracts, manuals or laws. Semantic: cutting where the topic changes, measured by the similarity of neighbouring sentences. Hierarchical: small sections for search, but the parent section as context for the model.
Metadata makes the difference
A section without provenance is hard to use. Every chunk needs document title, heading path, date, version, tenant and permissions. They enable filters, such as current versions only or only the user’s own tenant, and they let the model cite sources correctly. Permissions belong in the search filter, not in the prompt.
Typical mistakes
Tables torn apart row by row, losing their header. Definitions separated from the section using them. PDFs with headers and footers appearing as noise in every chunk. And a chunk size chosen once and never reviewed.
Decide with evaluation
Which strategy works best depends on the documents and the questions. The only reliable method is evaluation with a fixed set of realistic questions: does search find the right sections, and is the answer correct? Changing chunking without measuring improves a metric you do not know. Storing and searching the sections is covered under vector database.