Intent-Driven Dynamic Chunking: keeping answers together

Intent-Driven Dynamic Chunking (IDC) grew out of a practical search problem. While working on semantic search for SAP Fiori technical documentation, I kept coming back to the way documents were being split. A useful answer could be spread across several chunks, while a single retrieved chunk could contain a lot of material the reader did not need.

The code is open source: https://github.com/unseen1980/IDC

Preprint paper: https://arxiv.org/abs/2602.14784

Chunking often starts with a length: split every few hundred tokens, perhaps add some overlap, then index the results. It is straightforward and often a sensible baseline. But the boundary is chosen before considering what someone might ask. An explanation can be cut in half simply because the window has ended.

IDC starts with the document and predicts questions it could answer. Those questions guide the segmentation. The aim is to keep enough context together for a useful answer, without surrounding it with unrelated information. The questions are generated during indexing; the system does not need to predict the exact query a person will eventually type.

I use a language model to generate the candidate questions, then embed those questions and the document’s sentences. Candidate chunks are scored by their alignment with the predicted questions, with penalties for excessive length and unnecessary boundaries. Dynamic programming chooses the segmentation that best satisfies that objective across the document.

That last part matters. Asking a language model to insert boundaries directly makes the segmentation dependent on its generated output. Here, the language model proposes information needs, while the algorithm chooses boundaries over the original sentences. The text is preserved, and the trade-offs in the scoring function are explicit. “Optimal” means optimal for that objective, of course; it does not guarantee that every future reader will get the answer they need.

The evaluation covers six question-answering datasets drawn from news, Wikipedia, academic papers, and technical documentation. All methods use the same preprocessing, embedding model, and retrieval setup, so the comparison focuses on the segmentation. On the long arXiv document, top-ranked answer retrieval rose from 0.400 for the best baseline to 0.667 with IDC. On the Fiori documentation, it rose from 0.333 to 0.533.

IDC also produced fewer chunks. For the arXiv document, it created 39 rather than 83 with fixed-length splitting, while keeping 93.3% of answers within a single chunk. Fewer segments are useful, but keeping the answer intact is the more interesting part. The additional work happens offline during indexing, rather than on each search request.

There are limits to these results. Several evaluation sets are small, and the method depends on the quality and coverage of the generated questions. On Qasper, it tied paragraph-based splitting on top-1 retrieval and performed worse on the other reported retrieval measures. When the existing structure already fits the questions, simple paragraph boundaries can be enough.

For me, the useful result is that segmentation deserves the same attention as the retriever. Before changing the search model, it is worth checking whether the units being retrieved contain a complete answer in the first place.

Thanks for reading - Christos