docs: document the length chunker boundary settings - #475
Merged
Conversation
Add content_chunker.length.boundary.enabled / lookback_percent / lookahead_percent to the system.properties reference table in config/search-semantic.rst, next to the existing content_chunker.length.overlap entry, across all 7 languages (en, ja, de, fr, es, ko, zh-cn). Also add a note explaining that content_chunker.length.chunk_size is now a target rather than a hard ceiling: - Each cut moves to a suitable break within the lookback/lookahead search window, preferring a line break or sentence end over a clause separator or space, and those over a script change. Only the cut point moves and no character is dropped, so concatenating a document's chunks still reproduces its content exactly. - The forward search may overshoot chunk_size by up to lookahead_percent. A second, independent overshoot of up to 32 characters can occur when a cut would otherwise land inside a grapheme cluster; that one ignores lookahead_percent and can happen even when it is 0. The two never occur on the same cut, so the worst case at the shipped defaults is about 841 characters. - Chunks can also come in up to lookback_percent shorter, so a document may produce slightly more chunks than before (see content_chunker.max_chunks_per_document). - Setting boundary.enabled=false, or both percentages to 0, restores the previous exact fixed-length behavior. The chunk_size row now points at that note, so a reader scanning the table does not miss the target-versus-ceiling caveat. Verified: docutils parses all 7 files with a message set identical (line-number-normalized) to the pre-change baseline; defaults, ranges and figures (true, 20 (0-50), 5 (0-25), 32, 841) are unchanged and consistent across languages; every touched line stays within each file's pre-existing maximum display width (East-Asian-Width aware).
marevol
force-pushed
the
feature/length-chunker-boundary-settings
branch
from
August 10, 2026 23:49
82d1ae1 to
386bffb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the three
content_chunker.length.boundary.*settings added by the boundary-aware chunking change infess, and the resulting change to whatcontent_chunker.length.chunk_sizemeans.Added to
15.8/config/search-semantic.rstin all seven languages (en, ja, de, fr, es, ko, zh-cn), as new rows next to the existingcontent_chunker.length.overlapentry plus one note after the table. All seven already carried a fully parallel translation of that table, so leaving five of them untouched would have left them describingchunk_sizeas an exact per-chunk character count.What is documented
content_chunker.length.boundary.enabledtruecontent_chunker.length.boundary.lookback_percent20content_chunker.length.boundary.lookahead_percent5The note covers:
chunk_sizeis now a target rather than a hard ceiling. The forward sentence search may overshoot by up tolookahead_percent; separately, and not governed bylookahead_percent, moving off a grapheme cluster may overshoot by up to 32 characters. The two never occur on the same cut, so the worst case is about 841 characters at the shipped defaults.lookback_percentshorter, so a document may produce slightly more chunks -- relevant tocontent_chunker.max_chunks_per_document.boundary.enabled=false, or both percentages at0, restores the previous fixed-length behaviour.The
chunk_sizerow now points at that note, so a reader scanning the table does not miss the target-versus-ceiling caveat.Notes for review
The tier precedence and every figure were checked against the implementation, and the numbers are identical across all seven languages. Each file parses with docutils with exactly the same message set as before the change.
The non-English text was not reviewed by a native speaker; corrections welcome, particularly for the de, fr, es and ko notes.