docs: correct deprecated ChunkSize and document backup chunking variables - #473
Open
g-despot wants to merge 5 commits into
Open
docs: correct deprecated ChunkSize and document backup chunking variables#473g-despot wants to merge 5 commits into
g-despot wants to merge 5 commits into
Conversation
…bles ChunkSize has had no effect since its removal in core, but the page documented it as a working option with a default, minimum and maximum. Mark it deprecated and point to BACKUP_CHUNK_TARGET_SIZE. Document BACKUP_MIN_CHUNK_SIZE, BACKUP_CHUNK_TARGET_SIZE and BACKUP_SPLIT_FILE_SIZE, which were undocumented, including how their values are raised against each other. Document the location parameter for text2vec-google. No version markers: these all landed across several stable lines, so no single version is correct for every supported release.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
…P_MAX_INDIVIDUAL_FILES Chunking runs on every backup, not only incremental ones, so the block no longer belongs under Incremental Backups > How it works. Move it to a new Technical Considerations > Chunking and file splitting section, modelled on Skip the storage access check. - Document BACKUP_MAX_INDIVIDUAL_FILES (default 100, added in v1.37.14 and v1.38.7, absent from 1.36) and note it is runtime-configurable via the backup_max_individual_files override key, unlike the three size variables which are startup-only. - Reframe BACKUP_MIN_CHUNK_SIZE as a floor: lowering it has no effect once at least BACKUP_MAX_INDIVIDUAL_FILES files already exceed it. - Record that chunks carrying a split-file part are not topped up, so large-file backups produce chunks smaller than BACKUP_CHUNK_TARGET_SIZE. - Note that unlimited/nolimit are accepted, which is how splitting is disabled. - Note that an incremental backup shrinks the individual-file budget by the number of files reused from the base, so it spans a backup chain. - Repoint the deprecated ChunkSize row at the new section anchor. - Add backup_max_individual_files to the runtime overrides table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VLMB1FhdGFGcdg63uVHxNY
…feedback Correct one factual error and apply the review gates' required changes to the Chunking and file splitting section. - Split-file chunks: their size derives from BACKUP_SPLIT_FILE_SIZE, not from BACKUP_CHUNK_TARGET_SIZE. Each part is larger than half the split size and no larger than it, so at the defaults these chunks are far larger than the target, not smaller as previously stated. - Shared chunks are re-uploaded on every incremental backup regardless of whether their contents changed, because only single big-file chunks can be skipped. - Note that getting an own chunk is necessary but not sufficient for reuse: Weaviate only reuses files it treats as immutable. - Standardize on "qualifying size" for the size threshold, name BACKUP_SPLIT_FILE_SIZE where the split threshold is described, and split the floor-vs-dial paragraph so the definition stands on its own. - State that BACKUP_MAX_INDIVIDUAL_FILES is changed without a restart through the backup_max_individual_files runtime configuration key. - Move the backup-chain budget paragraph above the admonition so the section ends on it, and note that the shard-file-count fallback compares against the reduced budget on an incremental backup. - Link shard on first mention, retitle the admonition to cover the fallback bullet, drop the negative version marker, and rewrite the How it works remnant so it no longer conflicts with or duplicates the new section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VLMB1FhdGFGcdg63uVHxNY
The lower bound on a split part was stated with a strict inequality. Because the part count is a ceiling division and the final part carries only the remainder, the last part can land on exactly half of the split size: a file of split size plus one byte splits into two parts of 26843545601 and 26843545600 bytes, the second being exactly 25GiB at the default 50GiB split. Say "at least half of the split size" instead of "larger than half", and soften "equal parts" to "roughly equal", since parts can differ by up to one byte per part. The upper bound and the paragraph's conclusion are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VLMB1FhdGFGcdg63uVHxNY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes documentation that is wrong or missing for users on current releases. No 1.39 content.
ChunkSizewas documented as a working backup option with a default, minimum and maximum. It has had no effect since it was removed in core (present as deprecated in v1.38.5 and every supported line). Now marked deprecated, pointing atBACKUP_CHUNK_TARGET_SIZE.BACKUP_MIN_CHUNK_SIZE,BACKUP_CHUNK_TARGET_SIZEandBACKUP_SPLIT_FILE_SIZEwere undocumented. Added, with a note on how their values are raised against each other.text2vec-googlegained alocationparameter that was never documented. Optional, defaults tous-central1.No version markers anywhere: each of these landed across several stable lines, so no single version is correct for every supported release.
Site build passes.