feat(capture): add file rotation support for Parquet format#1707
Draft
scottopell wants to merge 1 commit intosopell/observer-bugfixesfrom
Draft
feat(capture): add file rotation support for Parquet format#1707scottopell wants to merge 1 commit intosopell/observer-bugfixesfrom
scottopell wants to merge 1 commit intosopell/observer-bugfixesfrom
Conversation
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Jan 14, 2026
scottopell
force-pushed
the
sopell/observer-bugfixes
branch
from
418a6e2 to
e192e68
Compare
scottopell
force-pushed
the
sopell/add-parquet-rotation
branch
from
fc572d4 to
83e2256
Compare
Add rotation API to CaptureManager that allows rotating to new output files without stopping the capture. This enables long-running capture sessions to produce multiple readable Parquet files with valid footers. Changes: - Add RotationRequest/RotationSender types for async rotation requests - Add start_with_rotation() that spawns event loop and returns (sender, JoinHandle) - Add replace_format() to StateMachine for IO-agnostic format swapping - Add rotate() trait method stub to OutputFormat (returns error by default) - Add rotate_to() inherent method on parquet Format<BufWriter<File>> - Return JoinHandle for graceful shutdown and data flush guarantees The rotation flow: 1. Caller sends RotationRequest with new file path via RotationSender 2. CaptureManager creates new file and format 3. StateMachine.replace_format() flushes and swaps formats 4. Old format is closed (writing Parquet footer) 5. Response sent back to caller The returned JoinHandle allows callers to await the capture task to ensure all buffered metrics are flushed before process exit. Combines commits d0aebbe and 0542dd6 from sopell/expose-observer-public. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
scottopell
force-pushed
the
sopell/add-parquet-rotation
branch
from
83e2256 to
8bfd82e
Compare
scottopell
force-pushed
the
sopell/observer-bugfixes
branch
from
e192e68 to
618001c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Related issues
A list of issues either fixed, containing architectural discussions, otherwise relevant
for this Pull Request.
Additional Notes
Anything else we should know when reviewing?