branch-4.1: [fix](outfile) handle delete_existing_files before parallel export #61223#61726
Open
suxiaogang223 wants to merge 1 commit intoapache:branch-4.1from
Open
branch-4.1: [fix](outfile) handle delete_existing_files before parallel export #61223#61726suxiaogang223 wants to merge 1 commit intoapache:branch-4.1from
suxiaogang223 wants to merge 1 commit intoapache:branch-4.1from
Conversation
…pache#61223) Issue Number: N/A Related PR: apache#38400 Problem Summary: When `select ... into outfile` uses `delete_existing_files=true`, parallel outfile writers can race on directory cleanup and delete files uploaded by other writers. This PR follows the same FE-side cleanup pattern used by export in apache#38400: remote outfile cleanup is executed once in FE before query execution, and the delete flag is cleared before sink options are sent to BE. This PR also aligns local outfile behavior with export: `file:///` does not support `delete_existing_files=true`, so FE rejects that combination during analysis instead of relying on BE-side cleanup. To reduce duplicated logic, the FE-side parent-directory cleanup used by export/outfile/TVF is refactored into shared `BrokerUtil` helpers. (cherry picked from commit 1576653)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick #61223 to branch-4.1
What problem does this PR solve?
Handle
delete_existing_files=truefor remote outfile once in FE before parallel export, clear the delete flag before sink options are sent to BE, and rejectfile:///withdelete_existing_files=trueto align outfile behavior with export.Cherry-pick commit
15766536326- fix handle delete_existing_files before parallel export ([fix](outfile) handle delete_existing_files before parallel export #61223)