[improvement](fe) Enhance FE meta service request validation#63782
Closed
CalvinKirs wants to merge 2 commits into
Closed
[improvement](fe) Enhance FE meta service request validation#63782CalvinKirs wants to merge 2 commits into
CalvinKirs wants to merge 2 commits into
Conversation
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: FE meta service endpoints are used by FE nodes for metadata synchronization and coordination. This change strengthens internal FE caller validation by carrying the cluster token on FE-to-FE meta requests and validating it on the receiver side, while preserving a temporary legacy switch for rolling upgrades.
### Release note
FE meta service internal requests now include cluster token validation by default. During rolling upgrades from older versions, set enable_meta_service_legacy_node_ident_auth=true temporarily on upgraded FEs if old FEs still need to call these endpoints without token headers. Disable it after all FEs are upgraded. The /dump endpoint now always checks HTTP user credentials, and /put only accepts the configured FE HTTP port.
### Check List (For Author)
- Test: Unit Test, Manual test
- mvn -pl fe-core -am -DskipUT=false -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=org.apache.doris.httpv2.meta.MetaServiceTest test
- mvn -pl fe-core -am -DskipUT=false -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=org.apache.doris.httpv2.meta.MetaServiceTest,org.apache.doris.common.util.HttpURLUtilTest test
- ./run-fe-ut.sh --run org.apache.doris.httpv2.meta.MetaServiceTest,org.apache.doris.common.util.HttpURLUtilTest
- Manual: started FE on HTTP 26030 and query port 27030; verified no-token FE meta request returns business code 401, and token-carrying /image?version=155292 returns HTTP 200.
- Behavior changed: Yes. FE meta service endpoints require the cluster token by default; legacy header-only node identity fallback is available only when enable_meta_service_legacy_node_ident_auth=true. /dump now always checks HTTP user credentials. /put rejects ports other than Config.http_port.
- Does this need documentation: Yes. Document rolling-upgrade use of enable_meta_service_legacy_node_ident_auth and cluster-token validation for FE meta service requests.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 31382 ms |
Contributor
TPC-DS: Total hot run time: 171487 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run nonConcurrent |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29881 ms |
Contributor
TPC-DS: Total hot run time: 173060 ms |
Contributor
ClickBench: Total hot run time: 25.12 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: FE meta service endpoints are used for FE-to-FE metadata synchronization and coordination. This PR enhances internal request validation by carrying the cluster token on FE meta requests and validating it on the receiver side. It also adds a temporary compatibility switch for rolling upgrades from FE nodes that do not send the token yet.
Release note
FE meta service internal requests now validate the cluster token by default. During rolling upgrades from older versions, set
enable_meta_service_legacy_node_ident_auth=truetemporarily on upgraded FEs if old FEs still need to call these endpoints without token headers. Disable it after all FEs are upgraded. The/dumpendpoint now always runs the HTTP user check, and/putonly accepts the configured FE HTTP port.Check List (For Author)
./run-fe-ut.sh --run org.apache.doris.httpv2.meta.MetaServiceTest,org.apache.doris.common.util.HttpURLUtilTestmvn -pl fe-core -am -DskipUT=false -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=org.apache.doris.httpv2.meta.MetaServiceTest,org.apache.doris.common.util.HttpURLUtilTest test/image?version=155292returns HTTP 200.enable_meta_service_legacy_node_ident_auth=true./dumpnow always runs the HTTP user check./putrejects ports other thanConfig.http_port.enable_meta_service_legacy_node_ident_authand cluster-token validation for FE meta service requests.