feat(server): Remote Materialization - #6649
Open
aniketpalu wants to merge 7 commits into
Open
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #6590 (clean single-commit history with proper DCO).
Add
?async=truequery parameter support to the existing/materializeand/materialize-incrementalendpoints. When set, materialization runs in a background thread and the endpoint returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Also adds?force=trueto allow operators to override stuck MATERIALIZING state (e.g. after a crashed SparkApplication pod).Client-side (SDK)
store.materialize(..., remote=True)delegates to the feature server(URL/TLS derived from
online_storeconfig)wait=Falsereturns immediately after triggeringstore.poll_materialization()for status polling via registry stateServer-side
?async=trueon existing/materializeand/materialize-incremental?force=trueto skip the 409 concurrency guardRegistry
apply_materialization()now set FV state toAVAILABLE_ONLINE(parity with file-based registry)Which issue(s) this PR fixes
Addresses #4526
Checks
git commit -s)Testing Strategy
Misc