fix: UDF/ODFV source rehydrate (+ Postgres / online cache) - #6655
Open
aniketpalu wants to merge 6 commits into
Open
fix: UDF/ODFV source rehydrate (+ Postgres / online cache)#6655aniketpalu wants to merge 6 commits into
aniketpalu wants to merge 6 commits into
Conversation
…istry reads Rehydrate BatchFeatureView/ODFV callables from body_text (strip leading decorators) before dill.loads to avoid Spark driver exit 139 and cross-Python serve failures. Treat empty Postgres feature_name_columns as SELECT *. Disable registry cache on the online request path so FeatureView state gates see AVAILABLE_ONLINE immediately after materialize. RayTransformation.from_proto left unchanged (follow-up). Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6655 +/- ##
==========================================
+ Coverage 46.36% 46.42% +0.05%
==========================================
Files 414 415 +1
Lines 50072 50206 +134
Branches 7154 7190 +36
==========================================
+ Hits 23218 23307 +89
- Misses 25231 25262 +31
- Partials 1623 1637 +14
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
franciscojavierarceo
requested changes
Jul 29, 2026
franciscojavierarceo
left a comment
Member
There was a problem hiding this comment.
Two blocking concerns from the remote diff:
feature_server.pyand_get_online_request_context()now passallow_cache=Falsefor feature-service and feature-view resolution on every online request. For SQL-backed registries this moves backing-registry I/O into the serving hot path, which can materially regress latency and registry load. Please refresh/invalidate the affected cached definitions when UDF state changes (or isolate and justify this serving behavior separately) instead of globally bypassing the cache.- CodeQL's open finding in
transformation/udf_rehydrate.pyis valid: the decorator-stripping regex has nested unbounded repetition and processes registry-provided source text. Please replace it with a linear parser/line-based approach and add an adversarial regression case.
Member
|
@aniketpalu Please fix CI |
Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid CodeQL ReDoS finding on nested @/newline regex when rehydrating body_text. Add multiline and adversarial @ spam unit tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes three issues found while testing BatchFeatureView UDFs and OnDemandFeatureViews with Spark transformations (SparkApplication / SparkComputeEngine path) against a remote feature server.
All three blocked a clean UDF/ODFV materialize → online path. This PR is scoped to that story. Shared helper
feast.transformation.udf_rehydrateis introduced so the same contract can later be wired intoRayTransformation.from_proto(intentionally not changed here).1. Spark BFV UDF — prefer source over dill (exit 139)
Problem: Materializing a BatchFeatureView with a Python UDF on Spark could segfault the driver (exit 139). Registry stores both dill (
body) and source (body_text/udf_string); the Spark path executed the dill-restored callable, which is unsafe with DataFrame ops on our Spark stack.Change:
SparkTransformationNoderesolves the UDF via sharedresolve_udf: exec trustedudf_stringwhen present; fall back to the existing callable / dill.SparkFeatureBuilderpassesudf_stringthrough.Files:
spark/nodes.py,spark/feature_builder.py,transformation/udf_rehydrate.py2. ODFV / local transformations — source-first
from_protoProblem: ODFV serve (and feast-apply rehydrate from SQL registry) used
dill.loads(body)only. That fails across Python minors (e.g. apply on Spark driver 3.10, serve on feature-server 3.12) and whenbody_textincludes@on_demand_feature_view(...)(NameError / CrashLoop if source is exec'd naively).Change:
PandasTransformation.from_protoandPythonTransformation.from_protouse the sameresolve_udf(strip leading decorators, then exec; dill fallback).Out of scope (follow-up):
RayTransformation.from_protoremainsdill.loadsfor now.Files:
pandas_transformation.py,python_transformation.py,udf_rehydrate.py3. Postgres empty
feature_name_columns→ SELECT *Problem: BatchFeatureView python/pandas mode signals "all columns" with empty
feature_name_columns. Postgres offline store still projected only join keys + timestamps, starving the UDF of input features.Change: When
feature_name_columnsis empty,pull_latest/pull_allusea.*/alias.*(aligned with Ray offline-store semantics).Files:
postgres_offline_store/postgres.py+ unit tests4. Fresh online registry reads
Problem: After materialize completes, online serving could briefly (or stickily) gate on cached
MATERIALIZING/ stale FV metadata.Change: Online request path uses
allow_cache=Falsewhen resolving features / feature services (utils._get_online_request_context, feature server_get_features).Files:
utils.py,feature_server.pyOut of scope
RayTransformation.from_proto/ Ray compute-engine dill paths (follow-up usingudf_rehydrate)Which issue(s) this PR fixes
Found during BYOS Spark UDF/ODFV E2E; no single upstream issue required. Related discussions: Spark UDF materialize reliability, ODFV cross-image Python skew.
Checks
git commit -s)Testing Strategy
Misc