fix: runfiles staging, copy_style, and pip auto-detection for macOS and Linux#3
Open
hwuiwon wants to merge 1 commit intoflyteorg:mainfrom
Open
fix: runfiles staging, copy_style, and pip auto-detection for macOS and Linux#3hwuiwon wants to merge 1 commit intoflyteorg:mainfrom
hwuiwon wants to merge 1 commit intoflyteorg:mainfrom
Conversation
hwuiwon
force-pushed
the
fix/runfiles-staging-and-copy-style
branch
2 times, most recently
from
051d45a to
6d740d3
Compare
Fix several bugs preventing flyte_deploy, flyte_build, and flyte_run (remote) from working on macOS and Linux. Extract all inline Python from Starlark into a shared flyte_runner.py entry point. Bug fixes: - Use explicit copy_style=all instead of None - Remove unused interactive_mode parameter - Stage runfiles into a real temp dir to fix symlink resolution errors - Flatten .venv.pth proto entries into the staging root - Auto-detect extra pip packages missing from the remote container Refactoring: - Move ~315 lines of inline Python from .format() strings into bazel/flyte_runner.py (real, lintable, testable Python) - Each macro now generates a one-line wrapper and passes config via py_binary args attribute - flyte.bzl reduced from 503 to ~200 lines, 0 inline Python - Macro API unchanged — no consumer BUILD file changes needed Signed-off-by: Hwuiwon Kim <hwuiwon.kim@gmail.com>
hwuiwon
force-pushed
the
fix/runfiles-staging-and-copy-style
branch
from
a5949b4 to
4f6a590
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix several bugs preventing
flyte_deploy,flyte_build, andflyte_run(remote) from working on macOS and Linux. Also simplifyflyte.bzlby extracting all inline Python into a sharedflyte_runner.py.Bug fixes
copy_style=None→copy_style="all"—Noneis undocumented and fragileinteractive_mode— the SDK raisesNotImplementedErrorifTrue, passingNoneis a no-oppath.relative_to(cwd)which fails since resolved symlinks are outside cwd. Fixed by staging runfiles into a real temp dir withshutil.copytree(symlinks=False)before callingflyte.init().venv.pthentries point into the original symlink tree and get excluded. Fixed by parsing.venv.pthfiles and copying those directories into the staging rootimportlib.metadatato find non-flyte packages and injecting them viaimage.with_pip_packages()Simplification
Moved ~315 lines of Python embedded in Starlark
.format()strings intobazel/flyte_runner.py. Each macro now generates a one-line wrapper and passes config viapy_binaryargs.flyte.bzlgoes from 503 to ~200 lines with zero inline Python. Macro API is unchanged — no consumer BUILD changes needed.Changes
MODULE.bazel— addaspect_rules_pybazel/flyte_runner.py(new) — shared entry point for run/deploy/buildbazel/BUILD.bazel— add_flyte_runner_libpy_librarybazel/flyte.bzl— replace code generation with CLI-arg-based dispatch