refactor: reusable workflows accept config-file input (PoC)#6126
Open
Eren-Jeager123 wants to merge 2 commits into
Open
refactor: reusable workflows accept config-file input (PoC)#6126Eren-Jeager123 wants to merge 2 commits into
Eren-Jeager123 wants to merge 2 commits into
Conversation
Add optional `config-file` input to the 4 PyTorch-relevant reusable workflows (sanity, security, telemetry, EFA). When provided, the reusable workflow reads all metadata directly from the YAML config file instead of requiring callers to parse and forward 10+ individual input parameters. This enables a future where matrix jobs can pass per-leg config file paths without the GitHub Actions limitation of matrix outputs only exposing the last-finishing leg. Backward compatible: existing callers that pass individual inputs continue to work unchanged (the else branch passes them through). Updated pr-pytorch-ec2-cuda.yml as the first caller to use the new config-file interface, eliminating the verbose load-config→parse→forward pattern for test jobs.
80e8723 to
dd3b901
Compare
Make config-file a required input for the 4 shared reusable workflows (sanity, security, telemetry, EFA). Each reusable workflow now reads its own metadata from the YAML config file internally, eliminating the need for callers to parse and forward 10+ individual parameters. Migrate all 37 caller workflows (PR, autorelease, dispatch) to pass config-file instead of individual metadata inputs. This removes ~440 lines of boilerplate forwarding. Add a Dockerfile comment to trigger the PyTorch EC2 CUDA PR workflow for CI validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
config-fileinput to 4 PyTorch-relevant reusable workflows (sanity, security, telemetry, EFA)config-fileis provided, the reusable workflow reads metadata from the YAML config directly — callers no longer need to parse and forward 10+ individual parameterspr-pytorch-ec2-cuda.ymlas PoC — eliminates the verboseload-config→parse→forwardpatternMotivation
GitHub Actions limitation: when a matrix job has outputs, only the last-finishing matrix leg gets its outputs exposed. If multiple PyTorch versions change in the same PR, downstream reusable workflow calls only receive metadata from one version.
By making reusable workflows self-sufficient (reading their own config), each matrix leg can pass its own
config-filepath without shared outputs clobbering each other.Workflow & Caller Inventory
13 Reusable Workflows
reusable-sanity-tests.ymlreusable-security-tests.ymlreusable-telemetry-tests.ymlreusable-efa-tests.ymlreusable-vllm-model-tests.ymlreusable-vllm-upstream-tests.ymlreusable-vllm-sagemaker-tests.ymlreusable-vllm-omni-model-tests.ymlreusable-sglang-model-tests.ymlreusable-sglang-upstream-tests.ymlreusable-sglang-sagemaker-tests.ymlreusable-release-image.ymlreusable-sagemaker-xgboost-integ-tests.ymlPR Workflows (20 callers)
pr-pytorch-ec2-cuda.yml(this PoC)pr-pytorch-ec2-cpu.ymlpr-pytorch-sagemaker-cpu.ymlpr-pytorch-sagemaker-cuda.ymlpr-base-v1.ymlpr-base-v2.ymlpr-vllm-ec2-amzn2023.ymlpr-vllm-ec2.ymlpr-vllm-sagemaker-amzn2023.ymlpr-vllm-sagemaker.ymlpr-vllm-hyperpod-amzn2023.ymlpr-vllm-omni-ec2-amzn2023.ymlpr-vllm-omni-sagemaker-amzn2023.ymlpr-sglang-ec2-amzn2023.ymlpr-sglang-ec2.ymlpr-sglang-sagemaker-amzn2023.ymlpr-sglang-sagemaker.ymlpr-ray-ec2-cpu.ymlpr-ray-ec2-gpu.ymlpr-ray-sagemaker-cpu/gpu.ymlpr-sagemaker-xgboost.ymlAutorelease Workflows (14 callers)
autorelease-pytorch-ec2-cuda.ymlautorelease-pytorch-ec2-cpu.ymlautorelease-pytorch-sagemaker-cpu.ymlautorelease-pytorch-sagemaker-cuda.ymlautorelease-vllm-ec2-amzn2023.ymlautorelease-vllm-ec2.ymlautorelease-vllm-sagemaker-amzn2023.ymlautorelease-vllm-sagemaker.ymlautorelease-vllm-hyperpod-amzn2023.ymlautorelease-vllm-omni.ymlautorelease-sglang-ec2.ymlautorelease-sglang-sagemaker.ymlautorelease-ray.ymlautorelease-base.ymlDispatch Workflows (3 callers)
dispatch-partner-release.ymldispatch-release-sagemaker-xgboost.ymlDesign
Blast Radius
pr-pytorch-ec2-cuda.ymlis triggered by this PR (it watches its own path)reusable-sanity/security/telemetry/efain their path triggerselsebranch forwards them unchangedTest plan
PR - PyTorch EC2 CUDAworkflow triggers on this PR