refactor: version PyTorch directory structure for multi-version support

[Eren-Jeager123]

Summary

Restructure PyTorch to support maintaining multiple versions concurrently (1-year support window). Versioned directories isolate each version's build artifacts. Workflows are version-agnostic — no new workflow files needed when adding a future version.

1. Versioned directory structure

docker/pytorch/2.11/
├── Dockerfile.cuda
├── Dockerfile.cpu
├── versions-cuda.env
├── versions-cpu.env
├── cuda/pyproject.toml
├── cuda/uv.lock
├── cpu/pyproject.toml
└── cpu/uv.lock

Config files versioned by name: pytorch-2.11-ec2-cuda.yml, etc.
Shared scripts (scripts/pytorch/) and tests (test/pytorch/) stay in place.

2. PR workflows — configs matrix (no output clobbering)

gatekeeper → detect-versions → build-images / unit-test / sanity / security / telemetry / single-gpu / efa

Key design: detect-versions reads each version's config YAML and outputs a rich configs matrix where each entry carries all metadata fields. All downstream jobs iterate over this matrix and pass matrix.* fields directly to reusable workflows as individual inputs.

This solves the GitHub Actions limitation where matrix job outputs only expose the last-finishing leg's values — metadata travels with the matrix, not as job outputs. Reusable workflows are completely unmodified.

# detect-versions outputs:
#   configs: [{"version":"2.11", "framework":"pytorch_runtime", "framework_version":"2.11.0", ...}]

sanity-test:
  strategy:
    matrix:
      include: ${{ fromJson(needs.detect-versions.outputs.configs) }}
  uses: ./.github/workflows/reusable-sanity-tests.yml
  with:
    framework: ${{ matrix.framework }}
    framework-version: ${{ matrix.framework_version }}
    ...

Version detection combines both path patterns in a single grep to avoid missing versions:

git diff --name-only origin/main...HEAD \
  | grep -oP '(?:docker/pytorch/|pytorch-)\K[0-9]+\.[0-9]+'

3. Autorelease workflows — simplified dispatch

workflow_dispatch:
  inputs:
    version:
      description: "PyTorch version to release (e.g., 2.11)"

Human dispatch now accepts just the version string (e.g., 2.11) instead of a full config file path. Cron schedules map version → config file internally.

4. Dockerfile ARG for version paths

Dockerfiles use ARG DLC_PYTORCH_VERSION=2.11 instead of hardcoding:

COPY docker/pytorch/${DLC_PYTORCH_VERSION}/cuda/pyproject.toml /tmp/build/

5. Additional improvements

• Unit tests extracted into separate job (parallel with sanity/security/telemetry)
• upload_cached_wheels.sh accepts Dockerfile path as parameter
• test_versions.py reads DLC_PYTORCH_VERSION env var to locate versioned env files
• No changes to reusable workflows — zero blast radius to other frameworks

Adding PyTorch 2.12

1. Copy docker/pytorch/2.11/ → docker/pytorch/2.12/, update version pins
2. Create 4 config files (pytorch-2.12-{ec2,sagemaker}-{cuda,cpu}.yml)
3. Autorelease: add one cron line + one case entry per workflow
4. PR workflows: auto-detect from diff — no changes needed

Test plan

• PR workflow detects version from docker/pytorch/2.11/** and pytorch-2.11-* changes
• Matrix builds all changed versions in parallel (CI tags include version — no collision)
• Configs matrix passes correct metadata per version to reusable workflows
• Unit tests run as separate job per version
• Reusable workflows (vLLM, SGLang, Ray, Base) work unchanged
• Autorelease workflow_dispatch with version input works
• DLC_PYTORCH_VERSION build-arg resolves correctly in Dockerfile COPY paths

[sirutBuasai]

Why do we need to determine pytorch version and then construct a config file name based off that? Why dont we simply use the config file as input, or simple do

case "$CRON" in
    "00 17 * * 1,3") CONF_FILE="<path_to_pt2.11_cpu>" ;;
    # "10 17 * * 1,3") CONF_FILE="<path_to_pt2.12_cpu>"  ;;
    *) echo "::error::Unknown cron: $CRON"; exit 1 ;;
esac

Then any usage of the actual Pytorch version field can just be derived from the config file using the output from load_config job.

[Eren-Jeager123]

Good point I forgot we can use the one from load_config

[sirutBuasai]

ditto

[sirutBuasai]

ditto

[sirutBuasai]

ditto

[sirutBuasai]

Multi-version PR silently tests only one version. so if we make a change for both PT 2.11 and PT 2.12, we would expect both jobs to run. The current logic only run one PR

[Eren-Jeager123]

I thought that it's rare to change more than 1 version each time previously; Ya however I will change this.

[sirutBuasai]

we should be able to parallelize the check-changes with load-config. Serializing this slows down the PR a little bit due to these two jobs being extremely small but the job startup each takes 30s

[sirutBuasai]

Take a look at https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idstrategymatrix matrix job submission. This would allow us to define a list of configurations that the workflow can in sets in parallel of each other.

https://github.com/aws/deep-learning-containers/blob/main/.github/workflows/dispatch-vllm-benchmark.yml is also another good example that we have that uses matrix run mechanism

[sirutBuasai]

this is too broad, this means that a change in cuda dockerfile will trigger cpu as well

[sirutBuasai]

With matrix running, we'll need to add some kind of version string to the ci image. This is because if PT 2.11 and PT 2.12 runs at the same time, we run into naming collision

[sirutBuasai]

ditto

[sirutBuasai]

ditto

[sirutBuasai]

ditto

[sirutBuasai]

If you're going to have to declare the version across the file, we should use ARG PYTORCH_VERSION=2.11 then use COPY .../${PYTORCH_VERSION}/...