-
Notifications
You must be signed in to change notification settings - Fork 121
tighten wheel size limits, expand CI-skipping logic, other small build changes #804
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
| needs: [conda-python-build, changed-files, compute-matrix-filters] | ||
| uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main | ||
| #if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python | ||
| if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_conda |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These skips have all been disabled for the last year: https://github.com/rapidsai/cuopt/pull/2244
From that PR, I'm really not sure why or what the issue was. But either way, that's a year of running more CI than necessary, which is a waste of money and time. Hopefully we can re-enable this now... it'd help get changes integrated faster and reduce cuopt's demands on our shared CI runners.
| echo "installing 'cvxpy' with cuopt" | ||
| python -m pip install \ | ||
| --constraint "${PIP_CONSTRAINT}" \ | ||
| --extra-index-url=https://pypi.nvidia.com \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This shouldn't ever be necessary.
The only packages installed in this command that aren't on pypi.org are cuOpt packages themselves, which should all have been downloaded from the same CI run before this runs.
|
/ok to test |
| with: | ||
| files_yaml: | | ||
| test_cpp: | ||
| build_docs: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docs builds use GPU runners. Those runners are scarce, so it's worth skipping those jobs when they're not necessary (like when a PR only changes the Helm chart).
| - '!ci/test_wheel*.sh' | ||
| - '!container-builder/**' | ||
| - '!helm-chart/**' | ||
| - '!helmchart/**' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This directory is called helmchart/ (no -). This was wrong before.
| - '!omniverse/**' | ||
| - '!regression/**' | ||
| - '!resources/**' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These directories no longer exist in the repo.
| - '!python/cuopt_self_hosted/**' | ||
| - '!python/cuopt_server/**' | ||
| - '!python/nvcf_client/**' | ||
| test_python_cuopt_server: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm hoping the test_python_wheels: group is granular enough that we don't need test_python_cuopt_server separately.
But please do break it down into more-specific groups if you feel that's doable! The more unnecessary CI runs we can save, the better 😁
| libcuopt = "libcuopt" | ||
|
|
||
| [project.scripts] | ||
| cuopt_cli = "libcuopt._cli_wrapper:main" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needed to be moved up to work around this: rapidsai/pre-commit-hooks#105
| packages: | ||
| # pip recognizes the index as a global option for the requirements.txt file | ||
| - --extra-index-url=https://pypi.nvidia.com | ||
| - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
every package in this group is available from pypi.org
📝 WalkthroughWalkthroughUpdates GitHub Actions changed-file gating and exclusions; adds/export RAPIDS_INIT_PIP_REMOVE_NVIDIA_INDEX and removes NVIDIA extra-index usage in CI scripts; tightens pyproject pydistcheck size limits; adds CUDA-aware size logic to ci/validate_wheel.sh; bumps pre-commit hooks and adds a verify-pyproject-license hook. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In @.github/workflows/pr.yaml:
- Around line 218-254: The test_python_wheels group currently excludes the
python/** path pattern (the '!python/**' entry in the test_python_wheels file
list), which prevents wheel tests from running on Python source changes; remove
the '!python/**' exclusion from the test_python_wheels list so python/** changes
will trigger the wheel job (make the test_python_wheels include python/** the
same way test_python_conda does).
In `@ci/validate_wheel.sh`:
- Around line 11-12: Add an explicit guard before computing RAPIDS_CUDA_MAJOR to
fail fast with a clear message when RAPIDS_CUDA_VERSION is unset or empty: check
if the variable RAPIDS_CUDA_VERSION is set/non-empty (e.g., using parameter
expansion or an if [ -z "${RAPIDS_CUDA_VERSION:-}" ] test), and if it is missing
echo a descriptive error and exit 1; then proceed to set
RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}".
|
@jameslamb Thank you for this awesome PR, now turnaround time for PR runs would be less for so many of those. |
|
@jameslamb I run few tests that are there in docs https://github.com/NVIDIA/cuopt/tree/main/docs/cuopt/source/cuopt-python/lp-qp-milp/examples in cuopt/ci/test_wheel_cuopt_server.sh Line 36 in 7c7e77d
May be we need to add paths to those example files so they will trigger tests, else we might need to move that test to different workflow. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.github/workflows/pr.yaml:
- Line 125: Update the four exclusion patterns that currently use the misspelled
filename string '.coderabbit_review_guid.md' in the GitHub Actions workflow to
the correct '.coderabbit_review_guide.md'; locate occurrences of the literal
'.coderabbit_review_guid.md' in the workflow (seen in the exclusion list
entries) and replace each with '.coderabbit_review_guide.md' so the exclusions
match the .coderabbit.yaml reference.
Ah thanks, hadn't noticed that! For now, I just pushed 0726dec removing That's probably too broad, but it's still an improvement over the current state on |
|
Thank you @jameslamb I will work on a follow-up PR to combine them under one tree so it easier to maintain |
|
It seems that some recent changes (maybe #805) added 10s of MB to the package sizes for Not a complaint, just letting you know @rgsl888prabhu Merging this PR will make such binary-size growth more obvious in the future so it's not a surprise. |
Thank you @jameslamb, this would definitely help. There might be additional growth in size due the grpc issue I was talking to you. In case of conda I think I have to build it as static and in case of wheel, I might need to bundle the .so with wheel. |
|
Oh yeah that's fine! The limit being enforced in CI doesn't mean you can never change it, it's just nice because then you have a line in the diff and reviewers can have a conversation about whether everyone is OK with the size growing. |
|
/merge |
Description
Proposes a batch of miscellaneous build / packaging / CI changes.
Changes
Tightens wheel size limits
Contributes to rapidsai/build-planning#219
CUDA 13 wheels can be as much as 50% smaller than the equivalent CUDA 12 wheels, because of some new compression features in
nvcc.To ensure surprising package-size growth is caught in CI, this PR tightens the limits in the following ways:
libcuoptlimits to{compressed_size} + 50Mi, rounded to the nearest 5Mi{compressed_size} + 10Mi, rounded to the nearest 5MiExpands CI-skipping logic
Contributes to rapidsai/build-planning#243
Tries to avoid unnecessary CI runs by making the CI-skipping rules finer-grained. For example, PRs that only touch
.pre-commit-config.yamlshould now not require any runners with GPUs 😁Removes some reliance on
pypi.nvidia.comContributes to rapidsai/build-planning#241
And removed/updated all relevant references. This project does not need any wheels from
pypi.nvidia.comat build-time or runtime, it can safely avoid searching that index.Its own packages are not available on
pypi.orgthough, so uses ofpypi.nvidia.comin docs are preserved here.Enforces PEP 639 license metadata in
pyproject.tomlContributes to rapidsai/pre-commit-hooks#95
Checklist
Summary by CodeRabbit
New Features
Chores
✏️ Tip: You can customize this high-level summary in your review settings.