fix(ci): pin kubernetes<36 in github workflow#4
Open
vicoooo26 wants to merge 2 commits into
Open
Conversation
|
🎉 Welcome to the Kubeflow Trainer! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
vicoooo26
force-pushed
the
fix/kubernetes-version-pin
branch
from
July 24, 2026 09:01
2367b8e to
2d7aa4b
Compare
There was a problem hiding this comment.
Pull request overview
This PR stabilizes Python CI by pinning the kubernetes dependency to a pre-36.x version range to avoid known unit test failures introduced by the kubernetes 36.0.0 client’s switch to Pydantic-based models.
Changes:
- Pin
kubernetes<36in the Python Test workflow dependency install step. - Add an in-workflow TODO documenting why the pin exists and what work is needed to remove it.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
vicoooo26
force-pushed
the
fix/kubernetes-version-pin
branch
3 times, most recently
from
July 24, 2026 10:43
f3cd174 to
a52f42a
Compare
kubernetes 36.0.0 regenerated the sync client with Pydantic models, breaking the FakeResponse deserialization hack used in SDK unit tests. Pin kubernetes<36 in all CI workflows that install the Python SDK (test-python, integration-tests, e2e-test-train-api, test-example-notebooks) to avoid impacting SDK release. Add a TODO for future Pydantic work. Signed-off-by: Vico Chu <vico24826@gmail.com>
vicoooo26
force-pushed
the
fix/kubernetes-version-pin
branch
from
July 24, 2026 10:46
a52f42a to
c8c1cc0
Compare
vicoooo26
changed the title
vicoooo26
force-pushed
the
fix/kubernetes-version-pin
branch
from
July 24, 2026 11:45
2232b90 to
889b982
Compare
JAX worker pods were OOMKilled (exit 137) with 3Gi memory limit. Increase to 5Gi to accommodate XLA compilation and training memory usage. GitHub-hosted runners have 16GB RAM, so 2 replicas × 5Gi is feasible. Signed-off-by: Vico Chu <vico24826@gmail.com>
vicoooo26
force-pushed
the
fix/kubernetes-version-pin
branch
from
July 24, 2026 16:51
889b982 to
82b2b71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pin
kubernetes<36in the Python Test CI workflow to fix 13 unit test failures caused by kubernetes 36.0.0 switching to Pydantic models, which breaks theFakeResponsedeserialization hack in the SDK unit tests.Changes
kubernetes<36in.github/workflowsinstall stepRoot Cause
kubernetes 36.0.0 (released 2026-05-20) regenerated the synchronous client with the modern OpenAPI Generator
pythonbackend, switching all models to Pydantic. This breaksFakeResponseinsdk/python/kubeflow/training/utils/utils.py, which relies on the oldApiClient.deserialize()reading a.dataJSON string attribute.The
kubernetes>=27.2.0constraint insetup.pyhas no upper bound, so CI installs the latest version. The fork's CI run on 2026-07-14 installed kubernetes 36.0.3, causing 13 tests to fail withRuntimeError. Upstream's run on 2026-04-30 installed kubernetes 35.0.0 and passed.Only the CI workflow file is modified — no SDK files are touched to avoid impacting release.
Test Plan