CI: run vendored template suites against the SDK checkout - #513
CI: run vendored template suites against the SDK checkout#513jdchawla29 wants to merge 1 commit into
Conversation
ea60e77 to
76ffbed
Compare
Each environments/ template installs its pinned deps, overrides hud with the checkout via an editable install, then runs its own ruff and pytest. Catches SDK changes that break the canonical templates at the PR boundary.
8c98a9c to
5b253c2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b253c2437
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - name: Install template with the SDK from this checkout | ||
| working-directory: environments/${{ matrix.template }} | ||
| run: | | ||
| uv sync --all-extras |
There was a problem hiding this comment.
Enforce the checked-in template locks
For both matrix entries, if a PR changes environments/<template>/pyproject.toml without updating its uv.lock, this command silently regenerates the lock in the runner and tests that uncommitted state, allowing CI to pass while the checked-in reproducible environment remains stale. uv sync --help documents --locked as “Assert that the uv.lock will remain unchanged”; use that option here so dependency and lock changes must be committed together.
Useful? React with 👍 / 👎.
| working-directory: environments/${{ matrix.template }} | ||
| run: | | ||
| uv sync --all-extras | ||
| uv pip install -e ../.. |
There was a problem hiding this comment.
Resolve the checkout in a clean template dependency graph
When the checkout removes a dependency previously supplied by the released hud package, the preceding sync installs that old dependency from the template lock and this additive editable install leaves it behind. Template tests can therefore keep importing an undeclared dependency and pass even though a clean installation against the checkout would fail; uv pip install --help confirms that removing extraneous packages requires --exact. Resolve hud from ../.. as part of the template environment rather than layering it over the released package's dependency set.
Useful? React with 👍 / 👎.
Stacked on #512.
Adds a
templatesCI job: for each ofenvironments/{coding,cua},uv syncthe template's pinned dependencies, install the SDK from the checkout over thehudpin (editable), then run the template's ownruff format --check,ruff check, andpytest.This makes template breakage an SDK PR failure rather than something discovered after release. The coding template's suite includes a full local rollout (serve → connect → run → grade, golden=1.0/baseline=0.0), so the v6 serving contract is exercised end to end on every PR. Docker-dependent tests self-skip; those move to the nightly/release-gate job in a follow-up.
Note
Low Risk
CI-only change with no runtime or SDK behavior modifications; risk is limited to longer CI runs or flaky template tests blocking merges.
Overview
Adds a
templatesCI job that runs on every PR alongside the existing SDK test matrix.For each vendored template (
environments/codingandenvironments/cua), the job syncs the template’s pinned deps, replaces the publishedhudpin with an editable install of the repo root (uv pip install -e ../..), then runs that template’s ruff format (check), ruff check, and pytest.This turns template breakage into an SDK PR failure instead of a post-release discovery. The coding template’s pytest suite includes hermetic local rollout coverage (serve → connect → run → grade), so the v6 serving contract is exercised end-to-end on every PR; Docker-only integration tests are expected to skip in this job.
Reviewed by Cursor Bugbot for commit 5b253c2. Bugbot is set up for automated code reviews on this repo. Configure here.
HUD-2276