ENG-3651/ENG-3652 Add Prime release e2e checks#644
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d935eafc3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6d935ea. Configure here.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8c98d7202
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| run( | ||
| [ | ||
| uv, | ||
| "pip", | ||
| "install", |
There was a problem hiding this comment.
In the sandbox this creates /workspace/venv and then calls uv pip install with only PATH pointing at that venv; uv's environment discovery for uv pip install uses an activated VIRTUAL_ENV/CONDA_PREFIX or a .venv directory in the cwd/parents (uv docs), so a /workspace/venv directory is not selected. As a result the release e2e fails during candidate CLI installation before any checks run; set VIRTUAL_ENV=str(VENV), create .venv, or pass --python for the venv Python.
Useful? React with 👍 / 👎.
2 participants
Adds a sandbox-backed release e2e workflow for prime env push/install, local eval uploads, and hosted eval submissions before release merges.
Note
Medium Risk
Adds a new GitHub Action and companion script that uses real API credentials to create sandboxes, push environments, and run/submit hosted evals, so failures could consume resources or affect remote state. Logic is isolated to CI tooling, but it exercises multiple external services and cleanup paths.
Overview
Introduces a new Release E2E Tests GitHub workflow (
.github/workflows/release-e2e.yml) that runs onrelease/*PRs (or manually) and executes release smoke tests inside a Prime sandbox using repo-provided SDKs and configured secrets/vars.Adds
packages/prime/scripts/release_e2e.py, which archives the current checkout (skipping common build artifacts, symlinks, and likely-secret files), provisions a sandbox, installs the candidate CLI from the uploaded source, then validates core flows:prime env push/info/install, local eval runs +eval push, and optional hosted eval submission with configurable wait/cancel behavior and best-effort cleanup.Includes tests (
packages/prime/tests/test_release_e2e_script.py) to ensure archive filtering, remote-script compilation/cleanup behavior, and workflow input handling are correct (and that any old workflow name/path isn’t present).Reviewed by Cursor Bugbot for commit e8c98d7. Bugbot is set up for automated code reviews on this repo. Configure here.