-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Python: chore(python): improve dependency range automation #4343
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eavanvalkenburg
merged 19 commits into
microsoft:main
from
eavanvalkenburg:dependency_strategy
Mar 13, 2026
Merged
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
7118ef8
chore(python): improve dependency range automation
eavanvalkenburg e436655
updated text and pyarrow
eavanvalkenburg 2f75cac
new lock
eavanvalkenburg 075838a
fixed workflow
eavanvalkenburg 91e8556
updated deps
eavanvalkenburg f2891c5
fix tiktoken
eavanvalkenburg 7690b20
chore(python): refine dependency validation workflows
eavanvalkenburg 192ff6d
docs(python): add high-level dependency validation comments
eavanvalkenburg 53a6aea
WIP
eavanvalkenburg 6cb99e4
added additional comments and excludes
eavanvalkenburg 7532710
added dev dependency handling and workflow and updates to package ranges
eavanvalkenburg f228884
added readme and simplified commands
eavanvalkenburg f447561
fix markers
eavanvalkenburg 7e96af9
chore(python): address dependency review feedback
eavanvalkenburg ede03fe
Tighten dependency bounds, remove stale overrides, restore Python 3.1…
eavanvalkenburg 876cc09
small tweaks
eavanvalkenburg f2182ab
add note in workflow
eavanvalkenburg ac3db35
fix workflows and several versions
eavanvalkenburg bb83d74
fix duplicate
eavanvalkenburg File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
216 changes: 216 additions & 0 deletions
216
.github/workflows/python-dependency-range-validation.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,216 @@ | ||
| # Probe the highest allowed dependency versions, then open issues/PRs from the passing updates. | ||
| name: Python - Dependency Range Validation | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
|
eavanvalkenburg marked this conversation as resolved.
|
||
|
|
||
| permissions: | ||
| contents: write | ||
| issues: write | ||
| pull-requests: write | ||
|
|
||
| env: | ||
| UV_CACHE_DIR: /tmp/.uv-cache | ||
|
|
||
| jobs: | ||
| dependency-range-validation: | ||
| name: Dependency Range Validation | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| # For now only run 3.13, if we do encounter situations where there are mismatches between packages and python versions (other then 3.10 and 3.14 which are known to not be able to install everything) | ||
| # then we will have to reevaluate. | ||
| UV_PYTHON: "3.13" | ||
|
eavanvalkenburg marked this conversation as resolved.
|
||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Set up python and install the project | ||
| uses: ./.github/actions/python-setup | ||
| with: | ||
| python-version: ${{ env.UV_PYTHON }} | ||
| os: ${{ runner.os }} | ||
| env: | ||
| UV_CACHE_DIR: /tmp/.uv-cache | ||
|
|
||
| - name: Run dependency range validation | ||
| id: validate_ranges | ||
| # Keep workflow running so we can still publish diagnostics from this run. | ||
| continue-on-error: true | ||
| run: uv run poe validate-dependency-bounds-project --mode upper --project "*" | ||
| working-directory: ./python | ||
|
|
||
| - name: Upload dependency range report | ||
| # Always publish the report so failures are inspectable even when validation fails. | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: dependency-range-results | ||
| path: python/scripts/dependencies/dependency-range-results.json | ||
| if-no-files-found: warn | ||
|
|
||
| - name: Create issues for failed dependency candidates | ||
| # Always process the report so failed candidates create actionable tracking issues. | ||
| if: always() | ||
| uses: actions/github-script@v8 | ||
| with: | ||
| script: | | ||
| const fs = require("fs") | ||
| const reportPath = "python/scripts/dependencies/dependency-range-results.json" | ||
|
|
||
| if (!fs.existsSync(reportPath)) { | ||
| core.warning(`No dependency range report found at ${reportPath}`) | ||
| return | ||
| } | ||
|
|
||
| const report = JSON.parse(fs.readFileSync(reportPath, "utf8")) | ||
| const dependencyFailures = [] | ||
|
|
||
| for (const packageResult of report.packages ?? []) { | ||
| for (const dependency of packageResult.dependencies ?? []) { | ||
| const candidateVersions = new Set(dependency.candidate_versions ?? []) | ||
| const failedAttempts = (dependency.attempts ?? []).filter( | ||
| (attempt) => attempt.status === "failed" && candidateVersions.has(attempt.trial_upper) | ||
| ) | ||
| if (!failedAttempts.length) { | ||
| continue | ||
| } | ||
|
|
||
| const failuresByVersion = new Map() | ||
| for (const attempt of failedAttempts) { | ||
| const version = attempt.trial_upper || "unknown" | ||
| if (!failuresByVersion.has(version)) { | ||
| failuresByVersion.set(version, attempt.error || "No error output captured.") | ||
| } | ||
| } | ||
|
|
||
| dependencyFailures.push({ | ||
| packageName: packageResult.package_name, | ||
| projectPath: packageResult.project_path, | ||
| dependencyName: dependency.name, | ||
| originalRequirements: dependency.original_requirements ?? [], | ||
| finalRequirements: dependency.final_requirements ?? [], | ||
| failedVersions: [...failuresByVersion.entries()].map(([version, error]) => ({ version, error })), | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| if (!dependencyFailures.length) { | ||
| core.info("No failing dependency candidates found.") | ||
| return | ||
| } | ||
|
|
||
| const owner = context.repo.owner | ||
| const repo = context.repo.repo | ||
| const openIssues = await github.paginate(github.rest.issues.listForRepo, { | ||
| owner, | ||
| repo, | ||
| state: "open", | ||
| per_page: 100, | ||
| }) | ||
| const openIssueTitles = new Set( | ||
| openIssues.filter((issue) => !issue.pull_request).map((issue) => issue.title) | ||
| ) | ||
|
|
||
| const formatError = (message) => String(message || "No error output captured.").replace(/```/g, "'''") | ||
|
|
||
| for (const failure of dependencyFailures) { | ||
| const title = `Dependency validation failed: ${failure.dependencyName} (${failure.packageName})` | ||
| if (openIssueTitles.has(title)) { | ||
| core.info(`Issue already exists: ${title}`) | ||
| continue | ||
| } | ||
|
|
||
| const visibleFailures = failure.failedVersions.slice(0, 5) | ||
| const omittedCount = failure.failedVersions.length - visibleFailures.length | ||
| const failureDetails = visibleFailures | ||
| .map( | ||
| (entry) => | ||
| `- \`${entry.version}\`\n\n\`\`\`\n${formatError(entry.error).slice(0, 3500)}\n\`\`\`` | ||
| ) | ||
| .join("\n\n") | ||
|
|
||
| const body = [ | ||
| "Automated dependency range validation found candidate versions that failed checks.", | ||
| "", | ||
| `- Package: \`${failure.packageName}\``, | ||
| `- Project path: \`${failure.projectPath}\``, | ||
| `- Dependency: \`${failure.dependencyName}\``, | ||
| `- Original requirements: ${ | ||
| failure.originalRequirements.length | ||
| ? failure.originalRequirements.map((value) => `\`${value}\``).join(", ") | ||
| : "_none_" | ||
| }`, | ||
| `- Final requirements after run: ${ | ||
| failure.finalRequirements.length | ||
| ? failure.finalRequirements.map((value) => `\`${value}\``).join(", ") | ||
| : "_none_" | ||
| }`, | ||
| "", | ||
| "### Failed versions and errors", | ||
| failureDetails, | ||
| omittedCount > 0 ? `\n_Additional failed versions omitted: ${omittedCount}_` : "", | ||
| "", | ||
| `Workflow run: ${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`, | ||
| ].join("\n") | ||
|
|
||
| await github.rest.issues.create({ | ||
| owner, | ||
| repo, | ||
| title, | ||
| body, | ||
| }) | ||
| openIssueTitles.add(title) | ||
| core.info(`Created issue: ${title}`) | ||
| } | ||
|
|
||
| - name: Refresh lockfile | ||
| # Only refresh lockfile after a clean validation to avoid committing known-bad ranges. | ||
| if: steps.validate_ranges.outcome == 'success' | ||
| run: uv lock --upgrade | ||
| working-directory: ./python | ||
|
|
||
| - name: Commit and push dependency updates | ||
| id: commit_updates | ||
| if: steps.validate_ranges.outcome == 'success' | ||
| run: | | ||
| BRANCH="automation/python-dependency-range-updates" | ||
|
|
||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git checkout -B "${BRANCH}" | ||
|
|
||
| git add python/packages/*/pyproject.toml python/uv.lock | ||
| if git diff --cached --quiet; then | ||
|
eavanvalkenburg marked this conversation as resolved.
|
||
| echo "has_changes=false" >> "$GITHUB_OUTPUT" | ||
| echo "No dependency updates to commit." | ||
| exit 0 | ||
| fi | ||
|
|
||
| git commit -m "chore: update dependency ranges" | ||
| git push --force-with-lease --set-upstream origin "${BRANCH}" | ||
| echo "has_changes=true" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Create or update pull request with GitHub CLI | ||
| # Only open/update PRs for validated updates to keep automation branches trustworthy. | ||
| if: steps.validate_ranges.outcome == 'success' && steps.commit_updates.outputs.has_changes == 'true' | ||
| run: | | ||
| BRANCH="automation/python-dependency-range-updates" | ||
| PR_TITLE="Python: chore: update dependency ranges" | ||
| PR_BODY_FILE="$(mktemp)" | ||
|
|
||
| cat > "${PR_BODY_FILE}" <<'EOF' | ||
| This PR was generated by the dependency range validation workflow. | ||
|
|
||
| - Ran `uv run poe validate-dependency-bounds-project --mode upper --project "*"` | ||
| - Updated package dependency bounds | ||
| - Refreshed `python/uv.lock` with `uv lock --upgrade` | ||
| EOF | ||
|
|
||
| PR_NUMBER="$(gh pr list --head "${BRANCH}" --base main --state open --json number --jq '.[0].number')" | ||
| if [ -n "${PR_NUMBER}" ]; then | ||
| gh pr edit "${PR_NUMBER}" --title "${PR_TITLE}" --body-file "${PR_BODY_FILE}" | ||
| else | ||
| gh pr create --base main --head "${BRANCH}" --title "${PR_TITLE}" --body-file "${PR_BODY_FILE}" | ||
| fi | ||
|
eavanvalkenburg marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| name: Python - Dev Dependency Upgrade | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: write | ||
| pull-requests: write | ||
|
|
||
| env: | ||
| UV_CACHE_DIR: /tmp/.uv-cache | ||
|
|
||
| jobs: | ||
| upgrade-dev-dependencies: | ||
| name: Upgrade Dev Dependencies | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| UV_PYTHON: "3.13" | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Set up python and install the project | ||
| uses: ./.github/actions/python-setup | ||
| with: | ||
| python-version: ${{ env.UV_PYTHON }} | ||
| os: ${{ runner.os }} | ||
| env: | ||
| UV_CACHE_DIR: /tmp/.uv-cache | ||
|
|
||
| - name: Upgrade dev dependencies and validate workspace | ||
| run: uv run poe upgrade-dev-dependencies | ||
| working-directory: ./python | ||
|
|
||
| - name: Commit and push dev dependency updates | ||
| id: commit_updates | ||
| run: | | ||
| BRANCH="automation/python-dev-dependency-updates" | ||
|
|
||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git checkout -B "${BRANCH}" | ||
|
|
||
| git add python/pyproject.toml python/packages/*/pyproject.toml python/uv.lock | ||
| if git diff --cached --quiet; then | ||
| echo "has_changes=false" >> "$GITHUB_OUTPUT" | ||
| echo "No dev dependency updates to commit." | ||
| exit 0 | ||
| fi | ||
|
|
||
| git commit -F- <<'EOF' | ||
| Python: chore: upgrade dev dependencies | ||
| EOF | ||
| git push --force-with-lease --set-upstream origin "${BRANCH}" | ||
| echo "has_changes=true" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Create or update pull request with GitHub CLI | ||
| if: steps.commit_updates.outputs.has_changes == 'true' | ||
| run: | | ||
| BRANCH="automation/python-dev-dependency-updates" | ||
| PR_TITLE="Python: chore: upgrade dev dependencies" | ||
| PR_BODY_FILE="$(mktemp)" | ||
|
|
||
| cat > "${PR_BODY_FILE}" <<'EOF' | ||
| ### Motivation and Context | ||
|
|
||
| This automated update refreshes Python dev dependency pins across the workspace and reruns the repo validation gates before opening a pull request. | ||
|
|
||
| ### Description | ||
|
|
||
| - Ran `uv run poe upgrade-dev-dependencies` | ||
| - Refreshed dev dependency pins in workspace `pyproject.toml` files | ||
| - Refreshed `python/uv.lock` with `uv lock --upgrade` | ||
| - Reinstalled from the frozen lockfile and reran `check`, `typing`, and `test` | ||
|
|
||
| ### Contribution Checklist | ||
|
|
||
| - [x] The code builds clean without any errors or warnings | ||
| - [x] The PR follows the [Contribution Guidelines](https://github.com/microsoft/agent-framework/blob/main/CONTRIBUTING.md) | ||
| - [x] All unit tests pass, and I have added new tests where possible | ||
| - [ ] **Is this a breaking change?** If yes, add "[BREAKING]" prefix to the title of the PR. | ||
| EOF | ||
|
|
||
| PR_NUMBER="$(gh pr list --head "${BRANCH}" --base main --state open --json number --jq '.[0].number')" | ||
| if [ -n "${PR_NUMBER}" ]; then | ||
| gh pr edit "${PR_NUMBER}" --title "${PR_TITLE}" --body-file "${PR_BODY_FILE}" | ||
| else | ||
| gh pr create --base main --head "${BRANCH}" --title "${PR_TITLE}" --body-file "${PR_BODY_FILE}" | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.