Skip to content

[codex] add test impact plan workflow#794

Merged
bill-ph merged 3 commits into
mainfrom
codex/test-impact-plan
Jun 18, 2026
Merged

[codex] add test impact plan workflow#794
bill-ph merged 3 commits into
mainfrom
codex/test-impact-plan

Conversation

@bill-ph

@bill-ph bill-ph commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a deterministic test impact planner that summarizes how a PR changes test coverage signals, similar to a lightweight tfplan for tests.

  • Adds scripts/test_impact_plan.py to classify test file deltas, test case/assertion deltas, added skips/known failures, workflow runner changes, path filters, continue-on-error, e2e retry changes, GitHub API files that lack patch data, renamed files moving into/out of test scope, and incomplete PR file-list data.
  • Adds just test-impact-plan so developers can preview the plan for a branch.
  • Adds a two-workflow comment flow that avoids pull_request_target:
    • test-impact-plan runs on pull_request with read-only permissions and records the trigger.
    • test-impact-plan-comment runs on workflow_run, checks out trusted default-branch code, verifies the PR context is current/open/targeting this repo, re-fetches PR metadata/files via GitHub API, runs the trusted planner, writes the job summary, and posts/updates one sticky PR comment.

Notes

The workflow is advisory and does not block merges by itself.

The privileged comment workflow does not trust artifacts or PR-produced output. It uses workflow_run only as a trigger, then re-fetches PR data from GitHub and runs the planner from trusted default-branch code.

Because the workflow_run workflow must exist on the default branch, sticky comments will apply to future PRs after this lands.

Validation

  • Red/green acceptance check for --github-pr-files-json
  • Red/green acceptance check for missing GitHub API patch data
  • Red/green acceptance check for renamed test files moving out of test scope
  • Red/green acceptance check for incomplete PR file-list warnings
  • python3 scripts/test_impact_plan.py --github-pr-files-json /tmp/duckgres-pr794-files.json --expected-file-count $(jq -r '.changed_files' /tmp/duckgres-pr794.json)
  • python3 scripts/test_impact_plan.py --base origin/main --head HEAD
  • PYTHONPYCACHEPREFIX=/tmp/duckgres-pycache python3 -m py_compile scripts/test_impact_plan.py
  • ruby -e 'require "psych"; Dir[".github/workflows/test-impact-plan*.yml"].each { |p| Psych.load_file(p); puts "ok #{p}" }'
  • git diff --cached --check
  • just lint in a clean temporary copy with only this branch's changes applied

Local just lint in the active worktree is currently blocked by an unrelated unstaged edit in server/conn_cleanup_test.go (testing import unused after a local test deletion).

@bill-ph bill-ph marked this pull request as ready for review June 18, 2026 19:39
Comment thread .github/workflows/test-impact-plan.yml Fixed
@bill-ph bill-ph force-pushed the codex/test-impact-plan branch from 857e17d to bf1e4a3 Compare June 18, 2026 20:14
@bill-ph bill-ph force-pushed the codex/test-impact-plan branch from bf1e4a3 to 7834629 Compare June 18, 2026 20:33
@bill-ph bill-ph merged commit 58fbedb into main Jun 18, 2026
23 checks passed
@bill-ph bill-ph deleted the codex/test-impact-plan branch June 18, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants