[TEST — do not merge] #417 empirical 401 check - #420
Closed
d-morrison wants to merge 1 commit into
Closed
Conversation
…on-skip THROWAWAY empirical test for #417 finding 1. This PR edits claude-review.yml (the caller stub) AND routes its review through #417's claude-code-review.yml (IS_DISPATCH bypass). A workflow_dispatch with pr_number set will bypass the self-review skip; we observe whether the action 401s (workflow validation) or posts a real verdict. To be closed after.
Collaborator
Author
github-actions Bot
added a commit
that referenced
this pull request
Aug 5, 2026
Empirical test (#420) confirmed the IS_DISPATCH bypass trades gha#286's silent skip for a red required-check failure instead of fixing it: a dispatched/mentioned run on a PR editing the caller workflow still executes that PR's modified copy, so claude-code-action's App-token exchange 401s regardless of trigger type. Reverting both the workflow change and its CLAUDE.md documentation back to main; gha#286's silent-skip limitation remains open and documented as-is. A real fix would need to pass claude-code-action a non-OIDC github_token override, which is a separate security/secret decision left for a future PR.
This was referenced Aug 5, 2026
d-morrison
added a commit
that referenced
this pull request
Aug 6, 2026
- The validation skip is reachable today, not only via a deliberate bypass: the self_mod guard reads files via 'gh api ... || true', so a transient gh api failure fails open (self_mod=false) on a caller-editing PR and the review proceeds into the content-validation skip. Note both paths (deliberate #417-style bypass; live fail-open). - Drop the self-contradictory '(e.g. by the github_token fix below)' from the diagnostic-tells intro (the override prevents the skip, and it was a forward reference). - Downgrade 'the real fix (proven via #420)' to 'likely (untested) fix': #420 only proved the bypass counterproductive; the override was never exercised and has no input wired up. Same softening in the changelog.
d-morrison
added a commit
that referenced
this pull request
Aug 6, 2026
#422) * docs: record the claude-code-action workflow-validation skip signature A PR that edits the review workflow and bypasses the self-review skip guard (e.g. an @claude review dispatch, as #417 attempted) still gets no verdict: anthropics/claude-code-action validates its own content against the default branch and gracefully skips on a mismatch, producing outcome=success, a sub-15s run, and no execution output -- so check-review-execution.sh reddens claude-review/require-review with no verdict. Record that this does NOT surface as a literal 401 in the log (the signature is 'Workflow validation failed' / 'Exiting due to workflow validation skip'), the diagnostic tells, that bypassing the self-review skip does not help since validation keys on workflow content vs the default branch, and that a github_token override on the action is the real fix. Proven empirically via throwaway test PR #420 on 2026-08-05. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: correct workflow-validation-skip note now that #417 is abandoned Address #422 review: - Fix the factual framing: dispatch does NOT bypass self_mod today (the guard's PR_NUMBER resolves the same regardless of trigger, per the gha#286 example); bypassing it was gha#417's proposal, now abandoned. Reframe as 'why the guard exists / why bypassing it fails', past-tense. - Timing tell is step-level (~4-11s), not a job-level ~15s; note total job time is unreliable (checkout/installs run first). - Name the caller workflow claude-review.yml; use self_mod terminology; keep '@claude review' code spans intact on one line. - Same corrections in the changelog fragment. * docs: correct fail-open path and untested-fix overclaim (#422 review) - The validation skip is reachable today, not only via a deliberate bypass: the self_mod guard reads files via 'gh api ... || true', so a transient gh api failure fails open (self_mod=false) on a caller-editing PR and the review proceeds into the content-validation skip. Note both paths (deliberate #417-style bypass; live fail-open). - Drop the self-contradictory '(e.g. by the github_token fix below)' from the diagnostic-tells intro (the override prevents the skip, and it was a forward reference). - Downgrade 'the real fix (proven via #420)' to 'likely (untested) fix': #420 only proved the bypass counterproductive; the override was never exercised and has no input wired up. Same softening in the changelog. * docs: semantic line breaks in the fail-open paragraph (one sentence per line) * docs: 'skips every step of the review job' (job runs green, not skipped) Address #422 review: the self_mod guard is step-level; the job still runs and reports success (green), so 'skips the whole review job' was imprecise and contradicted the paragraph's own 'green job, every post-guard step skipped' clause. * docs: cross-link the two OIDC-validation sections; reconcile github_token confidence; add fail-open path to changelog Address #422 review round 4: - Cross-link the workflow-validation-skip note and the 'Test changes against a template repo' section (same OIDC mechanism, two angles) so neither re-derives the other uncited (item 10). - Add the 'untested' caveat to that section's github_token mention so both passages state the fix with the same confidence. - Fold the fail-open gh-api path into the changelog fragment, which had only the deliberate-bypass path (a changelog reader lacks the CLAUDE.md context). * docs: drop forward-ref to 'Never just theorize' section; add (#422) to changelog bullet * docs: drop 'for the reason below' forward signpost (reason emerges inline) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway test for #417 finding 1. Edits
claude-review.yml(so the self-review guard would skip on automatic runs) and repoints its callee at thefix-self-review-mention-skipbranch (so the review runs #417'sIS_DISPATCHbypass). Aworkflow_dispatchwithpr_numberset triggersIS_DISPATCH=true→ bypasses the skip. Observing: does the action 401 (workflow validation, reviewer's finding 1) or post a real verdict (fix works)? Will be closed after.