-
-
Notifications
You must be signed in to change notification settings - Fork 104
Add get-reviews agent skill
#765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
2chanhaeng
wants to merge
15
commits into
fedify-dev:main
Choose a base branch
from
2chanhaeng:get-switch
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+430
−0
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
305bca6
Add get-reviews skill for handling PR reviews
2chanhaeng 91acdd0
Fetch PR reviews and link review threads to their reviews and comments
2chanhaeng b36f9ff
Add harnesses for agents often mistaking parts
2chanhaeng 58fdc33
Tidy `get-reviews` skill helper files
2chanhaeng cf9a553
Declare GraphQL variables in `fetch_reviews.sh`
2chanhaeng 1758a7b
Wire cursor pagination and fix `get-reviews` SKILL.md typos
2chanhaeng 81575ed
Rewrite review.md frontmatter and fix two body typos
2chanhaeng 66d4912
Add harnesses for agents often mistaking parts
2chanhaeng cbb7868
Add harnesses for about non-review thread
2chanhaeng 01f2b85
Add harnesses for agents often mistaking parts
2chanhaeng 1d832bd
Fail fast on missing env vars in `fetch_reviews.sh`
2chanhaeng 62156ab
List required env vars for `fetch_reviews.sh` in SKILL.md
2chanhaeng fb98820
Replace unused comment with file path logging
2chanhaeng 0628a9f
Improve get-reviews skill instructions and review template
2chanhaeng d5acf08
Reframe get-reviews to assist, not decide
2chanhaeng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| --- | ||
| name: get-reviews | ||
| description: >- | ||
| This skill is utilized when fetching reviews from GitHub pull requests. | ||
| Get the reviews, organize, and help to resolve them. | ||
| argument-hint: "Provide the number of the pull request to get the reviews." | ||
| --- | ||
|
|
||
| <!-- deno-fmt-ignore-file --> | ||
|
|
||
| Get reviews from GitHub pull requests | ||
| ===================================== | ||
|
|
||
| This skill is utilized when requesting reviews from GitHub pull requests. | ||
|
|
||
| 1. Get the reviews. | ||
| 2. Organize the reviews. | ||
| 3. Help to resolve the reviews. | ||
|
|
||
|
|
||
| Get the reviews | ||
| --------------- | ||
|
|
||
| To get the reviews from a GitHub pull request, you can use the GitHub API. | ||
| Check the [`gh` CLI tool][gh] is installed and authenticated. | ||
| `gh auth status` can be used to check the authentication status. | ||
| If `gh` isn't installed, try installing it by `apt install gh`. | ||
| If authentication is not set up, tell the contributor to run `gh auth login` | ||
| to authenticate with GitHub. | ||
|
|
||
| Use the GraphQL API to fetch the reviews for a specific pull request. | ||
| Check [fetch\_reviews.sh](./fetch_reviews.sh) to fetch the reviews | ||
| and save them in a JSON file: | ||
|
|
||
| - Fill the variables in the command and run the command to fetch the reviews: | ||
|
|
||
| ~~~~ bash | ||
| # Omit the new lines | ||
| PR_NUMBER= NUMBER_OF_PR_COMMENTS= NUMBER_OF_REVIEWS= \ | ||
| NUMBER_OF_THREADS= NUMBER_OF_COMMENTS_PER_THREAD= LAST_CURSOR= \ | ||
| bash .agents/skills/get-reviews/fetch_reviews.sh | ||
| ~~~~ | ||
|
|
||
|
|
||
| - `PR_NUMBER`: The number of the pull request to fetch reviews from. | ||
| - `NUMBER_OF_PR_COMMENTS`: The number of PR comments to fetch. | ||
| - `NUMBER_OF_REVIEWS`: The number of reviews to fetch. | ||
| - `NUMBER_OF_THREADS`: The number of review threads to fetch. | ||
| - `NUMBER_OF_COMMENTS_PER_THREAD`: | ||
| The number of comments per review thread to fetch. | ||
| - `LAST_CURSOR`: The cursor for incremental fetches. Optional. | ||
|
|
||
| - For incremental fetches, save `pageInfo.endCursor` from the previous | ||
| fetch and pass it as `after: $LAST_CURSOR` (a base64 cursor, not a | ||
| review thread node ID) so the query returns only new review threads. | ||
|
|
||
| - Use `jq` to filter the reviews and information if necessary. | ||
|
|
||
| The fetched JSON files in *plans/{PR\_NUMBER}/fetched/* contain the raw data | ||
| of PR comments, reviews, and review threads (with `pullRequestReview` | ||
| back-references on thread comments). When more context is needed later | ||
| (e.g., to resolve which review a thread belongs to, or to check the original | ||
| body of a comment), refer back to these files instead of re-fetching. | ||
|
|
||
| [gh]: https://cli.github.com/ | ||
|
|
||
|
|
||
| Organize the reviews | ||
| -------------------- | ||
|
|
||
| After fetching the PR and its reviews, organize the reviews. | ||
| [*plans* directory in the root](../../../plans/) is a good place to store them. | ||
|
2chanhaeng marked this conversation as resolved.
|
||
| *plans/* is already ignored in `.gitignore`, so don't check that it is ignored. | ||
|
|
||
| - *plans/{PR\_NUMBER}/index.md*: The main file for the PR. | ||
| - The body of the PR | ||
| - *plans/{PR\_NUMBER}/reviews/{REVIEW\_ID}.md*: The file for each review | ||
| which is not resolved. | ||
| - After applying or dismissing the review, move the file to | ||
| *plans/{PR\_NUMBER}/reviews/resolved/{REVIEW\_ID}.md*. | ||
| - If the review file is too long, move the content to | ||
| *plans/{PR\_NUMBER}/reviews/{REVIEW\_ID}/index.md*, and separate the | ||
| content into multiple files in the same directory. In this case, after | ||
| resolving the review, move the whole directory to | ||
| *plans/{PR\_NUMBER}/reviews/resolved/{REVIEW\_ID}/*. | ||
|
2chanhaeng marked this conversation as resolved.
|
||
| - *plans/{PR\_NUMBER}/reviews/resolved/{REVIEW\_ID}.md*: The file for each | ||
| review which is resolved. | ||
|
|
||
| **Don't use the first comment of the review thread as the review ID.** | ||
| The ID of the review thread starts with “PRRT\_”. | ||
| Use the first comment ID of the review thread only on the link. | ||
|
|
||
| Review threads aren't the only place that asks for changes. A PR-level | ||
| comment (in `comments` of the fetched JSON, whose ID starts with “IC\_”) | ||
| or the body of a review (in `reviews` of the fetched JSON, whose ID starts | ||
| with “PRR\_”) can also point out things to fix. When such a comment or | ||
| review body requests modifications, organize it as its own review file | ||
| alongside the review-thread files, using the same directory layout and | ||
| naming rules: | ||
|
|
||
| - Use the node ID of the PR comment (“IC\_…”) or the review (“PRR\_…”) | ||
| as the `{REVIEW\_ID}` in the file path. | ||
| - If a PR comment or review body only contains approval, general | ||
| impressions, questions without a modification request, or other content | ||
| with nothing to fix, skip it and do not create a file for it. | ||
| - If only a part of a longer PR comment or review body requests changes, | ||
| create a file only for that request and quote the relevant excerpt in | ||
| the file so the context is preserved. | ||
|
|
||
| The format of review files should be as [review.md](./review.md). | ||
| Read the review and draft the format based on the relevant information. | ||
| The files should be written in the contributor's language. But the title of | ||
| the item in the file (e.g., “Summary”, “Judgement”, “Plans”) should be in | ||
| English for consistency. | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| Empty the space between the “Title” and the “Summary” sections. | ||
|
|
||
| All related information with the review should be stored in | ||
| *plans/{PR\_NUMBER}/reviews/{REVIEW\_ID}.md* or the files in | ||
| *plans/{PR\_NUMBER}/reviews/{REVIEW\_ID}/*. | ||
|
|
||
| After organizing the reviews, show the links to the files to the contributor. | ||
|
|
||
|
|
||
| Help to resolve the reviews | ||
| --------------------------- | ||
|
|
||
| The agent's role in this step is to *help* the contributor resolve the reviews, | ||
| not to resolve them on the agent's own authority. The final judgement always | ||
| belongs to the contributor; the agent's job is to surface the information needed | ||
| to decide, and then to carry out whatever the contributor decides. | ||
|
|
||
| For each unresolved review, give the contributor what they need to judge it, | ||
| rather than judging on their behalf: | ||
|
|
||
| - Add relative links to the code or documentation the review points at. | ||
| - Summarize the relevant facts, the trade-offs, and any available options. | ||
| - Where the facts are clear, the agent may include a *suggested* judgement, | ||
| but it must be clearly marked as a suggestion and never substitute for the | ||
| contributor's decision. | ||
|
|
||
| Let the contributor read the review files, decide the judgement and the plans | ||
| for each review, and update the review files if necessary. Do not apply or | ||
| dismiss anything until the contributor has decided. After the contributor | ||
| decides the judgement and the plans, apply or dismiss the reviews based on the | ||
| files. | ||
|
|
||
| Categorize the reviews and the plans, and apply them at once by category. | ||
| After applying the review, use [`/commit` skill](../commit/SKILL.md) to commit | ||
|
2chanhaeng marked this conversation as resolved.
|
||
| the changes. The commit message should include the related review links: | ||
|
|
||
| `https://github.com/fedify-dev/fedify/pull/{PR_NUMBER}#discussion_r{REVIEW_THREAD.COMMENTS[0].DATABASE_ID}` | ||
|
|
||
| Because these changes are produced with agent assistance, the commit message | ||
| must also carry the `Assisted-by: AGENT_NAME:MODEL_VERSION` trailer required by | ||
| [*AI_POLICY.md*](../../../AI_POLICY.md), in addition to the review links. | ||
|
|
||
| After committing the changes, update the review file to include the commit hash | ||
| and the comment section. If the review is dismissed, update the review file to | ||
| include the reason for dismissing and the comment section. | ||
|
|
||
| If the `Comments` are written only in the contributor's language, provide an | ||
| English translation and have the contributor review it. If they are written in | ||
| both languages, check for any discrepancies between the two. If differences | ||
| exist between the two versions, review them based on the facts and revise | ||
| the English version to match the content in the contributor's language. | ||
|
|
||
| Post all review comments in English, even if the file written in the | ||
| contributor's native language. The comments should be polite and constructive. | ||
|
|
||
| Before pushing commits, posting comments, or marking any review as resolved, | ||
| the contributor must verify the actual applied changes and the relevant | ||
| test/check results (e.g., `mise run check` and the related tests). Per | ||
| [*AI_POLICY.md*](../../../AI_POLICY.md), AI-created work must be fully verified | ||
| with human use; do not post comments or mark reviews resolved for changes whose | ||
| correctness is only hypothetical. The agent prepares everything for this | ||
| verification but leaves the verification itself to the contributor. | ||
|
|
||
| After resolving the reviews, pushing commits, posting comments, and updating | ||
|
2chanhaeng marked this conversation as resolved.
|
||
| the reviews as resolved, move the review files to | ||
| *plans/{PR\_NUMBER}/reviews/resolved*. Before moving the files, check status and | ||
| comments of the PR from GitHub. Use [fetch\_reviews.sh](./fetch_reviews.sh), | ||
| but instead of fetching all reviews and attributes, fetch only the necessary | ||
| attributes to check the review status and comments. | ||
|
2chanhaeng marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| : "${PR_NUMBER:?PR_NUMBER is required}" | ||
| : "${NUMBER_OF_PR_COMMENTS:?NUMBER_OF_PR_COMMENTS is required}" | ||
| : "${NUMBER_OF_REVIEWS:?NUMBER_OF_REVIEWS is required}" | ||
| : "${NUMBER_OF_THREADS:?NUMBER_OF_THREADS is required}" | ||
| : "${NUMBER_OF_COMMENTS_PER_THREAD:?NUMBER_OF_COMMENTS_PER_THREAD is required}" | ||
|
|
||
| PR_PATH="plans/$PR_NUMBER" | ||
| FETCHED_PATH="$PR_PATH/fetched" | ||
| mkdir -p "$FETCHED_PATH" | ||
| TIMESTAMP=$(date +"%m%d%H%M") | ||
| FETCHED_FILE="$FETCHED_PATH/$TIMESTAMP.json" | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| gh api graphql -f query='query( | ||
| $owner: String!, | ||
| $repo: String!, | ||
| $number: Int!, | ||
| $prComments: Int!, | ||
| $reviews: Int!, | ||
| $threads: Int!, | ||
| $comments: Int!, | ||
| $after: String | ||
| ) { | ||
| repository(owner: $owner, name: $repo) { | ||
| pullRequest(number: $number) { | ||
| comments(first: $prComments) { | ||
| nodes { | ||
| id | ||
| databaseId | ||
| author { login } | ||
| body | ||
| url | ||
| createdAt | ||
| } | ||
| } | ||
| reviews(first: $reviews) { | ||
| nodes { | ||
| id | ||
| databaseId | ||
| author { login } | ||
| body | ||
| state | ||
| url | ||
| createdAt | ||
| } | ||
| } | ||
| reviewThreads(first: $threads, after: $after) { | ||
| pageInfo { | ||
| hasNextPage | ||
| endCursor | ||
| } | ||
| nodes { | ||
| id | ||
| isResolved | ||
| isOutdated | ||
| path | ||
| line | ||
| comments(first: $comments) { | ||
| nodes { | ||
| id | ||
| databaseId | ||
| author { login } | ||
| body | ||
| url | ||
| createdAt | ||
| pullRequestReview { | ||
| id | ||
| databaseId | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }' \ | ||
| -F owner=fedify-dev \ | ||
| -F repo=fedify \ | ||
| -F number="$PR_NUMBER" \ | ||
| -F prComments="$NUMBER_OF_PR_COMMENTS" \ | ||
| -F reviews="$NUMBER_OF_REVIEWS" \ | ||
| -F threads="$NUMBER_OF_THREADS" \ | ||
| -F comments="$NUMBER_OF_COMMENTS_PER_THREAD" \ | ||
| ${LAST_CURSOR:+-F after="$LAST_CURSOR"} \ | ||
| | jq . > "$FETCHED_FILE" | ||
|
|
||
| echo "Fetched reviews and comments are saved to $FETCHED_FILE" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.