readme-updater: make the AI README generator feedback-aware#365
Conversation
Regenerating a README overwrote the branch wholesale, so any human review fixes on the open PR were silently dropped on the next run (e.g. EvilOlaf's "bash/ python" note on the build README's requirements list). Now, before generating, the workflow collects the human review + issue comments (bots filtered out) from the still-open `chore/update-readme` PR for that repo and passes them to generate_readme.py via --feedback. The script folds them into the prompt as corrections to apply and not re-introduce. Reviewer fixes survive a regeneration; the PR body invites reviewers to comment for exactly this reason. Also sharpen the system prompt: when stating what the project is built with / its requirements, name the actual languages and tooling evidenced by the files (shebangs, extensions, manifests, invoked commands) rather than vague generalities -- the root of that review comment. Signed-off-by: Igor Pecovnik <igor@armbian.com>
github-actions
Bot
added
size/medium
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughThe README maintenance workflow collects non-bot reviewer comments from the open README pull request and writes them to Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/maintenance-update-readme.yml:
- Around line 98-100: Update both feedback-gathering gh api jq filters in the
README maintenance workflow to forward comments only when the author has an
explicitly trusted permission or an allowed author_association, while retaining
the existing bot exclusion. Apply the same trust check to review comments and
issue comments before appending them to feedback.txt.
In `@scripts/generate_readme.py`:
- Around line 177-181: Make feedback acquisition fail closed: in
scripts/generate_readme.py lines 177-181, update the feedback-reading logic
around the visible file-read helper to raise or report an explicit error instead
of returning an empty string on OSError; in
.github/workflows/maintenance-update-readme.yml lines 97-100, remove || true
from both GitHub API requests so either failure stops README generation.
- Around line 177-181: Update the feedback-reading logic around the visible
file-open helper so an OSError is reported or propagated when an explicit
--feedback path is supplied, instead of returning an empty string. Preserve the
empty-string behavior only when no feedback file was requested, ensuring
unreadable feedback causes README generation to fail closed.
- Around line 192-202: Limit the reviewer feedback before appending it in the
feedback handling block of the README prompt construction. Apply
MAX_CONTEXT_CHARS or an appropriate dedicated bound to feedback so the combined
prompt remains within the model context limit, preserving the existing
reviewer-feedback instructions and truncating only the feedback content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f196cc2f-9c0d-4378-9b6d-662aba430132
📒 Files selected for processing (2)
.github/workflows/maintenance-update-readme.ymlscripts/generate_readme.py
There was a problem hiding this comment.
Pull request overview
This PR updates the automated README regeneration workflow to preserve human review corrections across reruns by collecting comments from an open chore/update-readme PR and feeding them back into the README generator as prompt “feedback”.
Changes:
- Add
--feedbacksupport toscripts/generate_readme.pyand incorporate reviewer feedback into the generation prompt. - Extend the GitHub Actions workflow to collect review + issue comments from an open README PR into
feedback.txt, and pass it to the generator. - Refine the system prompt to require specific, file-evidenced languages/tooling when describing repo requirements.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/generate_readme.py | Adds feedback loading and prompt augmentation so regenerations retain reviewer corrections. |
| .github/workflows/maintenance-update-readme.yml | Collects PR comments as feedback and forwards them into the generator; updates PR body to instruct reviewers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address review on PR #365: - Security: the feedback filter only excluded bots, so any human commenter on the public README PR could inject text the generator treats as guidance. Gate both gh api jq filters on author_association (OWNER/MEMBER/COLLABORATOR = write access) in addition to the bot exclusion. - Data integrity: drop the `|| true` on the comment fetches so a gh/API error aborts the run instead of silently generating without feedback (which would re-drop the reviewer's fixes). load_feedback() now errors out when an explicit --feedback file can't be read; an empty file is still fine. Signed-off-by: Igor Pecovnik <igor@armbian.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
.github/workflows/maintenance-update-readme.yml:103
gh apiis not paginated here, so only the first page of review/issue comments (typically 30) will be included infeedback.txt. That can silently drop older reviewer feedback on long-lived PRs, defeating the purpose of the feedback loop.
gh api "repos/$REPO/pulls/$pr/comments" \
--jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i")|not) and (.author_association=="OWNER" or .author_association=="MEMBER" or .author_association=="COLLABORATOR")) | "- (\(.path):\(.line // .original_line)) \(.user.login): \(.body)"' >> feedback.txt
gh api "repos/$REPO/issues/$pr/comments" \
--jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i")|not) and (.author_association=="OWNER" or .author_association=="MEMBER" or .author_association=="COLLABORATOR")) | "- \(.user.login): \(.body)"' >> feedback.txt
scripts/generate_readme.py:185
load_feedback()reads the entire feedback file with no size cap. If a PR accumulates a lot of comments (or long pasted logs), this can bloat the prompt beyond the model/context limits and make the job fail unpredictably. Consider truncating feedback to a reasonable maximum and logging when truncation happens.
try:
with open(path, "r", encoding="utf-8", errors="replace") as fh:
return fh.read().strip()
except OSError as e:
|
Addressed both CodeRabbit findings in ed5d28c: 1. Restrict forwarded feedback to trusted reviewers (Security, Major) — both 2. Fail closed on feedback-acquisition failure (Data Integrity, Major) — removed the |
build_context caps embedded file contents to MAX_CONTEXT_CHARS, but the reviewer feedback appended after it was unbounded (load_feedback reads the whole file), so an oversized review could push the combined prompt past the model context limit. Cap feedback at MAX_FEEDBACK_CHARS (16000), truncating only the feedback content and preserving the reviewer-feedback instructions. Signed-off-by: Igor Pecovnik <igor@armbian.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
.github/workflows/maintenance-update-readme.yml:103
gh apiis paginated; without--paginate, only the first page (typically 30) of PR review comments / issue comments will be captured, so older reviewer corrections may be dropped on subsequent runs. Also, GitHub can returnuser: nullfor deleted accounts; referencing.user.loginunguarded would makejqfail and abort the job.
gh api "repos/$REPO/pulls/$pr/comments" \
--jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i")|not) and (.author_association=="OWNER" or .author_association=="MEMBER" or .author_association=="COLLABORATOR")) | "- (\(.path):\(.line // .original_line)) \(.user.login): \(.body)"' >> feedback.txt
gh api "repos/$REPO/issues/$pr/comments" \
--jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i")|not) and (.author_association=="OWNER" or .author_association=="MEMBER" or .author_association=="COLLABORATOR")) | "- \(.user.login): \(.body)"' >> feedback.txt
Address review on PR #365: - Security: the feedback filter only excluded bots, so any human commenter on the public README PR could inject text the generator treats as guidance. Gate both gh api jq filters on author_association (OWNER/MEMBER/COLLABORATOR = write access) in addition to the bot exclusion. - Data integrity: drop the `|| true` on the comment fetches so a gh/API error aborts the run instead of silently generating without feedback (which would re-drop the reviewer's fixes). load_feedback() now errors out when an explicit --feedback file can't be read; an empty file is still fine. Signed-off-by: Igor Pecovnik <igor@armbian.com>
Follow-up to the Update README (AI) workflow, prompted by review feedback on one of its generated PRs (EvilOlaf on armbian/build#10245: the requirements list was too vague — the project is bash + python + tooling).
Problem
Each run regenerated the README from scratch and force-updated the
chore/update-readmebranch, so any human review fix on the open PR was silently dropped on the next run. A reviewers correction never stuck.Change — feed reviewer comments back in
chore/update-readmePR for that repo, intofeedback.txt, and passes it to the generator via--feedback.generate_readme.pyfolds that feedback into the prompt as corrections to apply and not re-introduce — still scoped to this repo, never a license to invent unevidenced content.Also sharpened the system prompt (the root of that comment): when stating what the project is built with / its requirements, name the actual languages and tooling evidenced by the files (shebangs, extensions, manifests, invoked commands) rather than vague generalities.
Validation
py_compile+ YAML parse clean.REVIEWER FEEDBACKblock + the comment text are present; without → absent).jqfilter keeps the human comment (EvilOlaf) and drops the bot (coderabbitai[bot]).matrix.repostill routed through env (no shell injection); reads use the existingACCESS_TOKEN.