Agent Persona Exploration - 2026-04-11

[github-actions[bot]]

Seventh run of the persona exploration series. Six scenarios tested across all five personas, targeting previously identified weak areas (CI failure trigger selection, external secrets handling). All scenarios simulated via general-purpose agents with gh-aw conventions — agentic-workflows agent type remains unavailable as a task tool.

Persona Overview

• Scenarios Tested: 6 (targeting gaps from prior runs)
• Average Quality Score: 4.8/5.0
• Score trend: 4.94 → 4.80 → 4.60 → 4.83 → 3.97 → 4.74 → 4.80

Key Findings

• Trigger selection improved: workflow_run vs check_run question now definitively answered — workflow_run is correct for CI monitoring (fires once per workflow completion, not per job; payload carries run ID for full log access; avoids N duplicate events per run)
• Secrets isolation pattern emerged: The credentials-health-check scenario produced a clean pattern where secrets live only in individual pre-step env: blocks, never in the agent context or frontmatter env: — agent receives only "valid" / "invalid" status strings
• network: defaults: false legitimate: When the agent makes no direct outbound HTTP calls (all reads via GitHub MCP, writes via safe-outputs), setting defaults: false is valid and more restrictive
• update-release confirmed real: The update_release safe-output exists with operation: replace|append|prepend — previously under-documented; repos (not releases) is the correct MCP toolset for release operations

Top Patterns (all 6 scenarios)

1. Pre-job data collection — heavy I/O (git checkouts, npm builds, API calls, log downloads) runs in a jobs: pre-step before the agent, which receives only structured summaries
2. Strict security defaults — strict: true (5/6), read-only permissions, all writes via safe-outputs, explicit bash allowlists
3. PR workflow hygiene — paths: filter + cancel-in-progress: true + update-existing: true on comments applied consistently to all 4 PR-triggered workflows
4. Explicit noop paths — every prompt names the exact noop condition with a ready-to-use message template
5. Anti-spam patterns — update-existing: true (PR comments), close-older-issues: true (recurring scheduled issues), dedup via issue search before create

High Quality Responses (all 6 scored 4.8/5)

CI Failure Analyzer (DevOps / workflow_run) — Score: 4.8

• Correctly chose workflow_run + if: conclusion == 'failure' guard, explained why vs check_run
• Pre-step fetches logs and runs grep heuristics; agent reads filtered hint files only
• cache-memory for cross-run flakiness tracking; dedup search before creating issue
• Issue title includes test name for future dedup searches

Credentials Health Check (DevOps / schedule) — Score: 4.8

• Secrets scoped to individual step env: — strongest secrets isolation seen in any run
• Pre-step emits only status enum values; agent never sees token values
• if: needs.check-credentials.outputs.any-failed == 'true' skips agent entirely on clean run
• close-older-issues: true prevents stale "rotate credentials" issue accumulation

OpenAPI Breaking Change Detector (Backend / pull_request) — Score: 4.8

• sparse-checkout for openapi.yaml only — fast regardless of repo size
• change-summary 5-value enum (breaking/clean/new_file/file_deleted/error) makes every branch explicit
• persist-credentials: false on both checkouts; oasdiff output capped at 50KB

Accessibility Audit (QA / pull_request + Playwright) — Score: 4.8

• Adaptive rendering: detects Storybook vs dev server vs HTML harness
• axe-core injected from local npm install (avoids CDN on network.allowed)
• Scoped axe.run('#audit-root') prevents harness noise in results

Release Notes Generator (PM / release: published) — Score: 4.8

• <!-- rng:generated --> marker as idempotency guard prevents overwriting manual edits
• Correct repos toolset (not releases) for list_releases/get_release_by_tag
• operation: replace vs prepend — correctly chose replace for full body ownership

Bundle Size Monitor (Frontend / pull_request + dual-branch build) — Score: 4.8

• Dual-branch build solved cleanly in jobs: pre-step with rm -rf dist node_modules between branches
• network: defaults: false — agent needs no outbound network
• add-label with allowed: ["bundle-size-warning"] list for safety

Areas for Improvement

• Trigger decision tree underdocumented — workflow_run vs check_run vs pull_request choice is non-obvious and recurs across runs. A concise decision matrix in .github/aw/create-agentic-workflow.md would reduce trial-and-error.

• Secrets isolation pattern not yet a named pattern — The pre-step env-scoping approach for external APIs is excellent but ad-hoc. Naming it (e.g., "secrets firewall pattern") and documenting it in .github/aw/github-agentic-workflows.md would help users discover it.

• strict: false trigger unclear — Agents sometimes set strict: false without fully understanding why. The current reason (inline shell in jobs: blocks) should be documented explicitly alongside what relaxations strict: false grants.

• update-release discoverability — The PM scenario uncovered that update_release is a real safe-output but users asking "how do I update a release body?" won't find it easily. Adding an example to .github/aw/ docs would close this gap.

Recommendations

1. Add workflow_run trigger guide to .github/aw/create-agentic-workflow.md — include decision matrix (workflow_run vs check_run vs pull_request) with the key insight: workflow_run fires once per full workflow completion and provides run ID for log access
2. Document secrets isolation pattern in .github/aw/github-agentic-workflows.md — name the "pre-step secrets firewall" pattern: secrets in individual step env: blocks, agent receives only sanitized status outputs
3. Add update-release example to .github/aw/ documentation — it's a real safe-output that's underused; include operation: replace|append|prepend options and the repos toolset (not releases) for reading release data

References:

• §24273696446

Generated by Agent Persona Explorer · ● 3.4M · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agent Persona Explorer.

A newer discussion is available at Discussion #25976.