feat(carve S3): agent runtime — deterministic issue feedback, build/lint gate, plan and restack workflows - #654
Conversation
65ec18d to
b4c5d1d
Compare
9e2dda7 to
4f947ee
Compare
Automated review — carve S3 (#654)Reviewed as its own diff only ( Every finding below was reproduced independently before reporting. Findings I could not reproduce were dropped rather than reported as "possible". Slice standalone-nessTypecheck: PASS. Reverts: CLEAN on the named risk. I checked the #643 Jira app-actor work line-by-line, since this slice contains 3-way merges and the fork was behind 1. BLOCKING — the default agent model changes here, but the IAM grant for it lands 5 slices later
Verified per-slice:
The fallback is reachable: it fires when the per-repo Failure: merge any of #654–#659 alone, deploy, onboard a repo without This is the clearest slicing defect in the stack: a behaviour change and the permission it requires are on opposite sides of a five-slice boundary. Either move the 2. MAJOR — 9 sites cite ADR-016 as the authority for "no Linear MCP"; ADR-016 does not say that until #662This slice adds code comments across 6 files (
So #654–#659 ship code justified by a decision the repo does not document, and a user-visible log line pointing at a doc that does not support it. Either move the ADR-016 revision into this slice, or drop the citations until the ADR lands. 3. MAJOR — self-reclone guard is bypassed by
|
| command | result |
|---|---|
git clone https://github.com/acme/app /workspace/app |
BLOCKED ✅ |
git clone --branch main https://github.com/acme/app … |
BLOCKED ✅ |
git clone -b main https://github.com/acme/app … |
ALLOWED ❌ |
git commit -m wip && gh repo clone acme/app … |
ALLOWED ❌ |
gh pr create --body "do not run gh repo clone acme/app" |
allowed ✅ (correct) |
git clone -b <branch> <own repo> is an ordinary single command, not a contrived chain. It reopens exactly the lost-work failure this guard was built to prevent: the agent works in the nested clone, the outer tree stays clean, and the delivery gate sees nothing. Fix is to require the free-text flag to appear after the clone verb, or to scan the whole string for the verb and only use the truncation to suppress a match found inside a quoted value.
4. MAJOR — private tracker ids and work-item shorthand used as the explanation for code
Confirmed by sweeping the added lines. These are not fixtures — they are the load-bearing explanation in comments and docstrings, unresolvable to anyone outside the author's team:
ABCA-###as the reason for a change — 11 sites inagent/src/pipeline.py(lines 559, 639, 671, 718, 1053, 1058, 1112, 1379, 1404, 1518) andagent/src/post_hooks.py:381. Highest-impact:_apply_delivery_gate, where every load-bearing sentence is anchored to "ABCA-815".PM-3as a section marker inlinear_reactions.pyandpipeline.py.- Dated incident framing:
live-caught on backgroundagent-dev(pipeline.py:1034) — names a private stack.
Test/fixture data carrying tracker ids is fine and I have not reported any. The fix is to state the invariant plainly ("a task that reported success but shipped nothing must fail, or the platform reports success for lost work") and drop the id.
5. MINOR (downgraded) — _NO_CLARIFICATION_WORKFLOW_IDS contains a workflow id that never matches
agent/src/runner.py:355 lists "web/research-v1"; the tree's only research workflow is knowledge/web-research-v1, and git grep 'web/research-v1' returns exactly one hit — this line. The opt-out therefore never fires for research tasks.
I am reporting this minor, not major: it is a stale entry in a deny list, so the consequence is that the clarification tool is offered where it was meant to be withheld — a wasted tool offering, not a broken path. The reviewing agent's claim that it breaks the research flow outright is not something I could confirm, so I am not asserting it.
Reviewed with Claude Code. Verification: tsc --noEmit run on this slice's own base; #643 survival checked line-by-line per slice; the clone-guard regexes executed against real command strings; ADR-016 and grant-list contents diffed per slice.
b4c5d1d to
5b75520
Compare
4f947ee to
77677a0
Compare
Note: force-push during review — one finding is now FIXEDMy review above was written against Finding 1 (BLOCKING — the The remaining findings all still hold at
Of these, #3 is the one I would prioritise: |
77677a0 to
fcc18a3
Compare
Review findings addressedBLOCKING #1 — model default flips here, IAM grant lands 5 slices later. Confirmed and fixed. Worth recording how I verified it, because my first check said the opposite: grepping Fixed by moving the grant entry into this slice, alongside the default that needs it. Also added a drift guard that reads the agent's own fallback out of MAJOR #3 — the clone guard is bypassed by
Your diagnosis of the mechanism was exactly right. The fix takes it a step further: the command is split on shell separators first, so a free-text argument only swallows the rest of its own segment — that is what catches the MAJOR #2 — 9 sites cite ADR-016 for a decision the repo does not document until the last slice. Confirmed (0 occurrences of "mcp" in that ADR here; 38 at the activation slice). Not moved, and here is why: the ADR revision is bundled with several other doc changes in the activation slice's bucket, and pulling one section forward would either split that file across two PRs or drag unrelated doc changes into this one. The citations are accurate about the final state of the stack, and the stack merges in order. If you would rather the ADR revision move here, that is a clean change and I will make it — flag it and I will. MAJOR #4 — private tracker ids as the load-bearing explanation. Fixed. ~60 sites across MINOR #5 — Gates: 2544 CDK tests, 1440 agent tests, |
5b75520 to
f2b5774
Compare
4b4a3d6 to
868197c
Compare
63ae670 to
fd0834a
Compare
Second-round review: my clone-guard fix broke two things it was meant to protectA second adversarial pass over my own fix found that it closed the New bypass — a wrapped clone walked straight through. Splitting on a bare newline separated the verb from the repo, so the slug was never in the verb's segment:
So the New false positive — a multi-line PR body was denied. A Both are mutation-verified: restoring Also removed Two lower-severity findings, verified and not changed:
Verified as still-correct across a 50-case attack table: the Gates: 1442 agent tests, 2545 CDK, |
fd0834a to
4367dc5
Compare
scottschreckengaust
left a comment
There was a problem hiding this comment.
Verdict: Approve with nits
Slice S3 of the linear-vercel carve. Reviewed only the incremental diff vs the parent branch carve/s2-ecs-rightsized-planning (56 files: agent runtime + tests, two workflow YAMLs, and the CDK Bedrock-grant / workflow-descriptor entries that must travel with the runtime). The change is large but coherent, exceptionally well-documented at the point of every non-obvious decision, and backed by a green suite (1421 tests, +28 new verify-command tests; all 4 CI checks pass). No blocking issues found.
Vision alignment
Advances the fire-and-forget tenet: moving issue feedback from a model-mediated MCP tool into deterministic pipeline steps makes a reaction/comment a consequence of reaching a state rather than a model choice — fewer moving parts, and tracker credentials leave the model's tool surface (blast-radius reduction). The build/lint gate keeps outcomes reviewable by catching breakage before a PR is opened. decompose-v1 (read-only, artifact terminal) and restack-v1 (writeable, PR terminal) are correctly scoped and platform-issued, not user-facing — consistent with policy-gated escalation. Backed by approved P0 issues #247 (parent/sub-issue orchestration) and #299 (auto-decomposition); rolled up under epic #668. Governance gate satisfied.
Blocking issues
None.
Non-blocking suggestions / nits
is_infra_failurekeys OOM detection on exit 137, but the new streaming path (_run_cmd_streaming) execs viasubprocess.Popenand returns the raw signal code — a SIGKILL is-9, not137(agent/src/shell.py:302,agent/src/post_hooks.py:107-136). For the defaultmise run build(direct-exec, notbash -lc-wrapped), if the exec'd process is itself OOM-killed with no ENOSPC/OOM string on stderr,is_infra_failure(-9, ...)returns False and the gate mislabels an infra kill as a genuine build failure — exactly the failure mode the function documents preventing. In practice the OOM killer usually reaps a child (node/jest) andmisepropagates a shell-convention 137, and the pre-existing bufferedsubprocess.runpath shares this negative-code behavior, so this is not a regression introduced here — but consider normalizingabs(returncode)/ adding-9/-15to the infra signatures so the direct-exec streamed path is covered symmetrically. Worth a follow-up issue rather than a change in this slice._run_cmd_streaming(cmd, label, cwd, timeout)acceptslabelbut never uses it (agent/src/shell.py:241-242) — theCMD:prefix line is logged byrun_cmdbefore delegation. Drop the parameter or thread it into the drain-thread log prefix; a dead parameter in a security/observability-critical helper invites confusion.- Default agent model bumped to
us.anthropic.claude-opus-4-8(agent/src/models.py,agent/src/config.py:553). Correctly co-landed with the Bedrock grant entry (cdk/src/constructs/bedrock-models.ts) and a drift-guard test that reads the agent's own fallback and asserts the grant covers it — this closes the AccessDenied-at-turn-0 gap the commit describes. Just flagging the cost/behavioral change for downstream awareness; it is disclosed transparently in the PR body rather than buried.
Documentation
Comment/docstring cleanup removes private tracker IDs (ABCA-###, #NNN) and preserves the reasoning in plain language — a net improvement for a public sample repo. No docs/guides/ or docs/design/ source edits in this slice, so the Starlight mirror-sync gate is N/A. Workflow behavior is documented inline in the two new agent/workflows/coding/*.yaml descriptors. Issue tracking is reflected via #247/#299/#668.
Tests & CI
Strong coverage: test_verify_commands.py (28 tests) exercises timeout/inert/infra-137/genuine-failure classification including the dangerous 137-with-plain-output fall-through; test_shell.py covers the streaming tee, stdout/stderr separation, redaction, and check=True raise; test_hooks.py mutation-verifies the clone-guard fixes (restoring \n as a separator fails the multi-line-body test; dropping the continuation-join fails the wrapped-clone test). New CDK descriptors (restack-v1, decompose-v1) have matching YAML files, prompt sections, and admission-table + prompt tests. Bootstrap synth-coverage: N/A — the CDK diff only adds a model-ID string to an existing InvokeModel grant list and two entries to the in-memory DESCRIPTORS table; it introduces no new CloudFormation resource type, so the ADR-002 bootstrap-policy checklist does not apply.
Review agents run
- code-reviewer (principal-architect hand-review over the incremental diff) — ran; findings above.
- silent-failure-hunter — ran (in scope:
linear_reactions.py,shell.py,post_hooks.pyerror handling). Best-effort WARN-and-continue inlinear_reactions.pyis appropriate for non-critical feedback; the verify path deliberately distinguishes timed_out/inert/infra_failed from genuine failure rather than swallowing — no silent-default traps found. The one residual concern (nit #1) is a mislabel, not a swallow. - type-design-analyzer — ran (in scope: new
VerifyOutcomedataclass,TaskConfignew fields).VerifyOutcomewith explicitpassed/timed_out/inert/infra_failedflags is a clean, well-justified widening of the old bare-bool; newbuild_command/lint_commanddefault to""with sensible fallback. No concerns. - comment-analyzer — ran. Comments are accurate and reasoning-rich; the tracker-ID scrub preserves substance. No stale/misleading comments found.
- pr-test-analyzer — ran; coverage assessment above. No happy-path-only gaps identified for the new logic.
- /security-review — ran (in scope: the
_is_self_recloneclone-guard rewrite and the Bedrock IAM grant). The guard now joins backslash-continuations before segmenting, checks each shell segment independently, and locates the clone verb before the free-text scan — closing the-b/--branchbypass, the wrapped-clone bypass, and the multi-line---bodyfalse positive, all mutation-pinned. Fail-closed on non-dict tool input is preserved. Grant remains scoped to the enumerated model IDs. No new secrets, network egress, or input-gateway surface. (semgrep MCP scan attempted but the plugin required an explicit--config; substituted the manual principal security pass over the two in-scope files.)
Human heuristics
- Proportionality — Pass. The atomic 56-file slice is justified (splitting produced 39 unresolved-symbol / non-compiling intermediate states; the no-MCP/build-gate/plan/restack hunks interleave within
pipeline.py/repo.py). Abstractions match the problem. - Coherence — Pass.
restack-v1/decompose-v1follow the existing descriptor + YAML + prompt-section pattern with real substance;VerifyOutcomereuses the established gating vocabulary; the sameread_only/requires_repoinvariants are threaded consistently CDK↔agent. - Clarity — Concern (nit #2): the unused
labelparam in_run_cmd_streamingobscures intent in an observability helper. Otherwise names communicate intent and error handling surfaces (not hides) failures. - Appropriateness — Pass. Integration behavior is verified against real subprocess/shell semantics (streaming, signal exit, shlex vs
bash -lc), not only self-written mocks; the clone-guard tests assert what the guard should do and are mutation-verified. Maintainable by this team.
| t_out.join(timeout=10) | ||
| t_err.join(timeout=10) | ||
| return subprocess.CompletedProcess( | ||
| cmd, proc.returncode or 0, "\n".join(out_lines), "\n".join(err_lines) |
There was a problem hiding this comment.
Nit (non-blocking): the streamed path returns proc.returncode from subprocess.Popen, which is the raw signal code for a killed process (SIGKILL → -9), not the shell's 128+signal convention. is_infra_failure (post_hooks.py:107) keys OOM detection on returncode == 137, so a direct-exec mise run build OOM-killed with no ENOSPC/OOM stderr string would be classed as a genuine build failure rather than an infra fault — the exact mislabel that function exists to prevent. Not a regression (buffered subprocess.run shares this), but consider normalizing to abs(returncode) or adding -9/-15 to the infra signatures. Also proc.returncode or 0 maps a None (shouldn't occur post-wait) to 0=success — a fail-open in the unlikely edge; prefer an explicit -1 default.
a837ad2 to
7ecfba5
Compare
4367dc5 to
0398ec2
Compare
…lint gate, plan/restack) Slice 3 of the carve — the full autonomous-agent runtime, landed as ONE unit. Stacked on S2. Unlike the CDK subsystems, the agent Python evolved as a tightly- coupled whole: the changes to pipeline.py / repo.py / post_hooks.py / shell.py / models.py interleave WITHIN shared files (a single pipeline.py carries the no-MCP, decompose-plan, restack, and clarify-resume hunks). Splitting it along the CDK subsystem lines produces non-compiling intermediate states, so it lands atomically: all 27 agent/src modules + their tests + the two workflow definitions. What it brings: - Linear no-MCP: strip the Linear MCP server; the platform pre-hydrates comments + attachments deterministically instead of the agent fetching them at runtime. - Per-repo build/lint verification gate before opening a PR (build_command / lint_command), with a structured verify outcome (passed / timed-out / infra-failed). - Stacked-child branch handling + predecessor re-merge (for the sub-issue DAG). - Agent-native decompose + restack workflows (clone, plan/re-stack, emit artifact). - Attachment download + screening + version-pinned integrity read. Behavioral note for reviewers: this also moves the default agent model and a couple of runtime defaults to their current values — intrinsic to the runtime as it runs today; called out here rather than split out, to keep the runtime internally consistent. Gates: agent ruff + ty + full pytest (1421 tests) green. Files are verbatim from the source branch (combination-verified: 0 diffs vs source). Comment cleanup for public readability follows as a separate commit on this branch.
…ble-workflow set The agent-vs-CDK cross-check for the writeable-workflow constant matched only a single-line `frozenset((...))`. The formatter renders the same literal across several lines once it grows past the line limit, and the pattern then found nothing — so the check failed on a purely cosmetic reformat instead of on a real drift between the two lists. Allow whitespace between the frozenset call and its inner tuple so the assertion tests the set's contents, which is what it is for.
…t-side changes The agent-side files in this slice were taken wholesale from the branch that carries the orchestration work. That branch predates the Jira app-identity feature already on the target, so copying its versions over silently dropped that feature from all 13 agent files — the app-actor env plumbing, the per-task credential scrub, and their tests — while the CDK, CLI and Forge halves of the same feature stayed. The result would have merged as a half-reverted feature. Reapply the agent-side changes as a three-way merge against the common ancestor instead of a wholesale copy, so both survive: the app-identity path and this slice's own agent changes. Verified per file that the result is exactly "branch version + app-identity delta" with no other drift, and that a symbol removed on purpose here (the Linear MCP endpoint) stays removed. One docstring conflicted, describing how Jira comments get posted; kept the target's wording since it names the app actor and is now correct.
…e model the agent defaults to Three fixes from the automated review of this slice, plus its comment cleanup. **The self-reclone guard let `git clone -b` through.** `-b` is `gh`'s `--body` short form, so it sits in the free-text-argument list, but it is ALSO `git clone`'s own `--branch`. The guard truncated the command at the first free-text flag BEFORE scanning for the clone verb, so the cut landed ahead of the very verb it was looking for and an ordinary `git clone -b main <own repo>` was allowed — reopening the stranded-work failure the guard exists to prevent. Two more shapes slipped through for the same reason: `gh repo clone -b main <own repo>`, and a clone chained after an unrelated `-m` (`git commit -m wip && gh repo clone <own repo>`), where the `-m` belongs to the commit, not the clone. Now each shell segment is checked on its own, and within a segment the verb is located first — a verb only counts as prose if a free-text argument opens before it in that same segment. Prose in a `--body` value is still ignored. **The agent's fallback model was not in the Bedrock grant list.** This slice flips the fallback to Opus 4.8 for a repo that pins no model, but the IAM grant is derived from `DEFAULT_BEDROCK_MODEL_IDS`, and that entry was landing several changes later. Merging this slice on its own would have produced AccessDenied at turn 0 on every task with no per-repo model. The grant now travels with the default, and a drift guard reads the agent's own fallback and asserts the grant covers it, so the two cannot diverge again. **Two workflow descriptors arrived here rather than in the base slice**, matching where their `agent/workflows/**` definitions live. `DESCRIPTORS` is the live admission table, so an entry without its file means a submission is accepted with a 201 and then dies when the agent cannot load it. Also removes private tracker ids and work-item shorthand used as the load-bearing explanation in comments, docstrings, log messages and test names. The reasoning is preserved in plain language — a comment that recorded a real defect still records it, just without an id only this team can resolve.
…to protect A second review pass on my own fix found that it closed the `-b` bypass while opening a new one and introducing a false positive. Both are worse than the original bug in a security guard, so both are fixed and pinned. **New bypass: a wrapped clone walked straight through.** Splitting on a bare newline separated the verb from the repo, so `git clone \` + newline + url never had the slug in the verb's segment. That command was BLOCKED before my rewrite and ALLOWED after it — including via the `-b` form the rewrite existed to catch. Backslash-continuations are now joined before scanning, which also covers a continuation splitting the verb itself (`git \` + newline + `clone`) — the case that genuinely needs the joining, since no segment handling reunites those. **New false positive: a multi-line PR body was denied.** A `--body` or `-m` value spanning lines is how an agent normally writes a PR or commit body, and that text routinely documents a clone command. Treating `\n` as a command separator put the quoted line in its own segment with no preceding free-text flag, so a legitimate `gh pr create` was refused — exactly the false positive the free-text list exists to prevent. A bare newline is no longer a separator; the real separators still are, so a clone chained after an unrelated `-m` is still caught. Also removes `_CLONE_VERB_RE`, which was never load-bearing: across every segment shape it was the sole matcher zero times, and it actually MISSES a subshell (`( git clone …`) that the anchored pattern catches. Dead alternatives in a security path are worse than none. Both regressions are mutation-verified: restoring `\n` as a separator fails the multi-line-body test, and dropping the continuation join fails the wrapped-clone test.
0398ec2 to
e8a74cb
Compare
Stacked on #653 → #647. Review those first; this PR's diff is only its own slice.
What this changes
The Python agent runtime. Three related shifts:
1. Issue feedback becomes deterministic instead of tool-mediated. The agent previously reached the issue tracker through an MCP server, which made a status update a model decision — it could be skipped, duplicated, or worded differently each run. Feedback now happens in the pipeline, so a reaction or comment is a consequence of the run reaching a state, not of the model choosing to report it. Fewer moving parts at runtime and no tracker credentials in the model's tool surface.
2. A build/lint gate before delivery. The runtime can run a repo's configured build and lint commands and treat the outcome as part of whether the work is deliverable, rather than opening a pull request and letting CI discover the breakage.
3. Two new workflows —
decompose-v1(produce a plan for a piece of work) andrestack-v1(rebase a stack of dependent branches after an earlier one changes).Why this is one slice
The 49 files are one atomic unit. The no-MCP, build-gate, plan and restack changes interleave within
pipeline.pyandrepo.py— attempting to take a subset produced 39 unresolved-symbol errors, because each piece needs models, hooks and shell helpers introduced by the others. Splitting it would mean shipping something that doesn't compile.Behavioral changes to be aware of
Verification
ruff+ty+pytest(1421 tests) all green.Tracking
Slice S3 of the linear-vercel → main carve. Tracking issue: #668 (slice table, why it is sliced rather than merged, and review guidance).
Lands part of #247 (parent/sub-issue orchestration) and #299 (auto-decomposition). Deliberately not
Closes— no single slice closes either; they close when S8 activates the arc.Stacked on #653 — review that first; this PR's diff is against its branch.