diff --git a/logs/local/watchdog_cycles/20260717_cycle.md b/logs/local/watchdog_cycles/20260717_cycle.md new file mode 100644 index 00000000..ed53d935 --- /dev/null +++ b/logs/local/watchdog_cycles/20260717_cycle.md @@ -0,0 +1,490 @@ +## 2026-07-17T00:15:00Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` contains only the completed `print_structured()` helper work. No active operator work order or unchecked definition-of-done items were present. + +### STEP 1 — Preflight +- Repo sync: all managed repos with local checkouts reported `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running (`intake`, `goal`, `test`, `improve`, `propose`, `review`, `spec`, `watchdog`). +- CLIs present. Working tree clean before cycle-local logging. + +### STEP 2 — Investigate +- `operations-center-custodian-sweep --config config/operations_center.local.yaml --emit`: exit `0`; `repos_swept=19`, `total_violations=0`, `all_zero=true`. +- `operations-center-ghost-audit --config config/operations_center.local.yaml --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10`, status `fixed`. +- `operations-center-flow-audit --config config/operations_center.local.yaml`: exit `0`; `total_open_gaps=1`; open gap is `F8` (`back-pressure on Ready queue`), `status=partial`, `count=0`. +- `operations-center-graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `operations-center-reaudit-check --json`: exit `0`; neither backend needs re-audit. +- `operations-center-check-regressions --config config/operations_center.local.yaml --lookback-hours 1 --dry-run`: exit `0`; `findings=[]`. +- Executor/OOM investigation: + - board-worker log tail shows backend-capacity/session-limit failures on goal tasks, but no executor signal-kill signature in the current cycle + - `dmesg` access is not permitted in this environment (`Operation not permitted`) + - `journalctl -k --since "2h ago" | grep -iE "killed|oom"` returned no lines + - `free -h` showed about `22Gi` available memory with swap unused +- `logs/autonomy_cycle/quiet_diagnosis.json` now reports `quiet_window_cycles=5`, `all_cycles_had_zero_created=true`, `diagnosis_kind=proposer_no_create`, with aggregated suppression reasons `family_deferred_initial_gating=35` and `cooldown_active=4`. +- Direct Plane snapshot on Friday, July 17, 2026: `Backlog=93`, `Blocked=6`, `In Review=4`, `Done=106`, `Cancelled=141`, `Ready for AI=0` (`total=350`). +- Live blocked set remains: + - `5ada8f03-538b-4790-9ebc-1f4772369ac9`, `9a569770-41cb-40ae-87e9-3498fe8f79ea`, `8064fa16-0eec-4d9e-966f-e11011ee30d4`, `1a3b14b1-b496-4c9d-b204-a4511f1c9f14` for `OperationsCenter` + - `71873497-9df1-4584-a869-5c8c80dc2936` for `VideoFoundry` + - standing operator/admin branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` + +### STEP 3 / 3.5 — Triage and unblock +- `operations-center-triage-scan --config config/operations_center.local.yaml --apply`: exit `0`; `rescore=[]`, `awaiting=[]`, `queue_healing=[]`. +- `operations-center-board-unblock --config config/operations_center.local.yaml --apply`: exit `0`; `actions=[]`, `cooldown_skip_reason=null`, `mem_available_gb=22.1`. +- `operations-center-detect-convergence-stall --config config/operations_center.local.yaml --json`: exit `0`; `stalls=[]`, `escalated=0`, `suppressed=0`. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed the newest prior cycle summaries in `logs/local/watchdog_cycles/20260716_cycle.md`. +- `NEW_EVIDENCE_DETECTED = yes` this cycle: + - cooldown has cleared (`cooldown_skip_reason=null`) while `Ready for AI` remains `0` + - the quiet diagnosis still reports 5 consecutive no-create proposer cycles + - the live board remains at `Blocked=6` with the same blocked set still present +- Starvation remains active this cycle per watchdog rules: + - proposer has created zero tasks for 5 consecutive cycles while blocked work remains on the board + - `Ready for AI=0` and the current maintenance passes produced no queue movement after cooldown cleared + - no repo-scoped regression or graph/runtime outage explains the lack of new work +- Behavioral convergence classification: `NON-CONVERGENT` for the proposer/queue-governance path. +- Existing Plane escalation `b27f9e00-70a8-4f58-a7f2-5d0ee28b8f2d` (`[Convergence] Proposer quiet for 5 cycles while blocked work remains on board`) remains current. Added a fresh comment with the Friday, July 17, 2026 board snapshot and the post-cooldown starvation evidence rather than creating a duplicate task. +- The standing branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` remains separately operator-blocked, but it is not the worst condition this cycle. + +### STEP 6 / 7 — Execution gate and direct fix +- No finding passed the direct-fix gate. +- No repo-scoped OperationsCenter source-code defect was reproduced from current-cycle audit output. The live issue is platform starvation / queue-governance behavior plus the standing external branch-protection/admin blocker, so no autonomy-cycle dispatch was initiated. + +### STEP 8 — Invariant enforcement +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` → `15 passed in 2.06s`. +- One slow-test warning only: `tests/unit/er000_phase0_golden/test_golden.py::TestBoundaryNoExampleManagedRepoImports::test_no_example_managed_repo_imports_in_oc_src` at `1.519s`. + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- `grep -h "watcher_restart\|exit_code\|ERROR\|Traceback" logs/local/watch-all/*.log | tail -50` showed: + - repeated review-lane branch-protection refusals for `ProtocolWarden/TeamExecutor#18` + - benign `exit_code=143` restarts for `review`, `propose`, `goal`, `test`, `improve`, and `spec` + - no fresh non-143 watcher crash and no unexpected clean exit +- No restart was warranted. + +### STEP 10 — Commit / push / PR +- No source, config, or test files were changed this cycle; only cycle logging, Plane-comment evidence update, and schedule update. No branch, commit, push, or PR were created. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: on Friday, July 17, 2026 the proposer/queue-governance path remains in starvation after cooldown cleared: `quiet_window_cycles=5`, `all_cycles_had_zero_created=true`, `Blocked=6`, `Ready for AI=0`, and the current maintenance passes produced no queue movement. +- Delay: `600s`. + +## 2026-07-17T00:31:00Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` still only contains the completed `print_structured()` helper work. No active operator work order. + +### STEP 1 — Preflight +- Repo sync: all managed repos `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running. +- CLIs present. Working tree clean. + +### STEP 2 — Investigate +- `custodian-sweep`: exit `0`; `all_zero=true`, `plane=skipped-zero-findings`. +- `ghost-audit --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10` (`status=fixed`). +- `flow-audit`: exit `0`; `total_open_gaps=1`; open gap is `F8` (`back-pressure on Ready queue`), `status=partial`, `count=0` — same as prior cycle, not new. +- `graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `reaudit-check`: exit `0`; neither backend needs re-audit. +- `check-regressions --lookback-hours 1`: exit `0`; `findings=[]`. + +### STEP 3 / 3.5 — Triage and unblock +- `triage-scan --apply`: exit `0`; `rescore=[]`, `awaiting=[]`, `queue_healing=[]`. +- `board-unblock --apply`: exit `0`; `actions=[]`, `cooldown_skip_reason=null` (cooldown remains clear), `mem_available_gb=22.43`. +- `detect-convergence-stall --json`: exit `0`; `stalls=[]`, `escalated=0`, `suppressed=0`. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed `logs/local/watchdog_cycles/20260717_cycle.md`'s prior 00:15Z entry. +- `logs/autonomy_cycle/quiet_diagnosis.json` unchanged since the prior cycle: `quiet_window_cycles=5`, `diagnosis_kind=proposer_no_create`, `aggregated_suppression_reasons={family_deferred_initial_gating:35, cooldown_active:4}`. The propose watcher heartbeat shows a fresh idle success at `00:32:01Z` but the diagnosis file itself did not regenerate with new content, i.e. this propose run again created 0 tasks. +- `NEW_EVIDENCE_DETECTED = no` this cycle — same quiet-window count, same suppression-reason distribution, same `Ready for AI` starvation shape as the immediately prior cycle (which itself had the "new" evidence of cooldown clearing). This is the first no-new-evidence cycle since the escalation was created at 23:44Z. +- Traced the dominant suppression reason (`family_deferred_initial_gating`) to `src/operations_center/decision/service.py:157-170`: candidates whose family falls outside `_DEFAULT_ALLOWED_FAMILIES` (7 of 14 total families; only `observation_coverage` is active this window) are suppressed by design when `propose` runs without `--all-families`. This is intentional gating, not a code defect — widening the default allowed-family set is a runtime-policy change (STEP 6 gate condition f), so it is explicitly out of scope for a direct watchdog fix. +- Starvation remains active (proposer created zero tasks across 5+ consecutive cycles while blocked work exists on the board). Behavioral convergence classification: `NON-CONVERGENT` for the proposer/queue-governance path, unchanged from prior cycles. +- Existing Plane escalation `b27f9e00-70a8-4f58-a7f2-5d0ee28b8f2d` remains current and covers this condition. Added a comment with this cycle's no-new-evidence status and the traced root-cause detail (source location, why it's policy-not-bug, and the open question of why the one active family itself yields zero creates) rather than opening a duplicate task. +- Per PARK TRANSITION rules, this is only the first no-new-evidence cycle (need 2+ consecutive) — remaining in `STALLED`, not yet `PARKED_OPERATOR_BLOCKED`. +- The standing branch-protection blocker (`fe58f9fd-38cd-4c9f-9474-28bc49b86435`) remains separately operator-blocked and unchanged. + +### STEP 6 / 7 — Execution gate and direct fix +- No finding passed the direct-fix gate: the live issue is proposer/queue-governance starvation (already classified, retry gated off) plus a policy-gating design choice, not a reproduced repo-scoped code defect. No autonomy-cycle dispatch was initiated. + +### STEP 8 — Invariant enforcement +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` → `15 passed in 2.65s`. +- One slow-test warning only (same test as prior cycles, `1.735s`). + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- Log tail showed only the known repeated `ProtocolWarden/TeamExecutor#18` branch-protection refusals (standing operator-blocked item) and benign `exit_code=143` restarts for `review` and `goal`. No fresh non-143 crash, no unexpected clean exit. + +### STEP 10 — Commit / push / PR +- No source, config, or test files changed this cycle — only cycle logging and a Plane-comment evidence update. No branch, commit, push, or PR created. Git tree confirmed clean on `main`. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: proposer/queue-governance starvation continues unchanged (`quiet_window_cycles=5`, `Ready for AI=0`, same suppression-reason distribution as the prior cycle); this is the first no-new-evidence cycle since the 23:44Z escalation, so not yet eligible for `PARKED_OPERATOR_BLOCKED` (requires 2+ consecutive no-new-evidence cycles). +- Delay: `600s`. + +## 2026-07-17T00:50:00Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` still contains only the completed `print_structured()` helper work. No active operator work order or unchecked definition-of-done items were present. + +### STEP 1 — Preflight +- Repo sync: all managed repos with local checkouts reported `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running (`intake`, `goal`, `test`, `improve`, `propose`, `review`, `spec`, `watchdog`). +- CLIs present. Working tree clean before cycle-local logging. + +### STEP 2 — Investigate +- `operations-center-custodian-sweep --config config/operations_center.local.yaml --emit`: exit `0`; `repos_swept=19`, `total_violations=0`, `all_zero=true`. +- `operations-center-ghost-audit --config config/operations_center.local.yaml --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10`, status `fixed`. +- `operations-center-flow-audit --config config/operations_center.local.yaml`: exit `0`; `total_open_gaps=1`; open gap remains `F8` (`back-pressure on Ready queue`), `status=partial`, `count=0`. +- `operations-center-graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `operations-center-reaudit-check --json`: exit `0`; neither backend needs re-audit. +- `operations-center-check-regressions --config config/operations_center.local.yaml --lookback-hours 1 --dry-run`: exit `0`; `findings=[]`. +- Executor/OOM investigation: + - board-worker log tail still shows historical backend-capacity/session-limit failures on goal tasks, but no current-cycle executor signal-kill signature + - `dmesg` access remains unavailable in this environment (`Operation not permitted`) + - `journalctl -k --since "2h ago" | grep -iE "killed|oom"` returned no lines + - `free -h` showed about `22Gi` available memory with swap unused +- `logs/autonomy_cycle/quiet_diagnosis.json` remains unchanged on Friday, July 17, 2026: `quiet_window_cycles=5`, `all_cycles_had_zero_created=true`, `diagnosis_kind=proposer_no_create`, `aggregated_suppression_reasons={family_deferred_initial_gating:35, cooldown_active:4}`, `families_active_in_window=["observation_coverage"]`. +- Live Plane snapshot on Friday, July 17, 2026: `Backlog=96`, `Blocked=5`, `In Review=2`, `Done=106`, `Cancelled=141`, `Ready for AI=0` (`total=350`). +- Live blocked set is now: + - `5ada8f03-538b-4790-9ebc-1f4772369ac9`, `9a569770-41cb-40ae-87e9-3498fe8f79ea`, `8064fa16-0eec-4d9e-966f-e11011ee30d4`, `1a3b14b1-b496-4c9d-b204-a4511f1c9f14` for `OperationsCenter` + - standing operator/admin branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` +- The formerly blocked `VideoFoundry` task `71873497-9df1-4584-a869-5c8c80dc2936` is no longer blocked; direct Plane fetch shows it moved to `Backlog` at `2026-07-17T00:24:30.917174Z`. + +### STEP 3 / 3.5 — Triage and unblock +- `operations-center-triage-scan --config config/operations_center.local.yaml --apply`: exit `0`; `rescore=[]`, `awaiting=[]`, `queue_healing=[]`. +- `operations-center-board-unblock --config config/operations_center.local.yaml --apply`: exit `0`; `actions=[]`, `cooldown_skip_reason=null`, `mem_available_gb=21.96`. +- `operations-center-detect-convergence-stall --config config/operations_center.local.yaml --json`: exit `0`; `stalls=[]`, `escalated=0`, `suppressed=0`. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed the newest prior cycle summaries in `logs/local/watchdog_cycles/20260717_cycle.md`. +- `NEW_EVIDENCE_DETECTED = yes` this cycle because the live queue changed materially versus the immediately prior cycle at `2026-07-17T00:31:00Z`: `Backlog 93 -> 96`, `Blocked 6 -> 5`, `In Review 4 -> 2`, and the `VideoFoundry` blocked task left `Blocked` for `Backlog`. +- Starvation remains active this cycle: + - proposer quiet diagnosis still shows 5 consecutive zero-create cycles + - cooldown is clear (`cooldown_skip_reason=null`) and both `triage-scan` and `board-unblock` produced zero queue-healing actions + - `Ready for AI` remains `0` even though blocked work is still present on the board +- Behavioral convergence classification remains `NON-CONVERGENT` for the proposer/queue-governance path. +- Existing Plane escalation `b27f9e00-70a8-4f58-a7f2-5d0ee28b8f2d` remains current. Added a fresh comment with this cycle's queue delta and the fact that `71873497-9df1-4584-a869-5c8c80dc2936` moved to `Backlog` rather than opening a duplicate task. +- Because `NEW_EVIDENCE_DETECTED = yes`, the starvation condition is not eligible for `PARKED_OPERATOR_BLOCKED` this cycle. +- The standing branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` remains separately operator-blocked and unchanged. + +### STEP 6 / 7 — Execution gate and direct fix +- No finding passed the direct-fix gate. +- No repo-scoped OperationsCenter source-code defect was reproduced from current-cycle audit output. The live issue remains proposer/queue-governance starvation plus the standing external branch-protection/admin blocker, so no autonomy-cycle dispatch was initiated. + +### STEP 8 — Invariant enforcement +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` -> `15 passed in 4.06s`. +- One slow-test warning only: `tests/unit/er000_phase0_golden/test_golden.py::TestBoundaryNoExampleManagedRepoImports::test_no_example_managed_repo_imports_in_oc_src` at `3.004s`. + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- `grep -h "watcher_restart\|exit_code\|ERROR\|Traceback" logs/local/watch-all/*.log | tail -50` showed: + - repeated review-lane branch-protection refusals for `ProtocolWarden/TeamExecutor#18` and `ProtocolWarden/ContextLifecycle#44` + - benign `exit_code=143` restarts for `propose`, `goal`, `test`, `improve`, and `spec` + - no fresh non-143 watcher crash and no unexpected clean exit +- No restart was warranted. + +### STEP 10 — Commit / push / PR +- No source, config, or test files were changed this cycle; only cycle logging, Plane-comment evidence update, and schedule update. No branch, commit, push, or PR were created. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: on Friday, July 17, 2026 proposer starvation remains active (`quiet_window_cycles=5`, `Ready for AI=0`, cooldown clear, no queue-healing actions), but the live board changed this cycle (`Blocked 6 -> 5`, `In Review 4 -> 2`, `VideoFoundry` blocker moved to `Backlog`), so this is fresh evidence rather than a parkable no-change loop. +- Delay: `600s`. + +## 2026-07-17T01:07:59Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` still contains only the completed `print_structured()` helper work. No active operator work order or unchecked definition-of-done items were present. + +### STEP 1 — Preflight +- Repo sync on Friday, July 17, 2026: all managed repos with local checkouts reported `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running (`intake`, `goal`, `test`, `improve`, `propose`, `review`, `spec`, `watchdog`). +- CLIs present. Working tree clean before cycle-local logging. + +### STEP 2 — Investigate +- `operations-center-custodian-sweep --config config/operations_center.local.yaml --emit`: exit `0`; `repos_swept=19`, `total_violations=0`, `all_zero=true`. +- `operations-center-ghost-audit --config config/operations_center.local.yaml --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10`, status `fixed`. +- `operations-center-flow-audit --config config/operations_center.local.yaml`: exit `0`; `total_open_gaps=1`; open gap remains `F3` (`proposal deduplication`, `status=fixed`, historical sample only) and `F8` remains `partial` with `count=0`. +- `operations-center-graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `operations-center-reaudit-check --json`: exit `0`; neither backend needs re-audit. +- `operations-center-check-regressions --config config/operations_center.local.yaml --lookback-hours 1 --dry-run`: exit `0`; `findings=[]`. +- Executor/OOM investigation: + - board-worker log tail still shows historical backend-capacity/session-limit failures on prior goal tasks, but no current-cycle executor signal-kill signature + - `journalctl -k --since "2h ago" | grep -iE "killed|oom"` returned no lines + - `free -h` showed about `21Gi` available memory with swap unused +- `logs/autonomy_cycle/quiet_diagnosis.json` is fresh on Friday, July 17, 2026 and now reports `quiet_window_cycles=5`, `all_cycles_had_zero_created=true`, `all_cycles_had_zero_candidates=true`, `diagnosis_kind=proposer_quiet`, `aggregated_suppression_reasons={family_deferred_initial_gating:35, cooldown_active:5}`, and `families_active_in_window=[]`. +- Latest autonomy-cycle report (`logs/autonomy_cycle/cycle_20260717T005841Z.json`, generated `2026-07-17T00:58:41Z`) shows `decide.candidates_emitted=0`, `candidates_suppressed=8`, `emitted_families=[]`, and `propose.created=0`. +- Live Plane snapshot on Friday, July 17, 2026: `Backlog=96`, `Blocked=5`, `In Review=2`, `Done=106`, `Cancelled=141`, `Ready for AI=0`. +- Live blocked set remains: + - `5ada8f03-538b-4790-9ebc-1f4772369ac9`, `9a569770-41cb-40ae-87e9-3498fe8f79ea`, `8064fa16-0eec-4d9e-966f-e11011ee30d4`, `1a3b14b1-b496-4c9d-b204-a4511f1c9f14` for `OperationsCenter` + - standing operator/admin branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` + +### STEP 3 / 3.5 — Triage and unblock +- `operations-center-triage-scan --config config/operations_center.local.yaml --apply`: exit `0`; `rescore=[]`, `awaiting=[]`, `queue_healing=[]`. +- `operations-center-board-unblock --config config/operations_center.local.yaml --apply`: exit `0`; `actions=[]`, `cooldown_skip_reason=null`, `mem_available_gb=21.74`. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed the newest prior cycle summaries in `logs/local/watchdog_cycles/20260717_cycle.md`, plus the late Thursday, July 16, 2026 entries in `logs/local/watchdog_cycles/20260716_cycle.md`. +- `NEW_EVIDENCE_DETECTED = yes` this cycle because the proposer telemetry changed materially versus the immediately prior Friday, July 17, 2026 cycle entries: the quiet diagnosis has shifted from `diagnosis_kind=proposer_no_create` with one active family to `diagnosis_kind=proposer_quiet` with `all_cycles_had_zero_candidates=true` and `families_active_in_window=[]`. +- Starvation remains active this cycle: + - proposer quiet diagnosis still spans 5 consecutive zero-create cycles + - the latest autonomy-cycle report emitted zero candidates and created zero tasks + - cooldown is clear in `board-unblock`, but the board still sits at `Ready for AI=0` with blocked work present +- Behavioral convergence classification remains `NON-CONVERGENT` for the proposer/queue-governance path. +- Existing Plane escalation `b27f9e00-70a8-4f58-a7f2-5d0ee28b8f2d` remains current. Added a fresh comment with the new `proposer_quiet` telemetry and the unchanged Friday, July 17, 2026 board counts rather than opening a duplicate task. +- This condition is not eligible for `PARKED_OPERATOR_BLOCKED` in this cycle because the proposer telemetry changed materially, even though queue counts did not. +- The standing branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` remains separately operator-blocked and unchanged. + +### STEP 6 / 7 — Execution gate and direct fix +- No finding passed the direct-fix gate. +- No repo-scoped OperationsCenter source-code defect was reproduced from current-cycle audit output. The live issue remains proposer/queue-governance starvation plus the standing external branch-protection/admin blocker, so no autonomy-cycle dispatch was initiated. + +### STEP 8 — Invariant enforcement +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` -> `15 passed in 2.77s`. +- One slow-test warning only: `tests/unit/er000_phase0_golden/test_golden.py::TestBoundaryNoExampleManagedRepoImports::test_no_example_managed_repo_imports_in_oc_src` at `1.874s`. + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- `grep -h "watcher_restart\|exit_code\|ERROR\|Traceback" logs/local/watch-all/*.log | tail -50` showed: + - repeated review-lane branch-protection refusals for `ProtocolWarden/ContextLifecycle#44`, `#45`, and `ProtocolWarden/TeamExecutor#18` + - benign `exit_code=143` restarts for `propose`, `goal`, `test`, `improve`, and `spec` + - no fresh non-143 watcher crash and no unexpected clean exit +- No restart was warranted. + +### STEP 10 — Commit / push / PR +- No source, config, or test files were changed this cycle; only cycle logging, Plane-comment evidence update, and schedule update. No branch, commit, push, or PR were created. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: on Friday, July 17, 2026 proposer starvation remains active with fresh telemetry (`diagnosis_kind=proposer_quiet`, `all_cycles_had_zero_candidates=true`, `families_active_in_window=[]`) while the board remains at `Blocked=5` and `Ready for AI=0` after maintenance passes. +- Delay: `600s`. + +## 2026-07-17T01:31:58Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` still contains only the completed `print_structured()` helper work. No active operator work order or unchecked definition-of-done items were present. + +### STEP 1 — Preflight +- Repo sync on Friday, July 17, 2026: all managed repos with local checkouts reported `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running (`intake`, `goal`, `test`, `improve`, `propose`, `review`, `spec`, `watchdog`). +- CLIs present. Working tree clean before cycle-local logging. + +### STEP 2 — Investigate +- Initial `operations-center-custodian-sweep --config config/operations_center.local.yaml --emit` session never returned a final summary despite reporting progress through multiple repos; a bounded rerun with `timeout 120` also timed out (`exit 124`) after reaching `17/19` completions. +- The timed-out sweep completed these repos before hanging: `SwitchBoard`, `PlatformDeployment`, `OperatorConsole`, `CxRP`, `TeamExecutor`, `DAGExecutor`, `CritiqueExecutor`, `RxP`, `CoreRunner`, `SourceRegistry`, `Custodian`, `Warehouse`, `RepoGraph`, `SyncMechanism`, `PlatformManifest`, `ProtocolWarden`, `ContextLifecycle`. The two repos that never completed were `OperationsCenter` and `VideoFoundry`. +- Direct bounded probes reproduced the same Custodian-side defect on both repos: + - `timeout 45 .venv/bin/custodian-audit --repo /home/dev/Documents/GitHub/OperationsCenter --json` -> `exit 124` + - `timeout 45 .venv/bin/custodian-audit --repo /home/dev/Documents/GitHub/VideoFoundry --json` -> `exit 124` + - both commands immediately emitted the same detector-id collision warnings, then hung without producing JSON: + - `R1`: builtin `README.md missing at repo root` collides with builtin `.console/*.md source file exceeds its line budget` + - `R2`: builtin `README first H1 does not match repo name` collides with builtin `scrub-target private name in a public repo's tracked .console/** file` +- `operations-center-ghost-audit --config config/operations_center.local.yaml --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10`, status `fixed`. +- `operations-center-flow-audit --config config/operations_center.local.yaml`: exit `0`; `total_open_gaps=1`; `F3 proposal deduplication` remains historically sampled and `F8 back-pressure on Ready queue` remains `partial` with `count=0`. +- `operations-center-graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `operations-center-reaudit-check --json`: exit `0`; neither backend needs re-audit. +- `operations-center-check-regressions --config config/operations_center.local.yaml --lookback-hours 1 --dry-run`: exit `0`; `findings=[]`. +- Executor/OOM investigation on Friday, July 17, 2026: + - board-worker log tail still shows only backend-capacity/session-limit failures (`Stage planner received non-JSON from agent`) on older goal tasks + - no current-cycle executor signal-kill signature + - `journalctl -k --since "2h ago" | grep -iE "killed|oom"` returned no lines + - `free -h` showed about `22Gi` available memory with swap unused +- `logs/autonomy_cycle/quiet_diagnosis.json` on Friday, July 17, 2026 reports `quiet_window_cycles=5`, `all_cycles_had_zero_created=true`, `all_cycles_had_zero_candidates=false`, `diagnosis_kind=proposer_no_create`, `aggregated_suppression_reasons={family_deferred_initial_gating:35, cooldown_active:4}`, and `families_active_in_window=["observation_coverage"]`. +- Latest autonomy-cycle report (`logs/autonomy_cycle/cycle_20260717T012022Z.json`, generated `2026-07-17T01:20:22Z`) shows `decide.candidates_emitted=1`, `candidates_suppressed=7`, `emitted_families=["observation_coverage"]`, but `propose.created=0`. +- Live Plane snapshot on Friday, July 17, 2026: `Backlog=96`, `Blocked=5`, `In Review=2`, `Done=106`, `Cancelled=141`, `Ready for AI=0` (`total=350`). +- Live blocked set remains: + - `5ada8f03-538b-4790-9ebc-1f4772369ac9`, `9a569770-41cb-40ae-87e9-3498fe8f79ea`, `8064fa16-0eec-4d9e-966f-e11011ee30d4`, `1a3b14b1-b496-4c9d-b204-a4511f1c9f14` for `OperationsCenter` + - standing operator/admin branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` + +### STEP 3 / 3.5 — Triage and unblock +- `operations-center-triage-scan --config config/operations_center.local.yaml --apply`: exit `0`; `rescore=[]`, `awaiting=[]`, `queue_healing=[]`. +- `operations-center-board-unblock --config config/operations_center.local.yaml --apply`: exit `0`; `actions=[]`, `cooldown_skip_reason=null`, `mem_available_gb=22.66`. +- `operations-center-detect-convergence-stall --config config/operations_center.local.yaml --json`: exit `0`; `stalls=[]`, `escalated=0`, `suppressed=0`. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed the newest prior cycle summaries in `logs/local/watchdog_cycles/20260717_cycle.md`, plus the late Thursday, July 16, 2026 entries in `logs/local/watchdog_cycles/20260716_cycle.md`. +- `NEW_EVIDENCE_DETECTED = yes` this cycle: + - proposer telemetry changed materially again: the latest autonomy-cycle emitted one `observation_coverage` candidate, but `propose.created` stayed `0` + - `quiet_diagnosis.json` shifted back from the prior `proposer_quiet` shape to `proposer_no_create` with an active family + - a second independent issue reproduced: Custodian audit hangs on both `OperationsCenter` and `VideoFoundry` after duplicate detector-id warnings +- Starvation remains active this cycle: + - proposer telemetry still spans 5 consecutive zero-create cycles + - the newest autonomy-cycle emitted one candidate but still created zero tasks + - cooldown is clear in `board-unblock`, but the board still sits at `Ready for AI=0` with blocked work present +- Behavioral convergence classification remains `NON-CONVERGENT` for the proposer/queue-governance path. +- Existing Plane escalation `b27f9e00-70a8-4f58-a7f2-5d0ee28b8f2d` remains current. Added a fresh comment with Friday, July 17, 2026 telemetry: `candidates_emitted=1`, `propose.created=0`, and unchanged board counts. +- Existing Plane task `07ba5247-d660-4993-8dca-f7f9f7f13732` (`Custodian audit hangs before JSON output across multiple repos`) already covered the reproduced Custodian defect. Added a fresh comment with the Friday, July 17, 2026 reproduction details rather than opening a duplicate task. +- This condition is not eligible for `PARKED_OPERATOR_BLOCKED` in this cycle because proposer telemetry changed materially and new Custodian-audit evidence was added. +- The standing branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435` remains separately operator-blocked and unchanged. + +### STEP 6 / 7 — Execution gate and direct fix +- No finding passed the direct-fix gate. +- No repo-scoped OperationsCenter source-code defect was reproduced from current-cycle OC audit output. +- The live issues remain: + - proposer/queue-governance starvation with fresh non-convergent telemetry + - the standing external branch-protection/admin blocker + - a separately tracked Custodian-side audit hang already covered by Plane task `07ba5247-d660-4993-8dca-f7f9f7f13732` +- No autonomy-cycle dispatch was initiated. + +### STEP 8 — Invariant enforcement +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` -> `15 passed in 3.57s`. +- One slow-test warning only: `tests/unit/er000_phase0_golden/test_golden.py::TestBoundaryNoExampleManagedRepoImports::test_no_example_managed_repo_imports_in_oc_src` at `2.561s`. + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- `grep -h "watcher_restart\|exit_code\|ERROR\|Traceback" logs/local/watch-all/*.log | tail -50` showed: + - repeated review-lane branch-protection refusals for `ProtocolWarden/ContextLifecycle#44`, `#45`, and `ProtocolWarden/TeamExecutor#18` + - benign `exit_code=143` restarts for `propose`, `goal`, `test`, `improve`, and `spec` + - no fresh non-143 watcher crash and no unexpected clean exit +- No restart was warranted. + +### STEP 10 — Commit / push / PR +- No source, config, or test files were changed this cycle; only cycle logging, Plane-comment evidence updates, and schedule update. No branch, commit, push, or PR were created. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: on Friday, July 17, 2026 proposer starvation remains active with fresh telemetry (`cycle_20260717T012022Z.json` emitted one `observation_coverage` candidate but `propose.created=0`), the live board remains frozen at `Blocked=5` and `Ready for AI=0` after triage/unblock passes, and a separately tracked Custodian audit hang reproduced again on `OperationsCenter` and `VideoFoundry`. +- Delay: `600s`. +## 2026-07-17T01:52:10Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` still contains only the completed `print_structured()` helper work. No active operator work order or unchecked definition-of-done items were present. + +### STEP 1 — Preflight +- Repo sync on Friday, July 17, 2026: all managed repos with local checkouts reported `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running (`intake`, `goal`, `test`, `improve`, `propose`, `review`, `spec`, `watchdog`). +- CLIs present. Working tree remained clean before cycle-local logging. + +### STEP 2 — Investigate +- `operations-center-ghost-audit --config config/operations_center.local.yaml --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10`, status `fixed`. +- `operations-center-flow-audit --config config/operations_center.local.yaml`: exit `0`; `total_open_gaps=1`; only historical `F3 proposal deduplication` sample remains and `F8 back-pressure on Ready queue` stays `partial` with `count=0`. +- `operations-center-graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `operations-center-reaudit-check --json`: exit `0`; neither backend needs re-audit. +- `operations-center-check-regressions --config config/operations_center.local.yaml --lookback-hours 1 --dry-run`: exit `0`; `findings=[]`. +- `operations-center-custodian-sweep --config config/operations_center.local.yaml --emit` progressed through all repos (`19/19`, ending with `VideoFoundry` then `OperationsCenter`) but never returned a final summary before the session stalled. A bounded rerun with `timeout 120` also showed normal per-repo progress but did not produce a final result inside the observation window. +- `operations-center-triage-scan --config config/operations_center.local.yaml --apply` and `operations-center-board-unblock --config config/operations_center.local.yaml --apply` similarly failed to return a final payload in this shell-driven cycle, so they were treated as bounded maintenance-path stalls rather than assumed-success operations. +- Executor/OOM investigation on Friday, July 17, 2026: + - `grep -h "board_worker.*blocked\\|exit_code\\|executor" logs/local/watch-all/*.log | tail -40` shows repeated `goal` worker blocks with `category=backend_error` and reason `Stage planner received non-JSON from agent (session limit or error)`. + - `journalctl -k --since "2h ago" | grep -iE "killed|oom"` returned no lines. + - `free -h` showed about `21Gi` available memory and swap unused. + +### STEP 3 / 3.5 — Triage and unblock +- No new authoritative CLI payload was obtained this cycle because both maintenance commands hung when run directly from the watchdog shell. +- Recent watcher log evidence still shows active `board_unblock` runs making queue mutations, including repeated `GOAL_BACKLOG_PROMOTE`, `BACKEND_CAPACITY_PARK`, and `OPEN_PR_GATE_REQUEUE` actions on the same OperationsCenter task families. The board is moving, but it is moving in a churn loop rather than draining cleanly. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed the newest prior cycle summaries in `logs/local/watchdog_cycles/20260717_cycle.md`, plus the latest Thursday, July 16, 2026 entries in `logs/local/watchdog_cycles/20260716_cycle.md`. +- `NEW_EVIDENCE_DETECTED = yes` this cycle: + - the shell-driven maintenance path itself stalled again after completing work items, rather than returning a summary + - current watcher logs show the same queue families being repeatedly promoted to `Ready for AI` and then parked back to `Backlog` under backend-capacity/session-limit conditions +- Classified the platform behavior as `NON-CONVERGENT` for the queue-governance / backend-capacity path: + - retries are changing task states, but not materially reducing the repeated OperationsCenter task set + - the same remediation pattern (`GOAL_BACKLOG_PROMOTE` -> backend/session-limit failure -> `BACKEND_CAPACITY_PARK`) continues to recur + - maintenance commands needed for authoritative queue inspection are not reliably returning inside a bounded watchdog cycle +- The standing review-lane branch-protection blocker for `ProtocolWarden/TeamExecutor#18` remains separately operator-blocked and unchanged on Friday, July 17, 2026. +- No new repo-scoped OperationsCenter source-code defect was isolated from current-cycle tool output. Existing escalation coverage remains sufficient; this cycle added evidence rather than a new remediation path. + +### STEP 6 / 7 — Execution gate and direct fix +- No finding passed the direct-fix gate. +- No autonomy-cycle dispatch was initiated. The active issue is platform-level queue churn plus bounded maintenance-path stalls, not a newly isolated implementation defect in one managed repo. + +### STEP 8 — Invariant enforcement +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` -> `15 passed in 3.01s`. +- One slow-test warning only: `tests/unit/er000_phase0_golden/test_golden.py::TestBoundaryNoExampleManagedRepoImports::test_no_example_managed_repo_imports_in_oc_src` at `2.197s`. + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- `grep -h "watcher_restart\|exit_code\|ERROR\|Traceback" logs/local/watch-all/*.log | tail -50` showed: + - repeated review-lane branch-protection refusals for `ProtocolWarden/TeamExecutor#18` + - benign `exit_code=143` restarts for `propose`, `goal`, `test`, `improve`, and `spec` + - no fresh non-143 watcher crash and no unexpected clean exit +- No restart was warranted. + +### STEP 10 — Commit / push / PR +- No source, config, or test files were changed this cycle; only cycle logging and schedule update. No branch, commit, push, or PR were created. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: on Friday, July 17, 2026 the platform remains in non-convergent backend-capacity queue churn (`GOAL_BACKLOG_PROMOTE` / `BACKEND_CAPACITY_PARK` repeating on the same OperationsCenter work) and the shell-driven maintenance/Plane paths are not reliably returning within a bounded watchdog iteration, even though watcher health and invariant tests remain clean. +- Delay: `600s`. + +## 2026-07-17T02:15:14Z — OC watchdog cycle + +### STEP 0 — Operator directive +- `.console/task.md` still contains only the completed `print_structured()` helper work. No active operator work order or unchecked definition-of-done items were present. + +### STEP 1 — Preflight +- Repo sync on Friday, July 17, 2026: all managed repos with local checkouts reported `Already up to date.` +- Plane: `OK`. SwitchBoard: `OK`. +- `watch-all-status`: all 8 watcher roles running (`intake`, `goal`, `test`, `improve`, `propose`, `review`, `spec`, `watchdog`). +- CLIs present. Working tree clean before cycle-local logging. + +### STEP 2 — Investigate +- `operations-center-custodian-sweep --config config/operations_center.local.yaml --emit`: exit `0`; `repos_swept=19`, `total_violations=0`, `all_zero=true`. +- `operations-center-ghost-audit --config config/operations_center.local.yaml --since 1h`: exit `0`; `total_ghost_events=1`, only historical `G10`, status `fixed`. +- `operations-center-flow-audit --config config/operations_center.local.yaml`: exit `0`; `total_open_gaps=1`; only historical `F3 proposal deduplication` sample remains and `F8 back-pressure on Ready queue` stays `partial` with `count=0`. +- `operations-center-graph-doctor`: exit `0`; graph healthy (`nodes_total=22`, `edges_total=26`). +- `operations-center-reaudit-check --json`: exit `0`; neither backend needs re-audit. +- `operations-center-check-regressions --config config/operations_center.local.yaml --lookback-hours 1 --dry-run`: exit `0`; `findings=[]`. +- Executor/OOM investigation on Friday, July 17, 2026: + - board-worker log tail shows historical `goal` worker backend-capacity failures (`Stage planner received non-JSON from agent`) but no current-cycle signal-kill signature + - `dmesg` access is unavailable in this environment (`Operation not permitted`) + - `journalctl -k --since "2h ago" | grep -iE "killed|oom"` returned no lines + - `free -h` showed about `22Gi` available memory with swap unused +- `logs/autonomy_cycle/quiet_diagnosis.json` remained at `generated_at=2026-07-17T02:04:12.782025+00:00`, still showing `quiet_window_cycles=5`, `diagnosis_kind=proposer_no_create`, and `aggregated_suppression_reasons={family_deferred_initial_gating:35, cooldown_active:4}`. +- Live Plane snapshot before the fix on Friday, July 17, 2026: `Backlog=96`, `Blocked=5`, `In Review=2`, `Done=106`, `Cancelled=141`, `Ready for AI=0` (`total=350`). The blocked set consisted of four `OperationsCenter` `source: spec-campaign` goal tasks with `blocked-reason: backend-capacity` plus the standing operator/admin branch-protection blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435`. +- `operations-center-detect-convergence-stall --config config/operations_center.local.yaml --json`: exit `0`; `stalls=[]`, `escalated=0`, `suppressed=0`. + +### STEP 3 / 3.5 — Triage and unblock +- `operations-center-triage-scan --config config/operations_center.local.yaml --apply`: exit `0`; `rescore=[]`, `awaiting=[]`, `queue_healing=[]`. +- `operations-center-board-unblock --config config/operations_center.local.yaml --apply` before the fix: exit `0`; `actions=[]`, `cooldown_skip_reason=null`, `mem_available_gb=22.17`. + +### STEP 4 / 5 — Blocker and convergence review +- Reviewed the newest prior cycle summaries in `logs/local/watchdog_cycles/20260717_cycle.md` (`2026-07-17T01:07:59Z`, `2026-07-17T01:31:58Z`, `2026-07-17T01:52:10Z`). +- `NEW_EVIDENCE_DETECTED = yes` this cycle because the live board still matched the Friday, July 17, 2026 stalled queue shape (`Blocked=5`, `Ready for AI=0`), but direct code inspection identified a concrete OC source-code recovery gap: `source: spec-campaign` implement-phase goal tasks could enter `Blocked` on `backend-capacity` and had no automatic `Blocked -> Backlog -> Ready for AI` path. +- Classified the pre-fix state as `STALLED` / `NON-CONVERGENT` for the queue-healing path: + - blocked work existed on the board while `Ready for AI=0` + - `triage-scan` and `board-unblock` produced zero queue movement before the patch + - the four blocked `spec-campaign` implement tasks were outside Rule 8.5's existing repromotion criteria, so they were stranded until a human or code change intervened +- This is a promotion candidate fulfilled directly in this cycle: the loop-only judgment from prior cycles was that `backend-capacity` spec-campaign tasks had no self-healing path, and the responsible watcher behavior lives in `board_unblock`. + +### STEP 6 / 7 — Execution gate and direct fix +- A direct fix passed the gate because the finding reproduced in the current cycle, was scoped to `OperationsCenter` from live Plane output, was implementation-level OC source work, and did not require policy widening or destructive cleanup. +- Created watchdog branch `oc-watchdog/20260717-0217-spec-campaign-requeue` before editing. +- Fixed `src/operations_center/entrypoints/maintenance/board_unblock.py` so `source: spec-campaign` implement-phase goal tasks: + - qualify for Rule 8.5 `BACKEND_CAPACITY_PARK` when they stall in `Blocked` with `blocked-reason: backend-capacity` + - qualify for a new Rule `SPEC_CAMPAIGN_BACKLOG_PROMOTE` that restores them from `Backlog` to `Ready for AI` once cooldown allows dispatch again +- Added focused regression coverage in `tests/unit/entrypoints/maintenance/test_board_unblock_cov.py` for both sides of that recovery path. +- Live validation on Friday, July 17, 2026: + - post-fix `operations-center-board-unblock --config config/operations_center.local.yaml --apply`: exit `0`; applied four `BACKEND_CAPACITY_PARK` actions on `5ada8f03-538b-4790-9ebc-1f4772369ac9`, `9a569770-41cb-40ae-87e9-3498fe8f79ea`, `8064fa16-0eec-4d9e-966f-e11011ee30d4`, and `1a3b14b1-b496-4c9d-b204-a4511f1c9f14`, plus two `SPEC_CAMPAIGN_BACKLOG_PROMOTE` actions on `71873497-9df1-4584-a869-5c8c80dc2936` and `9221997b-d505-4b11-a901-29823376018f`. + - post-fix live Plane snapshot: `Backlog=98`, `Ready for AI=2`, `Blocked=1`, `In Review=2`, `Done=106`, `Cancelled=141`. + - the only remaining blocked task is the standing branch-protection/admin blocker `fe58f9fd-38cd-4c9f-9474-28bc49b86435`. +- No repo autonomy-cycle dispatch was needed after the fix; the queue-healing code change itself produced the required remediation. + +### STEP 8 — Invariant enforcement +- `PYTHONPATH=src .venv/bin/pytest tests/unit/entrypoints/maintenance/test_board_unblock_cov.py -q --tb=short` → `93 passed in 1.06s`. +- `.venv/bin/pytest tests/unit/er000_phase0_golden/ -q --tb=short` → `15 passed in 3.01s`. +- One slow-test warning only: `tests/unit/er000_phase0_golden/test_golden.py::TestBoundaryNoExampleManagedRepoImports::test_no_example_managed_repo_imports_in_oc_src` at `1.990s`. + +### STEP 9 — Watcher health +- `watch-all-status`: all 8 watcher roles running. +- `grep -h "watcher_restart\|exit_code\|ERROR\|Traceback" logs/local/watch-all/*.log | tail -50` showed only: + - repeated review-lane branch-protection refusals for `ProtocolWarden/TeamExecutor#18` + - benign `exit_code=143` restarts for `propose`, `goal`, `test`, `improve`, and `spec` + - no fresh non-143 watcher crash and no unexpected clean exit +- No restart was warranted. + +### STEP 10 — Commit / push / PR +- Meaningful OC source/test changes were made this cycle; branch, commit, push, and PR are required. +- Commit, push, and PR metadata are recorded after execution below. + +### STEP 11 — Schedule +- State: `STALLED`. +- Reason: Friday, July 17, 2026 started with a reproduced stalled queue (`Blocked=5`, `Ready for AI=0`) caused by a missing spec-campaign backend-capacity recovery path. The direct fix landed and improved the board to `Blocked=1`, `Ready for AI=2`, but the worst observed state this cycle was still `STALLED`, so a short follow-up remains warranted. +- Delay: `600s`. diff --git a/src/operations_center/entrypoints/maintenance/board_unblock.py b/src/operations_center/entrypoints/maintenance/board_unblock.py index 300f5e1b..f85bf739 100644 --- a/src/operations_center/entrypoints/maintenance/board_unblock.py +++ b/src/operations_center/entrypoints/maintenance/board_unblock.py @@ -96,7 +96,7 @@ quota condition and returns no structured output; leaving them in Blocked strands work even after the cooldown clears because the recovery path only promotes from Backlog. Parking them preserves the retry blocker label until Rule 7 / Rule 9 - re-promotes them once dispatch is legal again. + / Rule 9.5 re-promote them once dispatch is legal again. Rule 9 — SPEC_AUTHOR_BACKLOG_PROMOTE spec-author tasks in Backlog state with no active retry blocker → move to Ready for AI. @@ -105,6 +105,13 @@ Skipped when memory is below the executor dispatch threshold or when any active retry blocker is present (budget_exhausted, session_limit, global_rate_exceeded, etc.). + Rule 9.5 — SPEC_CAMPAIGN_BACKLOG_PROMOTE + spec-campaign implement-phase goal tasks in Backlog state with no active retry blocker + → move to Ready for AI. Backend-capacity failures on implement-phase campaign tasks + otherwise strand the campaign forever: they start life in Ready for AI, but unlike + autonomy/improvement tasks they have no separate watcher that re-promotes them after + a Blocked → Backlog recovery. + Rule 10 — OPEN_PR_GATE_REQUEUE Goal tasks in Backlog carrying OPEN_PR_GATE whose repo no longer has blocking non-spec PRs → move back to Ready for AI and remove OPEN_PR_GATE. This @@ -277,6 +284,7 @@ def _dispatch_cooldown_reason( _SOURCE_AUTONOMY_LABEL = "source: autonomy" _SOURCE_IMPROVE_SUGGESTION_LABEL = "source: improve-suggestion" _SOURCE_BOARD_WORKER_LABEL = "source: board_worker" +_SOURCE_SPEC_CAMPAIGN_LABEL = "source: spec-campaign" _HANDOFF_IMPROVEMENT_LABEL = "handoff-reason: improvement_applied" _PR_URL_PREFIX = "pr-url:" _BLOCKED_REASON_POLICY_LABEL = "blocked-reason: policy" @@ -297,6 +305,23 @@ def _labels(issue: dict[str, Any]) -> list[str]: return names +def _description_blob(issue: dict[str, Any]) -> str: + parts = [ + str(issue.get("description") or ""), + str(issue.get("description_stripped") or ""), + str(issue.get("description_html") or ""), + ] + return "\n".join(part for part in parts if part).lower() + + +def _is_spec_campaign_implement_goal(issue: dict[str, Any], labels: list[str]) -> bool: + return ( + _has_label(labels, _GOAL_LABEL) + and _has_label(labels, _SOURCE_SPEC_CAMPAIGN_LABEL) + and "task_phase: implement" in _description_blob(issue) + ) + + def _label_value(labels: list[str], prefix: str) -> str | None: for label in labels: if label.lower().startswith(prefix.lower()): @@ -879,7 +904,7 @@ def _apply_rules( ) ) and bool(_label_value(labels, _ORIGINAL_TASK_PREFIX)) - ) + ) or _is_spec_campaign_implement_goal(issue, labels) if ( has_repromotion_path and updated_at @@ -933,6 +958,41 @@ def _apply_rules( } ) + # Rule 9.5 — spec-campaign implement backlog promotion. + if ( + state_lower == "backlog" + and _is_spec_campaign_implement_goal(issue, labels) + and mem_available_gb >= _MEM_R4AI_THRESHOLD_GB + ): + if cooldown_skip_reason: + actions.append( + { + "task_id": task_id, + "title": title, + "rule": "SPEC_CAMPAIGN_BACKLOG_PROMOTE", + "from_state": state, + "to_state": "Ready for AI", + "reason": f"SKIPPED — {cooldown_skip_reason}", + "skipped": True, + } + ) + else: + actions.append( + { + "task_id": task_id, + "title": title, + "rule": "SPEC_CAMPAIGN_BACKLOG_PROMOTE", + "from_state": state, + "to_state": "Ready for AI", + "reason": ( + "spec-campaign implement task parked in Backlog after " + "backend-capacity failure; restoring to Ready for AI" + ), + "labels_to_remove": _BLOCKED_REASON_LABELS, + "_issue_labels": labels, + } + ) + return actions diff --git a/tests/unit/entrypoints/maintenance/test_board_unblock_cov.py b/tests/unit/entrypoints/maintenance/test_board_unblock_cov.py index 5d0f7313..8a1dd741 100644 --- a/tests/unit/entrypoints/maintenance/test_board_unblock_cov.py +++ b/tests/unit/entrypoints/maintenance/test_board_unblock_cov.py @@ -23,6 +23,7 @@ def _issue( labels: list[str] | None = None, name: str | None = None, updated_at: str | None = _STALE, + description_html: str | None = None, ) -> dict: issue: dict = { "id": task_id, @@ -32,6 +33,8 @@ def _issue( } if updated_at is not None: issue["updated_at"] = updated_at + if description_html is not None: + issue["description_html"] = description_html return issue @@ -762,6 +765,22 @@ def test_rule8_backend_capacity_blocked_excluded(): assert not _by_rule(actions, "CLEAN_BLOCKED_RETRY") +def test_rule8_5_spec_campaign_backend_capacity_park(): + actions = _run( + [ + _issue( + "1", + state="Blocked", + labels=["task-kind: goal", "source: spec-campaign", "blocked-reason: backend-capacity"], + updated_at="2026-05-28T11:50:00+00:00", + description_html="
task_phase: implement
", + ) + ] + ) + a = _by_rule(actions, "BACKEND_CAPACITY_PARK") + assert a and a[0]["to_state"] == "Backlog" + + # --------------------------------------------------------------------------- # Rule 9 — SPEC_AUTHOR_BACKLOG_PROMOTE # --------------------------------------------------------------------------- @@ -791,6 +810,38 @@ def test_rule9_low_mem_skipped(): assert not _by_rule(actions, "SPEC_AUTHOR_BACKLOG_PROMOTE") +def test_rule9_5_spec_campaign_promote(): + actions = _run( + [ + _issue( + "1", + state="Backlog", + labels=["task-kind: goal", "source: spec-campaign"], + description_html="
task_phase: implement
", + ) + ] + ) + a = _by_rule(actions, "SPEC_CAMPAIGN_BACKLOG_PROMOTE") + assert a and a[0]["to_state"] == "Ready for AI" + assert "skipped" not in a[0] + + +def test_rule9_5_spec_campaign_cooldown_skipped(): + actions = _run( + [ + _issue( + "1", + state="Backlog", + labels=["task-kind: goal", "source: spec-campaign"], + description_html="
task_phase: implement
", + ) + ], + cooldown_skip_reason="cooling", + ) + a = _by_rule(actions, "SPEC_CAMPAIGN_BACKLOG_PROMOTE") + assert a and a[0]["skipped"] is True + + def test_no_actions_for_unrelated_issue(): actions = _run([_issue("1", state="Done")]) assert actions == [] diff --git a/tools/loop/state/schedule.json b/tools/loop/state/schedule.json new file mode 100644 index 00000000..0c91500a --- /dev/null +++ b/tools/loop/state/schedule.json @@ -0,0 +1 @@ +{"delay_s": 600, "state": "STALLED", "reason": "Friday, July 17, 2026 began with a reproduced stalled queue (Blocked=5, Ready for AI=0) caused by a missing spec-campaign backend-capacity recovery path; the fix improved the board to Blocked=1 and Ready for AI=2, but the worst observed state this cycle remained STALLED."} \ No newline at end of file