You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(liveness): re-verify the last ten preview-only live claims — 8 of 10 were wrong (#3686) (#3711)
Closes the sweep opened in #3685. Final tally across all 13 properties the
2026-06 pass marked `live` on preview-renderer evidence: 3 stand, 10 were
wrong — a 77% error rate for that standard.
Corrected to dead + authorWarn (with actionable hints):
- action.shortcut / action.bulkEnabled — registered into ActionEngine, but
getShortcuts()/handleShortcut()/getBulkActions()/executeBulk() have no
non-test caller and no keydown listener feeds them. Bulk toolbars are
driven by the LIST VIEW's bulkActions/bulkActionDefs instead.
- flow.active — deprecated no-op; `status` is what gates binding/execution.
Sharper than inert: the spec default is false while an unset flow runs, so
the key reads as if it disabled the flow.
- skill.triggerPhrases — a runtime path reads it, but only to hand it back to
API clients. Phrases are never matched against user messages; activation is
triggerConditions + the agent's skills[] + explicit /skill-name pinning.
- tool.category / active / builtIn — not part of AIToolDefinition; the tool
set reaches the LLM as name/description/parameters only. Contrast
agent.active and skill.active, which ARE enforced.
- tool.requiresConfirmation — SAFETY-shaped and unenforced on every path
(LLM tool set, ToolRegistry.execute, the REST execute route, the MCP
bridge). Every real requiresConfirmation read is action.ai.* — a different,
live property. ADR-0033 already resolved to delete this placeholder.
Kept live, evidence corrected to the real reader:
- action.execute — ActionRunner.ts:704 + the spec's execute->target transform.
Records a live divergence: the transform prefers `target`, ActionRunner
prefers `execute`, so an action declaring both runs different code on the
two sides.
- flow.status — engine.ts:1374-1382 gates binding AND execution since
497bda8. The file-level _note claiming "status/active gate nothing" was
true when written and became false a month later; rewritten.
README: count tables synced (action 33/1/2, flow 26/-/5, skill 8/-/2,
tool 5/1/5) and the methodology section extended with the final tally, the
two failure directions (over- AND under-stating liveness), and the three
search traps that produced false negatives during this work — including that
macOS `git grep -E` silently ignores `\b`.
check:liveness green.
Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
\*`skill.permissions` was subsequently pruned outright — it was never enforced.
73
+
74
+
Note the two failure directions the sweep exposed. Most entries **overstated**
75
+
liveness. But `flow.status` was *understated*: the file-level note still said
76
+
"status/active gate nothing", true when written and falsified a month later by
77
+
`497bda853`. **A ledger entry is a claim with a timestamp; code moves under it
78
+
in both directions.**
79
+
80
+
When in doubt, the honest status is `dead` + `authorWarn`: an author who gets a
81
+
warning for a property that turns out to work loses nothing; an author who gets
82
+
silence for a property that does nothing ships a bug.
83
+
84
+
### How to verify a claim without fooling yourself
85
+
86
+
Three false conclusions were published during this sweep, all from the same
87
+
mistake: **a strong negative claim ("nothing reads X") resting on a search whose
88
+
result set was silently truncated or filtered.** Concretely:
89
+
90
+
1.`… | head -3` hid the real hit further down the list.
91
+
2. A pathspec glob `packages/*/src` never matched the nested
92
+
`packages/app-shell/src/layout/…`.
93
+
3.**On macOS, `git grep -E` silently does not honour `\b`** — `git grep -cE
94
+
"\.active\b" flow.zod.ts` returns *nothing* on a file that provably contains
95
+
three `.active` occurrences (`git grep -cw active` finds them). Any absence
96
+
conclusion drawn from a `\b` pattern on this platform is a false negative.
97
+
98
+
So: a grep can only prove **presence**. To prove absence, either close the call
99
+
graph by hand (declaration → registration → accessor → *caller*, which is how
100
+
`action.shortcut` and `tool.active` were settled) or — cheapest and most
101
+
decisive for this ledger — **author the property, boot the app and look**. That
102
+
is how the `app.badge`/`separator` question was finally settled after two wrong
103
+
grep-based rounds.
71
104
72
105
## Runtime proofs — prove-it-runs (ADR-0054)
73
106
@@ -225,14 +258,14 @@ EOF
225
258
|---|---|---|---|---|---|
226
259
| object | 40 | – | 0 | 1 | aspirational tier (versioning/softDelete/search/recordName/keyPrefix) + tags/active/abstract REMOVED (#2377) — tombstoned in UNKNOWN_KEY_GUIDANCE; `enable.trash`/`mru` REMOVED (#2377 close-out) — tombstoned in the now-`.strict()` ObjectCapabilities; `isSystem` + `enable.searchable` CORRECTED to live (#2377 — sharing default-model + global-search opt-out; 2026-06 audit missed both readers); `tenancy.strategy`/`crossTenantAccess` REMOVED post-15.0 (#2763) |
227
260
| field | 55 | – | 0 | – | healthy — full dead set (vectorConfig/fileAttachmentConfig/dependencies, then referenceFilters/columnName/index) REMOVED (#2377); columnName also dropped the ADR-0062 D7 lint + StorageNameMapping column helpers |
| action |35| 1 |0| – |`disabled` went LIVE via metadata-admin authoring UI (2026-06 audit missed objectui); `type:'form'` CORRECTED to live (objectui ActionRunner.executeForm, #2377); dead `timeout` REMOVED (#2377) |
261
+
| flow |26| – |5| – | dead = description/template/nodes.outputSchema/errorHandling.fallbackNodeId (engine uses fault edges) + `active` CORRECTED to dead 2026-07 (deprecated no-op — `status` is what gates binding/execution since 497bda853; the file `_note` claiming otherwise is fixed) |
262
+
| action |33| 1 |2| – |`type:'form'` CORRECTED to live (objectui ActionRunner.executeForm, #2377); dead `timeout` REMOVED (#2377); `disabled` live for real since objectui#2863 (six surfaces); `shortcut` + `bulkEnabled` CORRECTED to dead 2026-07 — registered into ActionEngine but their accessors have no non-test caller (#3686 sweep) |
230
263
| hook | 11 | – | 2 | – | model-healthy; only label/description dead (benign) |
231
264
| permission | 32 | – | 0 | – | CRUD/FLS/RLS live; dead `contextVariables` REMOVED (ADR-0105 D11 — RLS resolves only the `current_user.*` built-ins plus runtime-staged `rlsMembership` sets) |
232
265
| position | 4 | – | – | – | (role's ADR-0090 successor) fully live |
233
266
| agent | 13 | 5 | 1 | – | dead `tenantId` + `planning.strategy`/`allowReplan` REMOVED (#2377) — only `planning.maxIterations` live; autonomy tier experimental; `knowledge` CORRECTED to dead 2026-07 — `search_knowledge` takes `sourceIds` from the LLM's tool-call args, never from the agent record (#1878 §3 recheck) |
234
-
| tool |9| 1 |1| – |`permissions` dead — tool invocation not permission-gated by it |
235
-
| skill |9| – |–| – |`permissions` REMOVED 2026-07 — never permission-gated anything (the cloud SkillRegistry reads only `active`/`triggerConditions`/`tools`); owner call was prune, not enforce. Gate at the agent (`access`/`permissions`, #1884) or on the tools' underlying actions|
267
+
| tool |5| 1 |5| – |the whole authoring surface is inert: `permissions`(not permission-gated), plus `category`/`requiresConfirmation`/`active`/`builtIn` CORRECTED to dead 2026-07 (#3686 sweep). ⚠️ `requiresConfirmation` is SAFETY-shaped and unenforced on every path — ADR-0033 already resolved to delete it |
268
+
| skill |8| – |1| – |`permissions` REMOVED 2026-07 (never gated anything — owner call was prune, #3704); `triggerPhrases` CORRECTED to dead — phrases are never matched against user messages; activation is `triggerConditions` + the agent's `skills[]` + explicit /skill-name pinning (#3686 sweep)|
| page | 16 | – | – | 1 | fully live + one planned |
238
271
| view | 70 | 0 | 5 | – | list/form drilled via `children` (#2998 Track B); dead = list.{responsive,performance} + form.{data,defaultSort,aria}, all but aria authorWarn'd; form.{buttons,defaults} now live — objectui ObjectForm folds them onto its flat props (framework#1894 / #2998); audit-era DEAD lines superseded by re-verification (submitBehavior, sharing.lockedBy, list ViewData providers, and the ADR-0021 chart shape — all live now); level-2 dead residue (userActions.buttons, addRecord.mode/formView, tabs[].order) noted on parents — one drill level only |
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Verdict stands but for a different reason: TWO independent runtime readers exist. NOTE a live divergence — the spec transform keeps `target` when both are set (action.test.ts:1032), while ActionRunner does `execute || target`, so an action declaring both runs different code client- vs server-side."
49
49
},
50
50
"params": {
51
51
"status": "live",
@@ -105,14 +105,18 @@
105
105
"note": "EVIDENCE CORRECTED 2026-07 (#1878 §3 recheck): previously cited the metadata-admin PREVIEW renderer, which only echoes the authored value. At that time the predicate was actually enforced on ONE of six surfaces (#1885 wired action:button alone) — the 'live' verdict was right for the wrong reason and hid a real silent no-op on the other five. objectui#2863 wired the rest; showcase specimen `showcase_archive_task` (#3643) dogfoods it."
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
108
+
"status": "dead",
109
+
"evidence": "registered into ActionEngine.shortcuts[] (objectui packages/core/src/actions/ActionEngine.ts:150) but getShortcuts()/handleShortcut() have no non-test caller and no keydown listener feeds them",
110
+
"authorWarn": true,
111
+
"authorHint": "Not wired — nothing dispatches keydown events to actions. objectui's keyboard stack (useKeyboardShortcuts / KeyboardProtocol) is hand-registered and never touches ActionEngine; register the key there and have its handler invoke the action by name.",
112
+
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Also declared in action.form.ts:69, so pruning would need a form change + a 4-locale i18n regen."
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
115
+
"status": "dead",
116
+
"evidence": "registered into ActionEngine (objectui ActionEngine.ts:151) but getBulkActions()/executeBulk() have no non-test caller; the multi-select toolbar is driven by the LIST VIEW's bulkActions/bulkActionDefs (plugin-grid ObjectGrid.tsx:1602-1612)",
117
+
"authorWarn": true,
118
+
"authorHint": "Not wired — the multi-select toolbar reads the list view's `bulkActions` / `bulkActionDefs`, not this flag. Declare the action on the view instead of setting bulkEnabled here.",
119
+
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Distinct from the live bulk fast-path (core/src/actions/bulkFastPath.ts executeBulkBatch), which never consults this flag. Also in action.form.ts:79."
Copy file name to clipboardExpand all lines: packages/spec/liveness/flow.json
+8-6Lines changed: 8 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
{
2
2
"type": "flow",
3
-
"_note": "FlowSchema. Seeded from docs/audits/2026-06-flowschema-property-liveness.md. Consumers: service-automation engine + node executors. runAs is marker-driven experimental (spec). status/active gate nothing (engine uses an in-memory flowEnabled map via toggleFlow).",
3
+
"_note": "FlowSchema. Seeded from docs/audits/2026-06-flowschema-property-liveness.md. Consumers: service-automation engine + node executors. runAs is marker-driven experimental (spec). CORRECTED 2026-07: the previous note claimed 'status/active gate nothing' — that was true when the audit ran but became FALSE with commit 497bda853 (2026-07-05, 'honor flow status for enable/disable'). `status` now gates binding AND execution; `active` remains a deprecated no-op.",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Verdict stands, evidence replaced: it is genuinely engine-consumed since 497bda853. The previously cited FlowPreview line number did not even contain a status read."
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
35
+
"status": "dead",
36
+
"evidence": "spec already marks it '(Deprecated: use status)'; the only reader anywhere is a display fallback in objectui FlowPreview.tsx:157 (`d.status ?? (d.active ? ...)`)",
37
+
"authorWarn": true,
38
+
"authorHint": "Deprecated no-op — writing `active: false` does NOT stop a flow. Use `status: 'obsolete'` (or 'invalid') to unbind and disable it, `status: 'active'` to arm it.",
39
+
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Sharper than a plain no-op: the spec default is `false` while the engine treats an unset flow as enabled, so the key reads as if it disabled the flow when it does nothing."
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
45
+
"status": "dead",
46
+
"evidence": "cloud service-ai/src/skill-registry.ts:266 copies it into toSummary() and assistant-routes.ts:98,129 serve it — but nothing consumes the field; skill activation is triggerConditions + the agent's skills[] allowlist + explicit /skill-name pinning (skill-registry.ts:128-175, assistant-routes.ts:189-196)",
47
+
"authorWarn": true,
48
+
"authorHint": "Phrases are never matched against the user's message. Routing is `triggerConditions` (AND of context field/operator/value) intersected with the agent's `skills[]`, plus explicit /skill-name pinning; put routing intent in triggerConditions, and describe intent in description/instructions (the LLM does see those).",
49
+
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. A runtime path does read it, but only to hand it back to API clients — a dead-end projection, not consumption."
0 commit comments