feat(migrate): 平台形态的迁移门禁 —— 带自检的数据迁移 + 部署级标记 (#3617)

.changeset/adr-0104-deployment-migration-gate.md

@@ -0,0 +1,58 @@
+---
+"@objectstack/spec": minor
+"@objectstack/platform-objects": minor
+"@objectstack/service-storage": minor
+"@objectstack/objectql": minor
+"@objectstack/cli": minor
+---
+
+feat(migrate): `os migrate files-to-references` — a data migration with a self-check, gated per deployment (#3617)
+
+The ADR-0104 file-as-reference migration ships as a command a deployment runs
+against its own database, and the deployment-level flag it records is what may
+later authorise irreversible behaviour — never the platform version.
+
+```bash
+os migrate files-to-references           # dry run: reports, writes nothing
+os migrate files-to-references --apply   # converts, verifies, records the flag
+```
+
+The run backfills legacy file-field values (inline metadata blobs, own-resolver
+URLs, `data:` URIs) into owned `sys_file` references, reconciles the ownership
+ledger against what records actually hold, and — only on an `--apply` run whose
+reconciliation reports **zero blocking discrepancies** — records
+`sys_migration { id: 'adr-0104-file-references', verified_at, blocking: 0 }`.
+
+**Why a flag rather than a release note.** ObjectStack is a development
+platform: third-party deployments upgrade on their own schedule and their data
+is not observable by anyone else, so no release-side soak can vouch for them.
+The evidence has to be produced where the data is. Consequences:
+
+- Installing a new version never starts deleting bytes. Running the migration
+  and passing its self-check is the consent.
+- Not run, or not passed → files are retained forever. Wasted storage, zero
+  data loss.
+- A later failing run **clears** `verified_at`: a deployment whose data has
+  drifted closes its own gate.
+- A dry run writes nothing at all — not the conversions, and not the flag,
+  even when the self-check would pass.
+- External URLs stay advisory. They are not `sys_file`s, so they can never
+  enter collection; whether to remodel them as a `url` field is the app
+  author's decision (ADR-0104 R7), not a gate.
+
+Ships alongside:
+
+- `@objectstack/spec` — `DataMigrationFlagSchema`, `FILE_REFERENCES_MIGRATION_ID`,
+  and the single `isDataMigrationFlagVerified` predicate both future consumers
+  (collection #3459, strict value-shape #3438) read, so the two gates cannot
+  disagree about the same fact.
+- `@objectstack/platform-objects` — the `sys_migration` object plus
+  `readDataMigrationFlag` / `isDataMigrationVerified` / `recordDataMigrationRun`.
+  Reads fail toward "not verified": a gate that cannot read its evidence stays
+  closed.
+- `@objectstack/objectql` — a read may now opt out of file-reference expansion
+  via the spec's `RAW_FILE_VALUES_CONTEXT_KEY`, and the storage service's
+  bookkeeping/scan reads do. Without it the read resolver rewrites stored ids to
+  their expanded form before the reconciliation sees them, which reports held
+  references as absent — noisy `stale_owner` findings, and a missed
+  `unowned_reference` would have been a false pass of the collection gate.

content/docs/deployment/cli.mdx

@@ -517,6 +517,48 @@ first. It never drops a table that is absent from your metadata, and on SQLite
 it reconciles via a table rebuild (copy → swap) that preserves your data.
 </Callout>
 
+#### Data migrations
+
+The commands above reconcile **schema**. A *data* migration rewrites rows, and
+whether it is done is a fact about **your** database, not about the platform
+version you installed — so each deployment runs it, and its result is recorded
+where the data lives.
+
+| Command | Description |
+|---------|-------------|
+| `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
+
+```bash
+os migrate files-to-references            # Dry run: full report, writes nothing
+os migrate files-to-references --apply    # Convert, verify, record the flag (prompts)
+os migrate files-to-references --apply --yes --json   # CI / scripts
+os migrate files-to-references --object product       # Restrict to one object (repeatable)
+```
+
+A `file` / `image` / `avatar` / `video` / `audio` field value is an opaque
+`sys_file` id that the platform owns. Values written before that (an inline
+`{url, name, …}` blob, or a URL naming this platform's own
+`…/storage/files/:id` resolver) are converted in place; **external URLs are
+reported, never re-hosted** — re-hosting third-party content is a licensing and
+privacy decision, and the right fix is usually to model the field as a `url`
+field instead.
+
+The run then reconciles what records actually hold against what `sys_file`
+records as each file's owner. Zero blocking discrepancies is what records the
+flag — and that flag, not the version number, is what later enables features
+that act irreversibly on migrated data (reclaiming the bytes of released files;
+rejecting rather than warning about legacy media values). Never running it is
+safe: files are simply retained forever.
+
+Exit status is `0` only when the self-check passes, so CI can gate on it.
+
+<Callout type="warn">
+A dry run writes **nothing** — not the conversions, and not the flag either,
+even when the self-check would pass. `--apply` is the only writing mode. A
+later run that *fails* its self-check clears the flag's verified state, so a
+database that has drifted closes its own gate.
+</Callout>
+
 ### Scaffolding
 
 | Command | Alias | Description |

content/docs/references/system/migration.mdx

@@ -5,15 +5,37 @@ description: Migration protocol schemas
 
 {/* ⚠️  AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */}
 
+Migration protocol — the two kinds of migration, kept apart on purpose.
+
+A **schema migration** (`ChangeSet` + its atomic operations) reshapes the
+
+physical database to match metadata: add a field, change a type, create an
+
+object, run SQL. It is derivable from the metadata and applied by
+
+`os migrate plan` / `os migrate apply`.
+
+A **data migration** rewrites the rows themselves, and whether it is done is
+
+a property of ONE DEPLOYMENT's database rather than of the installed code
+
+version — so it cannot be expressed as a ChangeSet, and its completion cannot
+
+be inferred from a release. `DataMigrationFlag` is the per-deployment record
+
+that one ran here and its self-check passed; consumers that would act
+
+irreversibly on migrated data gate on the flag instead of the version.
+
 <Callout type="info">
 **Source:** `packages/spec/src/system/migration.zod.ts`
 </Callout>
 
 ## TypeScript Usage
 
 ```typescript
-import { AddFieldOperation, ChangeSet, CreateObjectOperation, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
-import type { AddFieldOperation, ChangeSet, CreateObjectOperation, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
+import { AddFieldOperation, ChangeSet, CreateObjectOperation, DataMigrationFlag, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
+import type { AddFieldOperation, ChangeSet, CreateObjectOperation, DataMigrationFlag, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependency, MigrationOperation, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
 
 // Validate data
 const result = AddFieldOperation.parse(data);
@@ -69,6 +91,25 @@ Create a new object
 | **object** | `{ name: string; label?: string; pluralLabel?: string; description?: string; … }` | ✅ | Full object definition to create |
 
 
+---
+
+## DataMigrationFlag
+
+Deployment-level record that a data migration ran here and its self-check passed — the evidence gate consumers read instead of the platform version
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **id** | `string` | ✅ | Migration id (e.g. adr-0104-file-references) — one row per data migration |
+| **last_run_at** | `string` | ✅ | When this migration last completed a gated (apply-mode) run on this deployment |
+| **verified_at** | `string \| null` | optional | When the self-check last PASSED. Null/absent until it does — and cleared again by a later failing run, so a regression closes the gate |
+| **applied_at** | `string \| null` | optional | When the backfill last ran in apply mode (writes enabled) |
+| **blocking** | `integer` | ✅ | Blocking discrepancies reported by the last self-check. The gate requires 0 |
+| **advisory** | `integer` | optional | Advisory findings from the last run (external URLs, stale owners, …) — cost storage or need a modelling decision, never block the gate |
+| **details** | `string` | optional | JSON-encoded counts from the last run, for diagnostics |
+
+
 ---
 
 ## DeleteObjectOperation

docs/adr/0104-field-runtime-value-shape-contract.md

@@ -591,10 +591,13 @@ legacy values would reject any update that rewrites such a field, breaking
 working apps until the backfill catches up. Backfilling first leaves the cutover
 nothing legacy to reject.
 
-R4's acceptance gate is now executable rather than aspirational: step 6 may not
-merge until step 5 reports **zero blocking discrepancies for ≥7 consecutive
-days** on real tenant data. R5 (public-posture inventory) and R6 (sub-key read
-scan) stand.
+R4's acceptance gate is now executable rather than aspirational: step 5's
+reconciliation must report **zero blocking discrepancies** before step 6 may
+act. (The original wording — "for ≥7 consecutive days on real tenant data" —
+assumed a tenant we operate and observe; see the 2026-07-27 platform-form
+addendum below, which relocates the same evidence requirement to the one place
+that can satisfy it.) R5 (public-posture inventory) and R6 (sub-key read scan)
+stand.
 
 Steps 4 and 6 are the two that can break something. Step 4 is a declared
 protocol major and breaks *loudly* — a rejected write, recoverable. Step 6
@@ -604,15 +607,86 @@ adoption that consumes the new form, while step 6 additionally requires the
 reconciliation evidence above, and neither the enable nor the migration run
 should be performed without an explicit human decision.
 
+## Addendum (2026-07-27, later) — the evidence gate is per deployment, not per release
+
+The gate above was written as an operations runbook: *run the backfill on the
+tenant, watch `verifyFileReferences` stay clean for a week, then flip the
+switch.* Every clause of that sentence assumes **we** run the tenant, **we**
+observe it, and **we** flip.
+
+ObjectStack is a development platform. Third-party developers build metadata
+apps on it and ship versions onward; each deployment decides for itself when to
+upgrade, and its data is not visible to us — nor should it be. **There is no
+observation window of ours that anyone else's data can wait inside.** A gate
+phrased as one is not strict, merely unlocatable: it can be satisfied by no
+deployment at all, including the deployments whose bytes are at stake.
+
+### What survives, and what moves
+
+R4's substance is unchanged: **a ledger may not be given the power to delete
+bytes until it has been shown to agree with reality.** What changes is *who
+produces the evidence* — from us, once, to each deployment, for itself.
+
+That rules out a one-shot script (a script whose output nobody must read is not
+a gate) and rules in a **data-migration step with a self-check, whose passing
+result is recorded on the deployment**:
+
+```
+os migrate files-to-references
+  1. backfill                 (dry run by default; --apply writes)
+  2. verifyFileReferences     (reconcile the ledger against what records hold)
+  3. zero blocking findings →  record sys_migration { id: 'adr-0104-file-references',
+                                 verified_at, blocking: 0 }
+  4. collection + strict enforcement read THAT ROW, never the version number
+```
+
+### The properties this buys
+
+- **Upgrading does not start deleting.** Installing a new version is not
+  consent; running the migration and passing its self-check is.
+- **A third-party developer need not understand R4.** They run one command.
+  Green means done; red names the record holding a file nobody owns.
+- **Not run, or not passed → files live forever.** Wasted storage, zero data
+  loss: "fail toward retention" holds on the *deployment* axis too. The same
+  rule applies to a regression — a later failing run clears `verified_at`, so a
+  deployment whose data has drifted closes its own gate.
+- **Each deployment gets its own 30-day tombstone window.** Nobody waits inside
+  anyone else's soak. The irreversible moment is day 30 after *that* deployment
+  enabled collection, not release day.
+- **A dry run changes nothing** — not the data, and not the flag either, even
+  when the self-check would pass. `--apply` is the only writing mode, so
+  "did this run change my posture" never depends on what the run found.
+
+### D1/D2 carry the identical error, and the identical fix
+
+The strict-by-default flip for value-shape and action-param validation (#3438)
+was scheduled as "flip in a later minor once telemetry is quiet" — again
+*our* telemetry, deciding for *their* deployments. A deployment that has not
+finished backfilling would have every media-field update rejected the moment it
+upgraded: a working app broken by a decision made on its behalf.
+
+Both therefore read the **same** flag. Strict enforcement becomes effective for
+a deployment when that deployment has completed and verified its migration —
+not when a version number arrives. One flag, not two gates that can disagree:
+they are gating on the same fact.
+
+### What cannot be automated, and does not block
+
+Whether an **external URL** (`https://cdn…`) should become an explicit `url`
+field is a modelling decision only the app's author can make (R7). It also
+cannot block: an external URL is not a `sys_file` and can never enter
+collection. It is reported as advisory, never as a gate failure.
+
 ### Why this stays inside ADR-0104 rather than a new ADR
 
 D3 is already this ADR's third phase; the two-wave split and the
 enforcement-point principle are a **refinement of the D4 rollout**, not a new
 decision, so they live here. Wave 2's migration mechanics (dual-read window,
 `os migrate` backfill, the R4/R5/R6 gates) are specified in §D3 above and need
-no separate record. Wave 2's implementation did surface one genuinely new
-decision — the exclusive-ownership model — and it is recorded as the 2026-07-27
-addendum above rather than a separate ADR, because it settles *how* D3's
-already-accepted "field values point into `sys_file`" behaves rather than
-revisiting whether to do it. A future choice that stands on its own (say, a
-chunked-migration protocol, or byte-layer content dedup) would earn its own ADR.
+no separate record. Wave 2's implementation did surface two genuinely new
+decisions — the exclusive-ownership model, and relocating the evidence gate to
+the deployment — and both are recorded as 2026-07-27 addenda above rather than
+separate ADRs, because each settles *how* D3's already-accepted "field values
+point into `sys_file`" reaches production rather than revisiting whether to do
+it. A future choice that stands on its own (say, a chunked-migration protocol,
+or byte-layer content dedup) would earn its own ADR.