feat(storage): governed download for field-owned files + R4 reconciliation gate — ADR-0104 D3 wave 2 (PR-4) (#3534)

* feat(storage): governed download for field-owned files — ADR-0104 D3 wave 2 (PR-4)

A file owned by a record's field (sys_file.ref_object / ref_id, set by PR-3) is
now authorized on download the same way an attachment is: the caller must be
able to READ the file's parent record, or be its uploader. Previously only
attachments-scope files were gated and every field file kept an anonymous
capability URL.

Parent resolution differs by surface, and that asymmetry is the point of the
ownership model. An attachment may hang off many records, so its readable-by
set is the union over its sys_attachment join rows. A field-owned file belongs
to exactly one record, so its readable-by set is that one record's and nothing
more. Under a shared reference model the field case would have had to union
too — which is exactly how copying a file id into a more public record would
silently widen access.

Denials report FILE_DOWNLOAD_DENIED, distinct from the attachments path's
ATTACHMENT_DOWNLOAD_DENIED: the file BELONGS TO one record rather than being
ATTACHED TO several, and the message should not claim otherwise.

acl: 'public_read' is the opt-out, and now an explicit declaration rather than
the silent default every field file used to get. Genuinely public images —
anything embedded in an <img src>, which cannot carry a bearer token — must
declare it.

Gates nothing that is open today. A pre-cutover field holds an inline blob or
an external URL, never a sys_file id, so no existing file has an owner recorded
and none of them start being gated; the gate engages only for files a field has
actually claimed, and disengages when ownership is released. Both are covered
by regression tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHpGw3GBA9aFpfwVArRWfd

* feat(storage): verifyFileReferences — the executable R4 acceptance gate

ADR-0104 D3 wave 2 planned to gate enabling collection on "reference counting is
verified", which is aspirational rather than checkable. This makes it runnable.

verifyFileReferences() compares GROUND TRUTH (what records' file-class fields
actually hold) against RECORDED OWNERSHIP (sys_file.ref_object/ref_id/ref_field)
and classifies every disagreement by whether it can cause DATA LOSS once
collection is on, because only that class should block a release:

  blocking
    unowned_reference  a record holds a file nothing owns — with collection on
                       it looks free and its bytes go
    foreign_owner      a record holds a file owned by a different slot; that
                       reference is invisible to the lifecycle, so the bytes go
                       when the recorded owner releases
    shared_reference   one file held by two slots — exclusivity was violated,
                       so copy-on-claim did not run, and one file's ACL is
                       serving two parents

  advisory
    stale_owner        owned but no longer held; fails toward retention
    unreferenced_file  committed and unpointed-at — storage cost, not risk

The scan is read-only: it never writes, tombstones or deletes. The point is that
a ledger may not be handed authority over an irreversible delete until it has
been shown to agree with reality, so this is meant to be run repeatedly, on real
tenant data, and to report zero blocking discrepancies on consecutive runs
before the gated collection change may merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHpGw3GBA9aFpfwVArRWfd

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/adr-0104-d3w2-pr4-governed-download.md

@@ -0,0 +1,52 @@
+---
+"@objectstack/service-storage": minor
+---
+
+feat(storage): governed download for field-owned files — ADR-0104 D3 wave 2 (PR-4)
+
+A file owned by a record's field (`sys_file.ref_object` / `ref_id`, set by
+PR-3) is now authorized on download the same way an attachment is: the caller
+must be able to READ the file's parent record, or be its uploader. Previously
+only `attachments`-scope files were gated and every field file kept an
+anonymous capability URL.
+
+**Parent resolution differs by surface, and that asymmetry is the point.** An
+attachment may hang off many records, so its readable-by set is the union over
+its `sys_attachment` join rows. A field-owned file belongs to exactly one
+record, so its readable-by set is that one record's — nothing more. Under a
+shared reference model the field case would have had to union too, which is
+what makes copying a file id into a more public record silently widen access.
+
+Denials are reported as `FILE_DOWNLOAD_DENIED` (403), distinct from the
+attachments path's `ATTACHMENT_DOWNLOAD_DENIED`, since the file *belongs to* one
+record rather than being *attached to* several.
+
+**`acl: 'public_read'` is the opt-out**, and now an explicit declaration rather
+than the silent default every field file used to get. Genuinely public images —
+anything embedded in an `<img src>`, which cannot carry a bearer token — must
+declare it.
+
+**Dual-mode safe, gates nothing that is open today.** A pre-cutover field holds
+an inline blob or an external URL, never a `sys_file` id, so no existing file
+has an owner recorded and none of them start being gated. The gate engages only
+for files a record's field has actually claimed, and disengages again when
+ownership is released.
+
+---
+
+Also adds `verifyFileReferences()` — the executable form of ADR-0104's R4
+acceptance gate. It compares ground truth (what records' file fields actually
+hold) against recorded ownership, and classifies disagreements by whether they
+could cause data loss once collection is enabled:
+
+- **blocking** — `unowned_reference` (a held file nothing owns), `foreign_owner`
+  (a record holds a file owned by another slot), `shared_reference` (one file
+  held by two slots, i.e. exclusivity was violated). Each would let a later reap
+  delete bytes a record still points at.
+- **advisory** — `stale_owner` (owned but no longer held; fails toward
+  retention) and `unreferenced_file` (storage cost, not a correctness problem).
+
+The scan is read-only — it never writes, tombstones, or deletes. A ledger may
+not be given authority over irreversible deletes until it has been shown to
+agree with reality, so this must report zero blocking discrepancies on real
+tenant data, on consecutive runs, before the gated collection change may merge.

packages/services/service-storage/src/index.ts

@@ -16,5 +16,17 @@ export { installAttachmentLifecycleHooks, createSysFileReapGuard, createUploadSe
 export type { AttachmentLifecycleEngine, AttachmentLifecycleLogger } from './attachment-lifecycle.js';
 export { installFileReferenceHooks, FileReferenceCopyError } from './file-reference-lifecycle.js';
 export type { FileReferenceEngine, FileReferenceLogger } from './file-reference-lifecycle.js';
+export {
+  verifyFileReferences,
+  formatFileReferenceReport,
+  BLOCKING_ISSUE_KINDS,
+} from './verify-file-references.js';
+export type {
+  FileReferenceIssue,
+  FileReferenceIssueKind,
+  FileReferenceReport,
+  VerifyReferencesEngine,
+  VerifyReferencesOptions,
+} from './verify-file-references.js';
 export { installAttachmentAccessHooks, installAttachmentReadVisibility } from './attachment-access-hooks.js';
 export type { AttachmentSharingLike } from './attachment-access-hooks.js';

packages/services/service-storage/src/storage-routes.test.ts

@@ -320,7 +320,7 @@ describe('Storage REST Routes', () => {
       expect(authorizeFileRead).not.toHaveBeenCalled();
     });
 
-    it('never gates a non-attachments file (avatars / field files stay open)', async () => {
+    it('never gates a file with neither an attachments scope nor a field owner', async () => {
       const { server, store: s, authorizeFileRead } = serverWith('deny');
       await commit(s, { id: 'a5', scope: 'user', key: 'user/a5.png' });
       const res = await hit(server, '/api/v1/storage/files/:fileId', 'a5');
@@ -334,6 +334,75 @@ describe('Storage REST Routes', () => {
       const res = await hit(server, '/api/v1/storage/files/:fileId/url', 'a6');
       expect(res._status).toBe(200);
     });
+
+    // ── Field-owned files (ADR-0104 D3 wave 2) ─────────────────────
+    describe('field-owned files', () => {
+      const owned = (id: string, over: any = {}) => ({
+        id,
+        scope: 'user',
+        key: `user/${id}.png`,
+        ref_object: 'product',
+        ref_id: 'p1',
+        ref_field: 'image',
+        ...over,
+      });
+
+      it('gates a field-owned file even though its scope is not attachments', async () => {
+        const { server, store: s, authorizeFileRead } = serverWith('deny');
+        await commit(s, owned('fo1'));
+        const res = await hit(server, '/api/v1/storage/files/:fileId/url', 'fo1');
+        expect(res._status).toBe(403);
+        // Distinct from the attachments code — this file BELONGS to one
+        // record, it is not attached to several.
+        expect(res._json?.code).toBe('FILE_DOWNLOAD_DENIED');
+        expect(authorizeFileRead).toHaveBeenCalledOnce();
+      });
+
+      it('401s an unauthenticated field-owned download', async () => {
+        const { server, store: s } = serverWith('unauthenticated');
+        await commit(s, owned('fo2'));
+        const res = await hit(server, '/api/v1/storage/files/:fileId', 'fo2');
+        expect(res._status).toBe(401);
+        expect(res._json?.code).toBe('AUTH_REQUIRED');
+      });
+
+      it('allows an authorized field-owned download', async () => {
+        const { server, store: s } = serverWith('allow', { downloadTtl: 120 });
+        await commit(s, owned('fo3'));
+        const res = await hit(server, '/api/v1/storage/files/:fileId/url', 'fo3');
+        expect(res._status).toBe(200);
+      });
+
+      it('opts out via acl: public_read (the embedding escape hatch)', async () => {
+        const { server, store: s, authorizeFileRead } = serverWith('deny');
+        await commit(s, owned('fo4', { acl: 'public_read' }));
+        const res = await hit(server, '/api/v1/storage/files/:fileId/url', 'fo4');
+        expect(res._status).toBe(200);
+        expect(authorizeFileRead).not.toHaveBeenCalled();
+      });
+
+      /**
+       * DUAL-MODE REGRESSION. A pre-cutover field holds an inline blob or an
+       * external URL, never a sys_file id — so no legacy file is ever claimed,
+       * `ref_object` stays unset, and this change gates nothing that used to
+       * be open. An unclaimed upload behaves the same way.
+       */
+      it('does not gate an unclaimed file (no ref_object) — legacy fields keep working', async () => {
+        const { server, store: s, authorizeFileRead } = serverWith('deny');
+        await commit(s, { id: 'fo5', scope: 'user', key: 'user/fo5.png' });
+        const res = await hit(server, '/api/v1/storage/files/:fileId', 'fo5');
+        expect(res._status).toBe(302);
+        expect(authorizeFileRead).not.toHaveBeenCalled();
+      });
+
+      it('does not gate a file whose ownership was released', async () => {
+        const { server, store: s, authorizeFileRead } = serverWith('deny');
+        await commit(s, owned('fo6', { ref_object: null, ref_id: null, ref_field: null }));
+        const res = await hit(server, '/api/v1/storage/files/:fileId', 'fo6');
+        expect(res._status).toBe(302);
+        expect(authorizeFileRead).not.toHaveBeenCalled();
+      });
+    });
   });
 
   describe('PUT/GET /_local/raw/:token', () => {

packages/services/service-storage/src/storage-routes.ts

@@ -27,17 +27,20 @@ export interface StorageRoutesOptions {
    * is a tracked follow-up needing cookie sessions or signed links).
    */
   resolveSession?: (req: IHttpRequest) => Promise<{ userId?: string } | null | undefined>;
-  /**
-   * Authorize a DOWNLOAD of an `attachments`-scope file (#2970 item 2). When
-   * wired, the download endpoints (`/files/:fileId` and `/files/:fileId/url`)
-   * consult this for `scope==='attachments'`, non-`public_read` files only:
+   /**
+   * Authorize a DOWNLOAD of a parent-governed file (#2970 item 2, extended by
+   * ADR-0104 D3 wave 2). When wired, the download endpoints
+   * (`/files/:fileId` and `/files/:fileId/url`) consult this for
+   * `scope==='attachments'` files AND for field-owned files (`ref_object` set),
+   * excluding `public_read` ones:
    *   - `unauthenticated` → 401 (no session)
-   *   - `deny` → 403 (session, but cannot read a parent record the file is
-   *     attached to and is not the owner)
+   *   - `deny` → 403 (session, but cannot read the parent record the file
+   *     belongs to / is attached to, and is not the owner)
    *   - `allow` → a short-lived signed URL is issued
-   * Non-attachments files (field files, avatars, org logos) keep the stable
-   * anonymous capability URL — they are embedded in `<img src>` which cannot
-   * carry a bearer token, and are out of scope for the attachments leak.
+   * A file with neither an attachments scope nor a field owner — an unclaimed
+   * upload, an org logo — keeps the stable anonymous capability URL, as does
+   * any file explicitly marked `acl: 'public_read'` (the opt-in for genuinely
+   * public embedding, since `<img src>` cannot carry a bearer token).
    * When absent (bare kernels, tests), all downloads stay open (back-compat).
    */
   authorizeFileRead?: (file: FileRecord, req: IHttpRequest) => Promise<FileReadVerdict>;
@@ -80,17 +83,30 @@ export function registerStorageRoutes(
   const sessionTtl = opts.sessionTtl ?? 86400;
   const downloadTtl = opts.downloadTtl ?? 300;
 
-  // ── Download authorization gate (#2970 item 2) ───────────────────────
-  // Only `attachments`-scope, non-public files are gated; everything else
-  // keeps the stable anonymous capability URL (image/avatar embedding).
+  // ── Download authorization gate (#2970 item 2, ADR-0104 D3 wave 2) ───
+  // Two kinds of file are gated, both deriving access from a PARENT record:
+  //   - `attachments`-scope files, via their sys_attachment join rows;
+  //   - field-owned files, via the single record whose field owns them
+  //     (`ref_object`/`ref_id`, ADR-0104 D3 wave 2).
+  // `acl: 'public_read'` opts a file back out to the stable anonymous
+  // capability URL — needed for genuinely public embedding (`<img src>`
+  // cannot carry a bearer token), and now an explicit declaration rather
+  // than the silent default it used to be for every field file.
+  //
+  // Dual-mode safe: a legacy field holds an inline blob or an external URL,
+  // never a `sys_file` id, so no legacy file has `ref_object` set and none of
+  // them start being gated by this change.
+  //
   // Returns the signed-URL TTL to use, or `false` if a response was already
   // sent (401/403) and the handler must stop.
   const authorizeDownload = async (
     file: FileRecord,
     req: IHttpRequest,
     res: IHttpResponse,
   ): Promise<number | false> => {
-    if (file.scope !== 'attachments' || file.acl === 'public_read' || !opts.authorizeFileRead) {
+    const fieldOwned = !!file.ref_object && file.ref_id != null && file.ref_id !== '';
+    const gated = file.scope === 'attachments' || fieldOwned;
+    if (!gated || file.acl === 'public_read' || !opts.authorizeFileRead) {
       return presignedTtl;
     }
     let verdict: FileReadVerdict;
@@ -104,10 +120,17 @@ export function registerStorageRoutes(
       return false;
     }
     if (verdict === 'deny') {
-      res.status(403).json({
-        error: 'You do not have access to a record this file is attached to',
-        code: 'ATTACHMENT_DOWNLOAD_DENIED',
-      });
+      res.status(403).json(
+        fieldOwned
+          ? {
+              error: 'You do not have access to the record this file belongs to',
+              code: 'FILE_DOWNLOAD_DENIED',
+            }
+          : {
+              error: 'You do not have access to a record this file is attached to',
+              code: 'ATTACHMENT_DOWNLOAD_DENIED',
+            },
+      );
       return false;
     }
     return downloadTtl;

packages/services/service-storage/src/storage-service-plugin.ts

@@ -463,12 +463,20 @@ function buildAuthSessionResolver(
 }
 
 /**
- * Authorize an `attachments`-scope download (#2970 item 2). Builds the FULL
- * caller ExecutionContext via `resolveAuthzContext` (the same shared resolver
- * rest-server uses — a bare `{ userId }` would lack the resolved permissions
- * the parent RLS needs), then allows when the caller is the file's owner or
- * can READ a record the file is attached to. Returns `undefined` (routes stay
- * open) when the auth service or engine is absent.
+ * Authorize a parent-governed download (#2970 item 2; extended for field-owned
+ * files by ADR-0104 D3 wave 2). Builds the FULL caller ExecutionContext via
+ * `resolveAuthzContext` (the same shared resolver rest-server uses — a bare
+ * `{ userId }` would lack the resolved permissions the parent RLS needs), then
+ * allows when the caller is the file's owner or can READ the file's parent
+ * record. Returns `undefined` (routes stay open) when the auth service or
+ * engine is absent.
+ *
+ * "Parent" resolves differently for the two surfaces, and that asymmetry is the
+ * point of the ownership model: an attachment may hang off MANY records, so its
+ * readable-by set is the union over its join rows; a field-owned file belongs
+ * to exactly ONE record, so its readable-by set is that record's and nothing
+ * more. A shared model would have had to union field references too, silently
+ * widening access whenever one file id was copied into a more public record.
  */
 function buildFileReadAuthorizer(
   ctx: PluginContext,
@@ -487,6 +495,22 @@ function buildFileReadAuthorizer(
       // Uploader / owner may always download.
       if (file.owner_id && String(file.owner_id) === String(authz.userId)) return 'allow';
 
+      // Field-owned (ADR-0104 D3 wave 2): exactly ONE record's field holds
+      // this file, so access is that record's READ access — never a union.
+      if (file.ref_object && file.ref_id != null && file.ref_id !== '') {
+        try {
+          const visible = (await (engine as any).find(String(file.ref_object), {
+            where: { id: file.ref_id },
+            fields: ['id'],
+            limit: 1,
+            context: authz,
+          })) as Array<Record<string, unknown>>;
+          return visible?.length ? 'allow' : 'deny';
+        } catch {
+          return 'deny'; // unknown/failing owner object — fail closed
+        }
+      }
+
       // Otherwise: readable via any parent record this file is attached to.
       const links = (await (engine as any).find('sys_attachment', {
         where: { file_id: file.id },