perf/reliability: seed + data-import bypass the engine's bulk-insert path → N round-trips, N summary recomputes, no transient retry

[os-zhuang]

Summary

The two bulk-write paths — the seed loader (defineDataset replay) and the user-facing data import (CSV / bulk records) — write records one at a time through the single-record engine path, even though ObjectQL's engine already has an efficient batched branch (driver.bulkCreate + parent-deduplicated summary recompute). Over a remote DB (turso) this makes both:

• slow — N round-trips + N summary recomputes instead of ~N/batch, and
• fragile — each round-trip can hit a transient network error that is caught → the row is dropped → no retry.

Surfaced while debugging the 2026-07-06 HotCRM first-boot incident: a HotCRM seed's crm_contact inserts intermittently dropped rows on transient turso errors → the seed reported errors → the runtime's seed-once stamp never landed → per-env kernel churn (re-seed every boot). Non-deterministic (network-sensitive): the same seed inserted 5/5 contacts on one env and 1/5 on another.

Evidence

Seed loader — per record

packages/metadata-protocol/src/seed-loader.ts — loadDataset() loops per record calling this.engine.insert(objectName, record, opts) with a single record. No array / bulk form is ever constructed.

Data import — per record

Entry points (packages/rest/src/rest-server.ts):

• POST /data/:object/import (sync, ≤5k rows) — ~L3524
• POST /data/:object/import/jobs (async, 5k–50k, persists sys_import_job) — ~L3598
• POST /data/:object/createMany (optional bulk endpoint, may be unwired) — ~L5749

Both sync + async delegate to packages/rest/src/import-runner.ts runImport(), which loops per row calling p.createData({...}) / p.updateData({...}) with a single record (~L201–274). It does not construct arrays or use the engine's bulk branch.

The engine ALREADY has the efficient bulk primitive

packages/objectql/src/engine.ts:

• insert(object, data: any | any[]) — the Array.isArray(hookContext.input.data) branch (~L2158) routes to driver.bulkCreate(...) — one round-trip per batch.
• recomputeSummaries() (~L1746–1782) deduplicates parent ids across the whole array and recomputes once per unique parent (1000 children of the same parent = 1 rollup recompute, not 1000). Called once per insert() call (~L2206).

Because neither seed nor import passes arrays, both get N single-record insert() calls → N round-trips + N recomputeSummaries() invocations.

No transient-error retry (silent row loss)

• import-runner.ts ~L260: catch → results.push({ ok:false, action:'failed', ... }) → continue. No retry.
• seed-loader.ts ~L413: catch → errors.push(...) → continue. No retry.

A transient driver/network blip (fetch failed / timeout / connection reset) on one row silently drops that row. Import returns a per-row report, so it's visible, but there's no automatic recovery.

Impact

Import 1000 rows into an object with a rollup summary, over a remote turso DB (~100ms/round-trip):

• 1000 inserts + up to 1000 summary recomputes = 2000+ round-trips ≈ 3+ minutes, plus a handful of transient failures that drop rows.
• With K rollup parents: 1000 inserts + K×1000 aggregate queries.

The seed variant additionally triggered kernel churn in the HotCRM incident (dropped seed rows → errs>0 → the cloud seed-once stamp never lands → re-seed every cold boot).

Proposed fix

A shared batched-write helper used by BOTH seed-loader and import-runner:

1. Batch rows (e.g. 100–500/batch) and call the engine's array/bulk insert → driver.bulkCreate (one round-trip per batch). The engine already dedups summary recomputes for arrays.
2. Retry TRANSIENT driver errors (network / timeout / connection reset) per batch with backoff. Do not retry logical errors (validation / constraint).
3. On a batch that fails on a logical error, degrade to per-row within that batch so one bad row doesn't drop the whole batch (bad row reported, others succeed).
4. (Optional) collapse summary recompute to once per parent at the end of the whole load.

Related runtime-side follow-up (cloud, tracked separately): the seed-once stamp guard is errs===0 — a single transient error blocks the stamp forever → churn; relax to stamp-on-progress + surface residual errors via reportObserved.

Acceptance criteria

• Seed + import route through the shared bulk helper; a 1000-row load over a bulkCreate-capable driver issues ~ceil(N/batch) writes, not N.
• Summary-recompute count is bounded by unique parents, not row count.
• A transient driver error is retried and does not drop the row; a logical error is reported per-row without failing the batch.
• Tests: bulk-path write-count assertion, transient-retry, logical-error per-row degradation, summary-dedup.

Refs

• HotCRM new-workspace "打开环境一直失败" incident — cloud#752 / Update Spec priority Roadmap (2026.02) #754 / feed api #756, framework#2661.