feat(import): server-side coercion + write-mode for /data/:object/import#2505
Open
baozhoutao wants to merge 6 commits into
Open
feat(import): server-side coercion + write-mode for /data/:object/import#2505baozhoutao wants to merge 6 commits into
baozhoutao wants to merge 6 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📓 Docs Drift CheckThis PR changes 4 package(s): 95 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
Contributor
Author
|
前端配套 PR:objectstack-ai/objectui#2133(向导走服务端 /import + 写入模式 UI)。 |
github-actions
Bot
added
the
dependencies
…upsert) for /data/:object/import Move all special-value handling for list import to the server so one code path serves every client. Callers POST raw spreadsheet values + a column mapping; the server coerces each cell from the object's field metadata and routes rows to insert / update / upsert. - spec: ImportRequest / ImportResponse / ImportRowResult / ImportWriteMode / ImportMapping schemas (writeMode, matchFields, runAutomations, trimWhitespace, nullValues, createMissingOptions, skipBlankMatchKey, dryRun) - rest/import-coerce: coerceRow/coerceFieldValue — boolean, number, date/ datetime/time→ISO, select label→code, multi-select split+match, lookup name→id via a caching RefResolver; createMissingOptions + trimWhitespace toggles - rest-server /import: normalize mapping (record | entry[]), validate writeMode↔matchFields, resolve field metadata, findExisting with blank/none/ ambiguous guards, per-row action report (created/updated/skipped/failed), hook suppression via skipAutomations, clearer 413 for oversized payloads - client SDK: data.import(object, request) on both data namespaces - tests: import-coerce (unit) + import-integration (upsert/update/dryRun) — 28
Accept format:'xlsx' with a base64-encoded workbook (xlsxBase64) and an optional sheet selector. The server flattens ExcelJS cells to their human-visible text (formula results, hyperlink text, rich-text runs, dates -> ISO) so a parsed xlsx yields the same cells a CSV export would, then feeds them through the existing coercion + upsert pipeline. - spec: extend ImportRequestSchema format enum with 'xlsx'; add xlsxBase64 + sheet fields. - rest: xlsxCellToString + parseXlsxToRows helpers (dynamic exceljs import so csv/json imports don't pay for it); wire the xlsx branch into the /import body parser; 400 on malformed workbooks. - tests: 3 integration cases (coercion parity with csv, named-sheet selection + format inference, malformed-payload 400).
…ity)
Reference cells (name/email/id) now resolve more safely:
- id fast-path: try an exact id match first, so a pasted record id is
authoritative and never shadowed by a name/label collision.
- ambiguity guard: when the first matching candidate field hits >1 record,
stop and report a new 'reference_ambiguous' error instead of silently
linking whichever row came back first ($top:2 detects the dup).
- structured resolver contract: RefResolver may return { id, ambiguous,
matchedField }; bare string|undefined from legacy resolvers still works
(normalizeRefMatch).
Tests: unit (structured result + ambiguous + not-found) and integration
(duplicate-name ambiguity, id fast-path).
…50k ceiling Add an asynchronous import path alongside the synchronous POST /import route for large files (up to 50,000 rows). A create request persists a sys_import_job row and returns 201 immediately; a background worker streams the batch through the SAME shared runImport() core the sync route uses, persisting progress and a capped per-row results report on the job row. - spec: import-job schemas + ImportJobApiContracts (create/progress/results/ list/cancel); IMPORT_JOB_MAX_ROWS=50k; status enum. - platform-objects: sys_import_job object (state/counters/results/timestamps), registered by the REST plugin via the manifest service. - rest: extract runImport()/prepareImportRequest() so sync + async paths are byte-identical; 5 async routes + fire-and-forget worker with onProgress persistence and cooperative cancellation (process-local signal, persisted status as durable source of truth). - tests: real engine + protocol integration for create→poll→results→list→ cancel, 50k ceiling (413), and unknown-job 404s. 195/195 rest tests green.
Add createImportJob / getImportJobProgress / getImportJobResults / listImportJobs / cancelImportJob to the top-level and scoped `data` namespaces, mirroring the async REST routes. Large payloads are posted once via createImportJob (returns a jobId immediately); callers poll progress / results / history and can cancel. Older servers return 404 → rejected promise, so consumers can feature-detect and fall back to the synchronous `import`. 5 client tests assert route/method/query-string shaping and response unwrapping.
Add POST /data/import/jobs/:jobId/undo — logically reverses a finished import: deletes the records it created and restores the fields it updated to their pre-import values. - import-runner: capture an ImportUndoLog (created ids + per-updated before-snapshots of only the written fields) when captureUndo is set. - sys_import_job: persist undo_log (json) + reverted_at (datetime). - rest-server: worker captures undo for non-dry-run jobs <= 5000 rows (IMPORT_JOB_UNDO_MAX_ROWS); undo route deletes/restores with automations skipped, then stamps reverted_at. undoable/revertedAt surfaced on progress + summary DTOs. - spec: undoable/revertedAt on progress+summary; UndoImportJobResponse; undoImportJob contract. - client SDK: data.undoImportJob(jobId) on both namespaces + test. - tests: end-to-end undo (delete created + restore updated), double-undo 409, unknown-job 404.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景 / Problem
列表数据导入需要把电子表格里的原始值写进对象。特殊值(布尔、数字、日期、下拉选项 select、关联 lookup、多选)如果在每个前端各自转换,逻辑会重复且不一致,而且 lookup 名称→id 必须查库,前端做不了。
方案 / Solution
把所有特殊值处理下沉到服务端
POST /api/v1/data/:object/import,前端只发原始值 + 列映射,一条代码路径服务所有客户端(小文件同步、后续大文件异步)。改动
export.zod.ts):新增ImportRequest/ImportResponse/ImportRowResult/ImportWriteMode/ImportMappingwriteMode:insert|update|upsert(后两者需matchFields)runAutomations(批量默认关钩子)、trimWhitespace、nullValues、createMissingOptions、skipBlankMatchKey、dryRuncoerceRow/coerceFieldValue,export-format 的逆运算:RefResolver,按 displayField→name/email/id 逐个候选查)createMissingOptions、trimWhitespace开关;逐字段收集错误而非首错即停/import:归一化 mapping(record 或 entry[])、校验writeMode↔matchFields、解析字段元数据、findExisting(blank/none/ambiguous 守卫)、逐行 action 报告(created/updated/skipped/failed)、skipAutomations抑制钩子、超限 413 提示走异步任务data.import(object, request)import-coerce(单元)+import-integration(upsert/update/dryRun)共 28 个,全绿未做(后续 P1/P2)
大文件异步 import_job + 流式 + 进度 + 5万上限、导入历史、真正的钩子开关落地(现在
skipAutomations已透传,等引擎支持);任务级 undo、字段类型自动探测。验证 / Verification
pnpm --filter @objectstack/rest exec vitest run import-coerce import-integration→ 28 passed。前端配套改动在 objectui PR(依赖本 PR 发布后的 client)。