Delta Migration Feature#1044
Open
yashin4112 wants to merge 51 commits intodevfrom
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Implements “delta migration” support by introducing an iteration counter, restart flow, and per-iteration mapping/UID persistence so subsequent runs can selectively recreate content types, deduplicate assets, and update existing entries.
Changes:
- Replaces upload-api runtime config import with a JSON-backed config file and adds a helper to refresh/update config at runtime.
- Adds “Restart Migration” UX + iteration tracking in UI (stepper/header/logs) and introduces an Entry Mapper table for iteration > 1.
- Adds API support for iteration-scoped LowDB files (content types, fields, entries, uid-mapper) plus delta helpers (asset dedup + entry update script + restart endpoint).
Reviewed changes
Copilot reviewed 50 out of 55 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/src/services/fileProcessing.ts | Reads refreshed config during file validation/processing. |
| upload-api/src/services/aws/client.ts | Switches AWS config source to JSON config file. |
| upload-api/src/routes/index.ts | Refreshes config per request and returns sanitized config via /config. |
| upload-api/src/helper/index.ts | Adds updateConfigFile helper for JSON config read/write. |
| upload-api/src/controllers/sitecore/index.ts | Invokes Sitecore extractEntries during mapper creation. |
| upload-api/src/config/index.ts | Removes TS-based config module. |
| upload-api/src/config/index.json | Adds JSON-based config template. |
| upload-api/package.json | Formatting-only change. |
| upload-api/migration-wordpress/libs/extractEntries.ts | Adds WordPress entry extraction to build entryMapping. |
| upload-api/migration-sitecore/libs/extractEntries.js | Adds Sitecore entry extraction to build entryMapping. |
| upload-api/migration-sitecore/index.js | Exposes extractEntries from migration-sitecore package entrypoint. |
| upload-api/migration-drupal/libs/extractEntries.js | Adds Drupal DB entry extraction to build entryMapping. |
| upload-api/migration-drupal/libs/createInitialMapper.js | Attaches Drupal entryMapping into initial mapper output. |
| upload-api/migration-drupal/index.js | Exposes extractEntries from migration-drupal package entrypoint. |
| upload-api/migration-contentful/libs/extractEntries.js | Adds Contentful export entry extraction grouped by content type. |
| upload-api/migration-contentful/libs/createInitialMapper.js | Attaches Contentful entryMapping into initial mapper output. |
| upload-api/migration-contentful/index.js | Exposes extractEntries from migration-contentful package entrypoint. |
| upload-api/migration-aem/package-lock.json | Lockfile updates (types/undici-types). |
| ui/src/services/api/migration.service.ts | Adds restart + entry-mapping API calls for delta migration UI. |
| ui/src/pages/Migration/index.tsx | Adds “Restart Migration” execution path and iteration increment. |
| ui/src/context/app/app.interface.ts | Adds iteration and stepValue to migration state and defaults. |
| ui/src/components/Stepper/HorizontalStepper/HorizontalStepper.tsx | Resets stepper state on restart/iteration increment. |
| ui/src/components/MigrationFlowHeader/index.tsx | Derives CTA label (“Restart Migration”, etc.) into Redux state. |
| ui/src/components/LogScreen/MigrationLogViewer.tsx | Avoids duplicate completion notifications; sets “Restart Migration” CTA. |
| ui/src/components/LegacyCms/legacyCms.scss | Adds layout styles for editable local-path display. |
| ui/src/components/LegacyCms/Actions/LoadUploadFile.tsx | Adds inline editing UX for local path in restart iterations. |
| ui/src/components/ContentMapper/index.tsx | Switches to Entry Mapper view when iteration > 1; refetches on iteration change. |
| ui/src/components/ContentMapper/index.scss | Scopes table wrapper styles under .content-mapper-container. |
| ui/src/components/ContentMapper/entryMapper.tsx | New Entry Mapper table for selecting entries to update in delta runs. |
| ui/src/components/ContentMapper/contentMapper.interface.ts | Adds EntryMapperType interface for entry-mapping table rows. |
| ui/package.json | Formatting-only change. |
| package.json | Formatting-only change. |
| api/src/utils/package.json | Adds a package.json under src/utils (appears unrelated to runtime). |
| api/src/utils/field-attacher.utils.ts | Makes content type creation conditional based on iteration and prior existence. |
| api/src/utils/entry-update.utils.ts | Adds logic to remove/update entries and build update config for CLI script. |
| api/src/utils/entry-update-script.cjs | Adds CLI migration script to update existing entries (incl. asset resolution). |
| api/src/utils/entry-duplicate.utils.ts | Flags duplicate entries in entry mapper DB. |
| api/src/utils/content-type-checker.utils.ts | Adds helper to skip content type creation if created in prior iterations. |
| api/src/utils/asset-update.utils.ts | Adds asset dedup logic across iterations and ref replacement in entry JSON. |
| api/src/services/updateEntryCli.service.ts | Adds service to run CLI “update entries” script after delta import. |
| api/src/services/runCli.service.ts | Persists uid-mapping output into iteration-scoped LowDB (uid-mapper.json). |
| api/src/services/projects.service.ts | Stores iteration on project creation; uses iteration-scoped DBs on delete. |
| api/src/services/migration.service.ts | Adds restart endpoint + delta pre/post steps (asset dedup + entry updates). |
| api/src/services/contentMapper.service.ts | Adds iteration-scoped DB access + entry-mapping endpoints + uid enrichment. |
| api/src/routes/migration.routes.ts | Adds POST /restart/:orgId/:projectId route. |
| api/src/routes/contentMapper.routes.ts | Adds entry mapping GET + entry status update PUT routes. |
| api/src/models/uidMapper.ts | Adds iteration-scoped LowDB model for combined asset/entry uid mapping. |
| api/src/models/project-lowdb.ts | Adds iteration field to Project model. |
| api/src/models/contentTypesMapper-lowdb.ts | Converts content type mapper DB to iteration-scoped per-project storage. |
| api/src/models/FieldMapper.ts | Converts field mapper DB to iteration-scoped per-project storage. |
| api/src/models/EntryMapper.ts | Adds iteration-scoped LowDB model for entry mapper rows. |
| api/src/controllers/projects.contentMapper.controller.ts | Exposes entry mapping + update status endpoints from controller. |
| api/src/controllers/migration.controller.ts | Exposes restart migration endpoint from controller. |
| api/package.json | Bumps CLI dependencies for migration/update flows. |
Files not reviewed (1)
- upload-api/migration-aem/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… service and utils; unit test cases added
…t tests for content type creation logic
…sting data reference
…ping resolution in Contentful service
…esModal with async handling for step changes
…le package.json and package-lock.json files
…r better handling of missing and media blocks
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…iple package.json and package-lock.json files
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
- Moved region configuration to a dedicated constants file for better organization and maintainability. - Updated various services and components to utilize the new REGION_CONFIG, ensuring optional chaining is applied for safer access to properties. - Corrected the spelling of 'failureNotification' in multiple locations for consistency.
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…ng and jetpack/story support - Added `unwrapSingleChildGroup` function to simplify processing of core/group blocks with a single inner block. - Updated `createSchema` to utilize the new function for better handling of nested structures. - Enhanced `schemaMapper` to support jetpack/story blocks, mapping them to a repeatable group with title, alt, caption, and image fields. - Improved unit tests to validate new functionality for single-child groups and jetpack/story mappings.
- Removed unnecessary console logging in contentMapper service. - Introduced `normalizeNicenameForUid` function to standardize taxonomy UIDs by replacing hyphens and spaces with underscores. - Enhanced `formatChildByType` to handle child elements more robustly, ensuring correct extraction of attributes and inner blocks. - Updated `extractItems` and `extractTaxonomy` to utilize the new normalization function for consistent taxonomy UID generation. - Improved schema mapping for cover blocks to ensure proper handling of inner blocks and attributes.
…d a Set to track source content type UIDs for improved filtering.
- Introduced `getAxiosResponseHeader` to handle both AxiosHeaders and plain object headers, improving header value retrieval. - Updated asset validation to utilize the new function for content-type checks, enhancing robustness in asset validation logic.
…tallations - Updated createAppManifest to support fetching marketplace installations when the extension mapper file is missing. - Introduced marketplaceSourceStackId parameter to allow specifying the source stack for installations. - Added error handling for installation fetching and ensured proper writing of the app manifest. - Enhanced unit tests to cover new functionality and edge cases related to marketplace installations.
…Stack function - Introduced tests for the fetchMarketplaceInstallationsForStack function to validate pagination handling, error scenarios, and logging behavior. - Ensured proper aggregation of installations based on stack UID and added fallback mechanisms for failed fetch attempts. - Enhanced test coverage for various edge cases, including handling of non-Error exceptions.
…ping resolution in Contentful service
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…iple package.json and package-lock.json files
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…nfig - Extract and map Contentful taxonomies from export; upload-api and mapper integration - Contentful service: field/widget helpers, taxonomy metadata locale resolution (mapper + sys.locale) - API: Vitest thresholds; migration and user unit tests (SSO, createTaxonomy mocks) - app.json placeholder updates and related fixes
…ty tests - Decreased the branch coverage threshold from 56 to 55 in Vitest configuration. - Added a new test case to the contentstack-oauth-url utility to handle empty or non-string inputs correctly.
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
- Added checks for valid destination stack ID and improved error messages for asset index loading. - Implemented asynchronous file system operations to verify asset index existence and type. - Updated tests to cover new validation logic and ensure proper handling of asset index scenarios.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.