refactor: migration validation

[Lars Kemper (larskemper)]

Overall refactor of the migration validation behavior. Should lead to much less needed fixes and more accurate / actually fixable errors.

sorry, for the big pr, but refactor was needed 😅. Reach out if I can explain anything and please give me your ideas for better validation!

Changes (file by file)

• src/Controller/DataProviderController.php, src/Controller/HistoryController.php, src/Controller/PremappingController.php, src/Controller/StatusController.php:
  Introduced all common route definition constants to unify with platform

• [Added] src/Controller/ErrorResolutionController.php:
  Added two new routes:

  • /error-resolution/validate: provides validation for user created fixes to make sure the fix matches original migration validation
  • /error-resolution/example-field-structure: in case we can not determine the field type in the administration this route provides an example field structure of json association fields, so the user can follow this structure to create a valid fix.

• src/Migration/ErrorResolution/MigrationErrorResolutionService.php: added missing early return if no fixes could be loaded to avoid event spam and unnecessary code execution

• [Added] src/Migration/ErrorResolution/MigrationFieldExampleGenerator.php: the generator used for the example field structure route. It takes in a field and tries to build a accurate example structure for the user

• [!!!] src/Migration/ErrorResolution/MigrationFix.php: see the comment

• src/Migration/Mapping/MappingService.php: removed the mapping lookup. We decided to not validate of a given association is actually existing and just validate the given data

• src/Migration/Service/MediaFileProcessorService.php: add where clause to only fetch unprocessed media

• src/Migration/Service/MigrationDataWriter.php: change to ->count() to return number of entities that are processed in the current batch, not the absolute total

• [!!!] src/Migration/Validation/MigrationEntityValidationService.php:

  • Deleted association mapping lookup
  • New definition of required field: not system managed (createdAt, ...), actually required in db, no default value
  • Validated nested converted data
  • More naive way of validating association field, cause the before validation by serializer needed context that we actually can't provide in the migration. Now more like a structural validation of association data.
  • Move Exceptions to validation domain for better separation

• [!!!] src/Migration/Validation/MigrationFieldValidationService.php:

  • same as entity validator: more naive validation, support of nested fields
  • splited from entity validator to provide standalone field validation for the error resolution controller

• src/Migration/Validation/MigrationValidationResult.php: added log key generation to avoid multiple log for the same invalidation (avoid multiple fixes for same issue)

• src/Profile/Shopware/Converter/OrderConverter.php: bug fix (custom field was set to null before, instead of just not adding it to the converted data)

• src/Resources/app/administration/src/module/swag-migration/component/swag-migration-error-resolution/swag-migration-error-resolution-field/swag-migration-error-resolution-field-scalar/index.ts: added display of error messages that are provides by the validation route

• src/Resources/app/administration/src/module/swag-migration/component/swag-migration-error-resolution/swag-migration-error-resolution-field/swag-migration-error-resolution-field/index.ts: added auto fetch of example field structure in case the admin could not determine a specific field type

• src/Resources/app/administration/src/module/swag-migration/service/swag-migration-error-resolution.service.ts: support of nested field paths and overall simplification

[Nikolas Evers (vintagesucks)]

🚀🚀🚀

[Jozsef Damokos (jozsefdamokos)]

Looks good! Great job!

[Malte Janz (MalteJanz)]

Lars Kemper (@larskemper) appreciate the effort you put into this and we are moving in the right direction 💪

I only had a rough look because of the size of the PR and left some comments, but overall I'm fine with merging this into our feature branch (feel free to create follow up issues / PRs for my comments or fix them before merging, your decision. The nitpicks are just nice to have / some further discussion points).

But I don't think we reached a good state for merging the feature branch into trunk yet. I noticed there are still quite a few confusing errors being reported when I migrate from my SW5 shop with demo data, compared to the trunk state. Most likely we:

• might broke some converters logic that is producing validation errors now
• the validation logic still reports false positives

which we should investigate further before release 🙂