feat: Add migration utilities and fix flaky job tests #1332
+672
−85
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add migrate_columns() and supporting functions for Phase 2 of the 0.14.6 → 2.0 migration: - analyze_columns(): Identify columns needing type labels - migrate_columns(): Add core type markers to column comments - NATIVE_TO_CORE_TYPE mapping for type conversion - Support for bool/datetime special cases - Dry-run mode for previewing changes Also adds placeholder stubs for Phase 3-4 migration functions: - migrate_external(): For external storage migration - migrate_filepath(): For filepath attribute migration - finalize_migration(): For Phase 4 finalization These functions implement the migration guide documented in datajoint-docs/src/how-to/migrate-from-0x.md. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The tests test_sigint, test_sigterm, test_suppress_dj_errors, and test_populate_exclude_error_and_ignore_jobs were flaky due to a race condition: jobs created with scheduled_time=NOW(3) might not pass the scheduled_time <= NOW(3) check if checked in the same millisecond. Fix by using delay=-1 in auto-refresh during populate(), ensuring jobs are scheduled 1 second in the past and immediately schedulable. Also update test_populate_exclude_error_and_ignore_jobs to use delay=-1 in its explicit refresh() call. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace bare except: with except ImportError: in diagram.py - Replace assert statements with explicit raises in blob.py - Replace assert False with explicit raises in expression.py and declare.py - Implement hash verification in objectref.py Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
dj.key was removed in 2.0 but was still listed in __all__ without being imported or defined. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove `schema` alias for `Schema` (use `dj.Schema` instead of `dj.schema`) - Remove `Di` alias for `Diagram` (use `dj.Diagram` or `dj.ERD`) - Update all examples, tests, and docstrings Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update outdated docs.datajoint.com URLs to new paths: - diagram.py: /how-to/installation/ - heading.py: /how-to/migrate-from-0x/ - expression.py: /how-to/migrate-from-0x/ Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove key_hash function (legacy job table debugging) - Remove hash.py module - Fix test_erd.py to use dj.Diagram instead of dj.Di - Bump version to 2.0.0a20 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove ERD alias (use dj.Diagram) - Rename test_erd_algebra to test_diagram_algebra - Remove test_diagram_aliases test - Bump version to 2.0.0a21 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
migrate.pydelay=-1for immediate job schedulingMigration utilities added:
analyze_columns(): Identify columns needing type labels in commentsmigrate_columns(): Add core type markers (:int32:,:<blob>:, etc.) to column commentsNATIVE_TO_CORE_TYPEmapping for MySQL native → DataJoint core type conversionAlso adds placeholder stubs for Phase 3-4 migration functions:
migrate_external(): For external storage migration (blob, attach)migrate_filepath(): For filepath attribute migrationfinalize_migration(): For Phase 4 finalizationFlaky test fix:
The tests
test_sigint,test_sigterm,test_suppress_dj_errors, andtest_populate_exclude_error_and_ignore_jobswere flaky due to a race condition: jobs created withscheduled_time=NOW(3)might not pass thescheduled_time <= NOW(3)check if checked in the same millisecond.Fixed by using
delay=-1in auto-refresh duringpopulate(), ensuring jobs are scheduled 1 second in the past and immediately schedulable.Test plan
🤖 Generated with Claude Code