feat: address migration gaps (WIP) - #9487
Draft
pjwerneck wants to merge 12 commits into
Draft
Conversation
- Fix C2 item from migration gaps review
- Fix D1 item from migration gaps review
- Fix C1 item from migration gaps review - Pin dependents to the published version, not the bumped one, so a package released later in the same run does not need a version PyPI lacks
- Fix A2 item from migration gaps review, private folders only - Code quality fixes
- Fix A3 item from migration gaps review, floor mechanism only - Every floor starts at 0, so no peer is refused; protocol 1 has never shipped, so 0 is the only correct value today.
- Fix A3 item from migration gaps review, peer gate policy - A peer with UNKNOWN version is still skipped, because nothing can be negotiated without its version
- Fix A2b item from migration gaps review, completing A2 - A folder this client owns is reused after an upgrade, becasuse a peer that has not upgraded still looks for the old name - Delete _filter_patch_compatible, which has no caller left.
- B2: version the crypto key file; refuse an unknown later version because a private key cannot be rebuilt
- B4: version the persisted caches; reset on an unknown later version, because the client rebuilds them. On-disk format becomes {"version", "entries"}
- D2: freeze the VersionInfo V1 field set; every field V2 adds needs a default.
- A4: delete the two unused version exception classes.
- Fix B3 item from migration gaps review - Stamp the format version under a reserved _meta key, so older clients that treat every top-level key as a peer email skip it safely - Log and skip an unknown peer state instead of dropping the peer in silence; the writer keeps other entries, so the record is not erased on Drive
- Fix B1 item from migration gaps review; A2a already fixed the folder half - A later client can reshape a field while the object still parses, which gives a wrong restore silently. Ever load site already falls back to downloading all events, so refusing costs one slow cold start.
- Fix C3 case 1 from migration gaps review - The existing tests assert the negotiated version only; removing the protocol-0 codec fails this test and leaves those passing
- Fix A5 item from migration gaps review; the entry named the wrong pair, the different is between the two dataset methods, not jobs vs datasets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes most of the migration gaps that block safe upgrades: private Drive folders are adopted by rename, P2P folders are found regardless of the version in the name, and durable surfaces (checkpoints, rolling state, crypto keys, peers JSON, local caches) carry or refuse versions instead of failing silently.
Adds a per-protocol minimum-supported floor (shipped inert at "0"), stops refusing peers on client semver mismatch, and hardens release/CI (export → publish → tag → bump, client migration tests in CI, numeric version ordering). Job protocol skew delivery is covered end-to-end; dataset transport multi-copy (A1) and a few process/policy items remain open.
Note
The base branch is
stephen/syft-client-migrations, notdev. This branch assumes that branch will be merged first.Changes
Done
• A2a Private Drive folders: adopt highest older version by rename (_find_or_adopt_versioned_folder); personal / checkpoints / rolling-state all use it.
• A2b P2P folders: version-tolerant lookup (_find_p2p_folder_id); filter_patch_compatible deleted.
• A3 floor Per-package MIN_SUPPORTED*_PROTOCOL_VERSION = "0", on-wire min_supported_version, enforced in negotiate_protocol_version (jobs + datasets).
• A3 peer gate Client-version INCOMPATIBLE no longer skips peers (get_peer_compatibility_status); force_ignore_protocol_version removed.
• A4 Dead ClientVersionMismatchError / ProtocolVersionMismatchError deleted (inert PROTOCOL_VERSION semver field stays).
• A5 Unknown-peer defaults documented; forced paths log warnings + test_unknown_peer_forced_path.py.
• B1 Checkpoints / rolling state refuse later version; local load reports failure instead of silent pass.
• B2 Crypto keys file stamped (CRYPTO_KEYS_VERSION); later version refused (no rebuild possible).
• B3 SYFT_peers.json version under _meta; unknown peer state log-and-skip (raw map rewrite keeps entries safe).
• B4 PersistedDict envelope {"version", "entries"}; old flat file = v0; later version resets empty.
• C1 Release order: export artifacts → publish → tag → bump; --dependents published; protocol_bump_missing.
• C2 Client migration tests in CI (just test-client-migrations + post-release).
• C3 case 1 Job protocol-0 skew delivery path covered (test_job_protocol_skew_delivery.py).
• D1 Numeric version ordering via _version_order (no more string "10" < "2").
• D2 VersionInfo additive field guard tests (test_version_info_fields.py).
• D3 Live peer-schema dict identity asserted in job + dataset negotiation tests (no typed holder).
Pending
• A1 Dataset collection transport still flat; metadata can point at v1/ the peer never gets; manager gets no peer schemas.
• A2c Two compatible private folders still hit _expect_one → manual delete on Drive.
• A3 policy Floor always "0" (inert); raise vs warn vs read-only undecided; login mismatch still deletes state.
• A3 residual target_protocol_versions_for_peers uses min without floor (dormant until A1).
• A4 residual Dead semver PROTOCOL_VERSION still required on the wire.
• B4 residual Cost of file_hashes reset (event replay) unmeasured.
• C1 residual One-time transition bump (dev still names published versions); 0.1.117 fixture not a true published dump.
• C1 accepted No provisional lock for unreleased protocol drift (by decision).
• C3 case 2 Dataset skew E2E (blocked on A1).
• C3 case 3 Upgrade-in-place + mixed peers E2E (blocked on A3 policy).
• C4 Migration guidance never walked end-to-end by a human.
• E1 Only VersionInfo has real object migrations; jobs/datasets still V1-only rehearsal gap.
Testing
Asana task
https://app.asana.com/1/1185126988600652/project/1216249688888494/task/1217005273770094?focus=true