Fix eval and checkpointing under load-balanced CP#4613
Draft
huytransformer wants to merge 1 commit into
Draft
Conversation
…rder setup_train_loop installed the context-parallel load-balance reorder as a bare map(reorder_fn, data_iterator) and passed the result everywhere the original iterator went. A map object supports iteration but loses both the MaxText iterator interface and the concrete iterator type. With ici_context_parallelism > 1 and context_parallel_load_balance (default true), affected runs fail at the first scheduled evaluation because the eval loop calls eval_data_iterator.reset(), or at the first grain checkpoint save or restore because checkpointing dispatches on the concrete iterator type (isinstance branches for RemoteIteratorWrapper, PlaceHolderDataIterator, and iterator lists in checkpointing.py, then data_iterator.local_iterator access). Wrapping the iterator in a forwarding class cannot restore the checkpoint behavior because the dispatch is by isinstance, not attribute presence. Instead, the original iterator now continues to setup_training_state and checkpointing unchanged, and only the batch consumers receive the reordered view: the DataLoader gets _ReorderedDataIterator (per element when the grain pipeline returns an iterator list for 0 < expansion_factor_real_data < 1, matching the list support in data_loader.py), and the eval iterator is wrapped directly since it is never checkpointed and only needs reset() and iteration. Checkpoint restore mutates the iterator in place, so the loader's view tracks restored state. Unit tests cover the reordered view (per-batch reorder, reset forwarding, iteration protocol) and the loader-view builder (single iterator wrapped, lists preserved with each element wrapped around the original iterator). An integration test in setup_train_loop_nnx_test.py pins both installation sites: with ici_context_parallelism=2 and context_parallel_load_balance the DataLoader holds the reordered view around the exact iterator object setup_train_loop returns for checkpointing, that returned iterator stays unwrapped, and the eval iterator is the reordered view. Local checks: DECOUPLE_GCLOUD=TRUE PYTHONPATH=src python3 -m pytest tests/unit/train_utils_test.py -q (20 passed) and tests/unit/checkpointing_test.py tests/unit/grain_utility_test.py tests/unit/multihost_dataloading_test.py -q (27 passed, 5 skipped; the env var stubs ml_goodput_measurement, absent locally and unrelated); the integration test executed end-to-end and passed on CPU with XLA_FLAGS=--xla_force_host_platform_device_count=4 in a venv providing ml_collections; pyink --pyink-indentation=2 --line-length=122 --check on all three files (unchanged); git diff --check clean.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Start with a short description of what the PR does and how this is a change from
the past.
The rest of the description includes relevant details and context, examples:
If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456
You can also provide a comma-separated list. If you don't want to close a bug but
simply to reference it, use BUGS, e.g.:
BUGS: b/123456
Notice 1: Once all tests pass, the "pull ready" label will automatically be assigned.
This label is used for administrative purposes. Please do not add it manually.
Notice 2: For external contributions, our settings currently require an approval from a MaxText maintainer to trigger CI tests.
Tests
Please describe how you tested this change, and include any instructions and/or
commands to reproduce.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.