fix: restore args.ckpt_step after load_other_checkpoint - #2243
Open
keepkeen wants to merge 1 commit into
Open
Conversation
The restore was guarded on the *saved* value instead of on whether an
override was applied:
old_ckpt_step = None
if model_tag == "ref" and self.args.ref_ckpt_step is not None:
old_ckpt_step = self.args.ckpt_step # normally None
self.args.ckpt_step = self.args.ref_ckpt_step
...
if old_ckpt_step is not None: # False -> never restored
self.args.ckpt_step = old_ckpt_step
When the process-wide args.ckpt_step is None (the normal case — it is a
user override, not something the loader sets), a --ref-ckpt-step or
--opd-teacher-ckpt-step override permanently leaks into args.ckpt_step.
Every subsequent load that resolves a path through
get_load_checkpoint_path_by_args then honors the leaked step
(model.py: `if getattr(args, "ckpt_step", None): iteration = args.ckpt_step`):
- with --keep-old-actor, the old_actor/rollout_actor weights load from
iteration <ref_ckpt_step> of the *actor* checkpoint — silently wrong
reference weights for the importance ratio if that iteration exists,
a crash if it doesn't;
- with OPD but no --opd-teacher-ckpt-step, the teacher load inherits the
ref step the same way (ref loads before teacher in __init__).
Save ckpt_step in the same old_args tuple as load/no_load_optim/
no_load_rng/finetune and restore unconditionally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
--ref-ckpt-step/--opd-teacher-ckpt-steppermanently leak intoargs.ckpt_step, so later checkpoint loads in the same process resolve against the wrong iteration.Root cause
load_other_checkpointrestoresargs.ckpt_steponly when the saved value was non-None, instead of when an override was applied:args.ckpt_stepis a user override that defaults toNone, so in the normal case the restore never fires and the ref/teacher step stays inargs.ckpt_stepfor the rest of the process.Consequence
get_load_checkpoint_path_by_argshonors the leaked value for every subsequent load (model.py):__init__loads in the order ref → teacher → old_actor, so:--keep-old-actor, the old_actor / rollout_actor weights load from iteration<ref_ckpt_step>of the actor checkpoint — silently wrong weights for the importance ratio if that iteration happens to exist, a crash if it doesn't;--opd-teacher-ckpt-step, the teacher load inherits the ref's step the same way.Fix
Save
ckpt_stepin the sameold_argstuple the function already uses forload/no_load_optim/no_load_rng/finetune, and restore unconditionally — identical pattern, one more field.Verification
slime/backends/megatron_utils/actor.pyneeds a live Megatron to import, so there is no CPU unit test to attach (none ofactor.pyis unit-tested today). The defect is a three-line control-flow inversion verifiable by inspection: the override is applied underref_ckpt_step is not None, the restore underold_ckpt_step is not None, and the two conditions differ exactly whenargs.ckpt_stepstarted asNone— its default. The GPU CI paths that pass--ref-load(e.g.test_qwen3_4B_ppo.py) don't set--ref-ckpt-step, which is why this never surfaced there.