Skip to content

fix: restore args.ckpt_step after load_other_checkpoint - #2243

Open
keepkeen wants to merge 1 commit into
THUDM:mainfrom
keepkeen:fix/ckpt-step-restore-leak
Open

fix: restore args.ckpt_step after load_other_checkpoint#2243
keepkeen wants to merge 1 commit into
THUDM:mainfrom
keepkeen:fix/ckpt-step-restore-leak

Conversation

@keepkeen

Copy link
Copy Markdown

What

--ref-ckpt-step / --opd-teacher-ckpt-step permanently leak into args.ckpt_step, so later checkpoint loads in the same process resolve against the wrong iteration.

Root cause

load_other_checkpoint restores args.ckpt_step only when the saved value was non-None, instead of when an override was applied:

old_ckpt_step = None
if model_tag == "ref" and self.args.ref_ckpt_step is not None:
    old_ckpt_step = self.args.ckpt_step          # normally None
    self.args.ckpt_step = self.args.ref_ckpt_step
...
if old_ckpt_step is not None:                    # False -> never restored
    self.args.ckpt_step = old_ckpt_step

args.ckpt_step is a user override that defaults to None, so in the normal case the restore never fires and the ref/teacher step stays in args.ckpt_step for the rest of the process.

Consequence

get_load_checkpoint_path_by_args honors the leaked value for every subsequent load (model.py):

if getattr(args, "ckpt_step", None):
    iteration = args.ckpt_step

__init__ loads in the order ref → teacher → old_actor, so:

  • with --keep-old-actor, the old_actor / rollout_actor weights load from iteration <ref_ckpt_step> of the actor checkpoint — silently wrong weights for the importance ratio if that iteration happens to exist, a crash if it doesn't;
  • with OPD but no --opd-teacher-ckpt-step, the teacher load inherits the ref's step the same way.

Fix

Save ckpt_step in the same old_args tuple the function already uses for load / no_load_optim / no_load_rng / finetune, and restore unconditionally — identical pattern, one more field.

Verification

slime/backends/megatron_utils/actor.py needs a live Megatron to import, so there is no CPU unit test to attach (none of actor.py is unit-tested today). The defect is a three-line control-flow inversion verifiable by inspection: the override is applied under ref_ckpt_step is not None, the restore under old_ckpt_step is not None, and the two conditions differ exactly when args.ckpt_step started as None — its default. The GPU CI paths that pass --ref-load (e.g. test_qwen3_4B_ppo.py) don't set --ref-ckpt-step, which is why this never surfaced there.

The restore was guarded on the *saved* value instead of on whether an
override was applied:

    old_ckpt_step = None
    if model_tag == "ref" and self.args.ref_ckpt_step is not None:
        old_ckpt_step = self.args.ckpt_step      # normally None
        self.args.ckpt_step = self.args.ref_ckpt_step
    ...
    if old_ckpt_step is not None:                # False -> never restored
        self.args.ckpt_step = old_ckpt_step

When the process-wide args.ckpt_step is None (the normal case — it is a
user override, not something the loader sets), a --ref-ckpt-step or
--opd-teacher-ckpt-step override permanently leaks into args.ckpt_step.
Every subsequent load that resolves a path through
get_load_checkpoint_path_by_args then honors the leaked step
(model.py: `if getattr(args, "ckpt_step", None): iteration = args.ckpt_step`):

- with --keep-old-actor, the old_actor/rollout_actor weights load from
  iteration <ref_ckpt_step> of the *actor* checkpoint — silently wrong
  reference weights for the importance ratio if that iteration exists,
  a crash if it doesn't;
- with OPD but no --opd-teacher-ckpt-step, the teacher load inherits the
  ref step the same way (ref loads before teacher in __init__).

Save ckpt_step in the same old_args tuple as load/no_load_optim/
no_load_rng/finetune and restore unconditionally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant