fix(nnx): fix gpt3 paxml checkpoint conversion state map and layout - #4555
Open
hsuan-lun-chiang wants to merge 1 commit into
Open
fix(nnx): fix gpt3 paxml checkpoint conversion state map and layout#4555hsuan-lun-chiang wants to merge 1 commit into
hsuan-lun-chiang wants to merge 1 commit into
Conversation
hsuan-lun-chiang
force-pushed
the
fix/nnx-gpt3-ckpt
branch
from
July 21, 2026 09:40
7ff3bbd to
7b0e353
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
hsuan-lun-chiang
force-pushed
the
fix/nnx-gpt3-ckpt
branch
4 times, most recently
from
July 24, 2026 03:17
bbcc873 to
05653a9
Compare
hsuan-lun-chiang
marked this pull request as ready for review
July 24, 2026 06:57
hsuan-lun-chiang
requested review from
RissyRan,
aireenmei,
bvandermoon,
gagika,
gobbleturk,
hengtaoguo,
igorts-git,
jiangjy1982,
parambole,
richjames0,
shralex,
shuningjin and
suexu1025
as code owners
July 24, 2026 06:57
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
NuojCheng
approved these changes
Jul 24, 2026
xibinliu
reviewed
Jul 24, 2026
hsuan-lun-chiang
force-pushed
the
fix/nnx-gpt3-ckpt
branch
from
July 27, 2026 03:23
05653a9 to
9f239a6
Compare
hsuan-lun-chiang
requested review from
A9isha,
SurbhiJainUSC,
abhinavclemson,
khatwanimohit and
vipannalla
as code owners
July 27, 2026 03:23
hsuan-lun-chiang
requested review from
darisoy,
dipannita08 and
huytransformer
as code owners
July 27, 2026 03:23
hsuan-lun-chiang
force-pushed
the
fix/nnx-gpt3-ckpt
branch
3 times, most recently
from
July 27, 2026 09:49
78284b4 to
7074735
Compare
xibinliu
reviewed
Jul 27, 2026
hsuan-lun-chiang
force-pushed
the
fix/nnx-gpt3-ckpt
branch
16 times, most recently
from
July 28, 2026 12:14
2ea8719 to
d078108
Compare
hsuan-lun-chiang
force-pushed
the
fix/nnx-gpt3-ckpt
branch
from
July 28, 2026 12:42
d078108 to
8c90d74
Compare
xibinliu
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix the Errors in b/535891456:
When executing
convert_gpt3_ckpt_from_paxml.py, verification failed with:Cause: Recent NNX updates added dynamic RNG state (such as AQT count) into model variables, which are not present in PaxML checkpoints.
Fix: convert NNX state to linen format before doing the checkpoint conversion, when the
rngswill be filtered out .When restoring the converted checkpoint in
train.py, loading failed with:Cause:
convert_gpt3_ckpt_from_paxml.pywas saving the rawTrainStateNNXstructure directly instead of callingtrain_state_nnx.to_checkpoint_dict(converted_state)to format it into the standard Linen-interchangeable checkpoint layout.Fix: Added
converted_state = train_state_nnx.to_checkpoint_dict(converted_state)prior to saving whencfg.pure_nnxis active.Test
Test with
convert_gpt3_ckpt_from_paxml.pyandtrain.pyas in the DAG.Logs
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.