refactor(grpo): migrate TypedDict configs to pydantic BaseModel#2518
Open
NolenLiang wants to merge 13 commits into
Open
refactor(grpo): migrate TypedDict configs to pydantic BaseModel#2518NolenLiang wants to merge 13 commits into
NolenLiang wants to merge 13 commits into
Conversation
NolenLiang
added
the
CI:Lfast
NolenLiang
changed the title
Contributor
Author
|
/ok to test b37bbb9 |
Contributor
Author
|
/ok to test 77aec62768f3f5e17bba09dfea25e8b58b4df7b1 |
@NolenLiang, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
Contributor
Author
|
/ok to test 77aec62 |
Contributor
Author
|
/ok to test 77aec62 |
Contributor
Author
|
/ok to test 36dc855 |
Contributor
Author
|
/ok to test 85be127 |
NolenLiang
added
CI:L1
Contributor
Author
|
/ok to test 363aace |
Contributor
Author
|
/ok to test 7bfac42 |
Convert 6 TypedDict classes to pydantic BaseModel with extra="allow": - GRPOConfig, AsyncGRPOConfig, AdvEstimatorConfig, RewardScalingConfig, GRPOSaveState (grpo.py) - RewardShapingConfig (reward_functions.py) Update all dict-style access (config["key"]) to attribute access (config.key) across grpo.py, reward_functions.py, utils.py, trajectory_collector.py, and related tests. Signed-off-by: nliang <nliang@nvidia.com>
Tests use MasterConfig.model_construct() which skips pydantic validation and does not auto-convert nested dicts to BaseModel instances. Wrap grpo dict with GRPOConfig.model_construct(), and nested reward_scaling/reward_shaping/async_grpo with their respective BaseModel.model_construct() calls. Signed-off-by: nliang <nliang@nvidia.com>
Signed-off-by: nliang <nliang@nvidia.com>
- Run ruff format on grpo.py and trajectory_collector.py - Add skip_reference_policy_logprobs_calculation and calculate_advantages_on_gpu to grpo_math_1B.yaml reference config (new BaseModel defaults now serialize these keys) Signed-off-by: nliang <nliang@nvidia.com>
Signed-off-by: nliang <nliang@nvidia.com>
Signed-off-by: nliang <nliang@nvidia.com>
…construct - advantage_estimator.py: change estimator_config["key"] to estimator_config.key (AdvEstimatorConfig is now BaseModel) - test_async_utils.py: wrap grpo dict in GRPOConfig.model_construct() and async_grpo in AsyncGRPOConfig.model_construct() Signed-off-by: nliang <nliang@nvidia.com>
Signed-off-by: nliang <nliang@nvidia.com>
…construct 8 occurrences of plain dict estimator_config in test_grpo.py converted to AdvEstimatorConfig.model_construct(). Signed-off-by: nliang <nliang@nvidia.com>
…_functions RewardShapingConfig is now BaseModel; config["key"] = val must use config.key = val. Signed-off-by: nliang <nliang@nvidia.com>
run_grpo.py, run_grpo_nemo_gym.py, run_grpo_sliding_puzzle.py still used config.grpo["key"] and "async_grpo" in config.grpo patterns. The "in" check on a BaseModel doesn't work like dict membership, causing async routing to fall through to sync grpo_train. Signed-off-by: nliang <nliang@nvidia.com>
checkpointer.load_training_info() returns a plain dict from JSON deserialization. With GRPOSaveState as BaseModel, cast() alone doesn't convert it. Add isinstance check to construct BaseModel from dict on checkpoint resume. Signed-off-by: nliang <nliang@nvidia.com>
All fields in GRPOConfig, RewardScalingConfig, AsyncGRPOConfig, AdvEstimatorConfig, and RewardShapingConfig now have default values matching the reference config (grpo_math_1B.yaml). This allows constructing configs with just the overrides needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: nliang <nliang@nvidia.com>
Contributor
Author
|
/ok to test 72c0892 |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Issues
List issues that this PR closes (syntax):
Usage
# Add a code snippet demonstrating how to use thisBefore your PR is "Ready for review"
Pre checks:
Results before / after the changes