Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements the Divergence Proximal Policy Optimization (DPPO) algorithm, including the configuration, the loss function implementation, and corresponding tests. The changes are well-structured and the implementation appears correct. I've only found a minor typo in the reference link to the DPPO paper in both the configuration file and the docstring, for which I've suggested corrections.
| cispo_eps_clip_high: 5 # offset for upper bound of importance sampling ratio clipping (as opposed to PPO token update clipping) | ||
|
|
||
| # DPPO parameters (only used when policy_loss_type: "dppo") | ||
| # See: https://arxiv.org/abs/2602.04879 |
There was a problem hiding this comment.
There appears to be a typo in the arXiv link. The ID 2602.04879 seems to be incorrect. The correct ID for 'Divergence Proximal Policy Optimization' is likely 2402.04879.
# See: https://arxiv.org/abs/2402.04879| Section G.2 the authors find Top-K masking provides no significant benefit | ||
| over the simpler binary approximation, so we only implement binary here. | ||
|
|
||
| See: https://arxiv.org/abs/2602.04879 |
There was a problem hiding this comment.
There appears to be a typo in the arXiv link. The ID 2602.04879 seems to be incorrect. The correct ID for 'Divergence Proximal Policy Optimization' is likely 2402.04879.
| See: https://arxiv.org/abs/2602.04879 | |
| See: https://arxiv.org/abs/2402.04879 |
1 participant
Referencing #1028
Implements DPPO, which replaces PPO's ratio based clipping with their own divergence based binary masking