fix(autotp): when using autotp, ignore the consistency of certain data within the tp_group.#8125
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a865dd365c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@1787648106 thanks for your fix. I agree with codex that the import from transformers should be catched. If there is import error, then the tensor won't be HF model KV cache. This would avoid break existing use cases. Also can you sign-off your commits to pass DCO check? Thanks! |
Done, thank you! |
1787648106
force-pushed
the
ignore_check_cache_while_autotp
branch
from
4d08564 to
fbfe8fd
Compare
|
Hi @1787648106 can you fix the formatting errors? You may want to refer to this link for essential pre CI checks. Thanks! |
done |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
There was a problem hiding this comment.
Hi @1787648106,
Thank you for submitting this PR, great catch! I left a comment below. Happy to hear you thought.
…tp_group Signed-off-by: lunzhongwang <1787648106@qq.com>
Signed-off-by: lunzhongwang <1787648106@qq.com>
Signed-off-by: lunzhongwang <1787648106@qq.com>
Signed-off-by: lunzhongwang <1787648106@qq.com>
Signed-off-by: lunzhongwang <1787648106@qq.com>
1787648106
force-pushed
the
ignore_check_cache_while_autotp
branch
from
11c88df to
2fd0a45
Compare
tohtana
left a comment
tohtana
left a comment
There was a problem hiding this comment.
Thank you for updating this PR! Can you check the following comment?
Signed-off-by: lunzhongwang <1787648106@qq.com>
tohtana
left a comment
tohtana
left a comment
There was a problem hiding this comment.
Thank you for the update! Looks good to me.
4 participants
When using
autotp, the consistency check for the KV cache within the TP group is bypassed. Recent popular methods, such as on-policy distillation, require the model to generate output during training, which would otherwise cause the data consistency check to fail.