feat: save forward autocast context in FunctionCtx#185
Open
Chamberlain0w0 wants to merge 2 commits into
Open
Conversation
kilinchange
requested changes
Jul 8, 2026
|
|
||
| const std::vector<bool> &needs_input_grad() const; | ||
|
|
||
| const AutocastContext &GetForwardAutocastContext() const; |
Collaborator
There was a problem hiding this comment.
命名按取设值函数来吧,forward_autocast_context,下面 SaveForwardAutocastContext 改成 set_forward_autocast_context
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
为
autograd::FunctionCtx增加了用于保存并恢复前向执行时的 autocast state (放在框架中是 AutocastContext 结构体)的功能,目标是对齐 PyTorchtorch.amp.custom_fwd/custom_bwd的语义:前向阶段记录当前生效的 AutocastContext;当自定义 backward 或重计算路径需要重新执行部分前向子图时,可以显式恢复该前向 AutocastContext。改动内容
GetCurrentAutocastContext(),用于快照当前线程本地的 autocast 上下文。AutocastGuard(const AutocastContext&)的构造方法,调用方可以通过 RAII 语义临时恢复一个已保存的 autocast 上下文。Function::Forward()之前,将前向 autocast 上下文保存到FunctionCtx。FunctionCtx::GetForwardAutocastContext(),供自定义 backward 和重计算路径获取前向阶段保存的 autocast 上下文。FunctionCtx保存变量时,同时重置已保存的前向 autocast 上下文。FunctionCtx暴露,只有显式重算前向计算的代码路径才会通过上述接口来主动恢复该上下文。FunctionCtx;PyTorch 对齐证据
torch 没有显式包出一个 AutoCastContext 结构体,但是的确存在 forward 会存下此 context 然后 backward 再度调用恢复的情况。对于常见的两种情况:
custom backward:相关逻辑实现位于
torch/amp/autocast_mode.py。args[0]就是 FunctionCtx。自定义前向会存下来相应信息,反向再用相应信息开启 autocast。重计算:相关逻辑实现位于
torch/utils/checkpoint.py。有一个存 autocast 信息的 helper,它在
Checkpoint.Function.forward()会被调用然后记录信息,在Checkpoint.Function.backward()重新恢复。