Skip to content

Fix CommsLogger.stop_profiling_comms not disabling global profiling#8137

Open
vineethsaivs wants to merge 1 commit into
deepspeedai:masterfrom
vineethsaivs:fix/stop-profiling-comms-flag
Open

Fix CommsLogger.stop_profiling_comms not disabling global profiling#8137
vineethsaivs wants to merge 1 commit into
deepspeedai:masterfrom
vineethsaivs:fix/stop-profiling-comms-flag

Conversation

@vineethsaivs

Copy link
Copy Markdown
Contributor

What

CommsLogger.stop_profiling_comms() set self.prof_all = True, the same value start_profiling_comms() sets, so calling stop was a no-op:

def start_profiling_comms(self):
    self.prof_all = True

def stop_profiling_comms(self):
    self.prof_all = True   # should be False

prof_all is the flag that makes the comms logger profile every collective (deepspeed/comm/comm.py gates on ... or comms_logger.prof_all or ...). Because stop set it back to True, once global comm profiling was started it could never be turned off. The sibling pair in the same class, start_profiling_op / stop_profiling_op, correctly adds then removes ops, which confirms the intended start=on / stop=off asymmetry.

Fix

Set prof_all = False in stop_profiling_comms().

Test

Added tests/unit/comm/test_comms_logger.py, a CPU-only unit test that starts then stops comm profiling and asserts prof_all ends up False. It fails before the change (stays True) and passes after.

start_profiling_comms() sets prof_all = True to profile every collective,
but stop_profiling_comms() also set prof_all = True, so calling it was a
no-op and global comm profiling could never be turned off once started.
Set prof_all = False, mirroring the start/stop_profiling_op pair which
correctly adds then removes ops.

Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! This looks good to me.
Can you just change the license header to "# Copyright (c) DeepSpeed Team."? DeepSpeed is not under Microsoft anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants