[DO NOT MERGE] Probe hang stack dump against a live Kit process - #7153
[DO NOT MERGE] Probe hang stack dump against a live Kit process#7153mataylor-nvidia wants to merge 4 commits into
Conversation
A test that crashes reported a traceback, because PYTHONFAULTHANDLER=1 installs faulthandler for SIGSEGV and friends. A test that hung reported nothing: the runner detects the hang and kills the process group with SIGKILL, which cannot be caught, so no handler ever ran. The report carried system tables and the last -v test name, and nothing that points at the hung code. The runner now asks the process where it is stuck before killing it. tools/hang_dump.py registers SIGUSR1 with faulthandler.register, and capture_test_output_with_timeout signals the process and drains the dump into pre_kill_diag, which already flows into the startup_hang, timeout, and shutdown_hang reports. The dump is taken twice: identical stacks seconds apart are what tell a wedged process from a slow one. SIGTERM and SIGABRT cannot be used for this. AppLauncher binds both to a handler that calls SimulationApp.close(), which is itself what a shutdown hang is stuck inside, so either would re-enter the hang. A Python-level signal handler would not run regardless, since those execute between bytecodes and a thread wedged in a native Kit, CUDA, or renderer call never returns to the interpreter loop. faulthandler.register installs a C-level handler that walks every thread from inside the signal handler, so it reports a process whose GIL will never be released.
The dump never reached CI. pytest captures at the file-descriptor level, so it has already pointed fd 2 at a temporary file of its own by the time the plugin loads; faulthandler.register(file=sys.__stderr__) stored fd 2 and wrote there. That buffer is discarded when the process is SIGKILLed, which is the only case the dump is ever written in, so a hung test still reported nothing but system tables. The dump now goes to a file named by ISAACLAB_HANG_DUMP, which the runner sets per test file and clears per attempt, mirroring the crash journal's ISAACLAB_TEST_JOURNAL. pytest does not redirect it, and the runner reads it after the process is gone. This is the same reason tools/ovrtx_log.py keeps the renderer log in a file. The regression tests missed this because they hung a bare `python script.py` child, which has no capture, so the dump reached stderr and they passed. They now hang a real `python -m pytest` child, reproducing the CI failure: against the previous implementation all three fail on `assert 'HANG STACK DUMP' in ''`. Found by the CI probe in the follow-up branch, which wedged a rendering correctness test and produced a timeout report with no stack.
Blocks test_rendering_cartpole.py forever after its first case has really rendered, so CI exercises the hang stack dump against a live Isaac Sim process rather than the toy subprocess the unit tests use. Kit is fully up and all of its threads are running when SIGUSR1 arrives, which is the part the unit tests cannot cover. Expect the file to be killed for "timeout" and its report to carry a HANG STACK DUMP section naming _probe_wedge. Not for merge.
13f9fad to
774826b
Compare
Run 1 result: the probe worked, and it found a real bugThe wedge landed exactly where intended, and the hang was detected on schedule — but no stack dump was produced:
Root causepytest captures at the file-descriptor level. It The decisive evidence is this probe's own The regression tests in #7152 missed this because they hung a bare Fix#7152 now writes the dump to a file named by Also added the Re-running to confirm the dump appears. Worth notingTwo things did work on run 1: the timeout fired at exactly 1700 s (1000 |
1 participant
Description
Do not merge. Throwaway CI probe for #7152, branched off it. Revert by deleting the marked block and the
_probe_wedge()call.What this proves
#7152 adds three regression tests that spawn a genuinely hung child and assert the stack dump appears. Those cover the mechanism, but the child is a toy
pythonprocess with one thread.What they cannot cover is the part that actually matters here: does a real Isaac Sim process answer
SIGUSR1? Kit runs dozens of native threads, installs its own signal handling, and parks the main thread inside C++ calls that never return to the interpreter loop. If Kit blocked or claimedSIGUSR1, the feature would pass CI and still be useless on the hangs it was built for.This probe wedges
test_rendering_cartpole.pyafter its first case has really rendered — so Kit is fully initialized, has produced actual frames, and all its threads are live at the moment the signal arrives.Note the injection is a hang, not a kill. Killing Kit produces a crash, and crashes already printed stacks before #7152 —
PYTHONFAULTHANDLER=1coversSIGSEGV/SIGABRT. The gap being closed is specifically hangs.Expected result
The
rendering-correctnessjob should showtest_rendering_cartpole.pykilled fortimeout, with both the job log and the JUnit report carrying:Two things to check:
_probe_wedgeis named, and the frames below it are the real pytest/Kit call chain — the diagnostic that did not exist before.Thread 0x...blocks. That is the evidencefaulthandler.registerwalks a live Kit process, not just the main thread.The two dumps should be identical, which is the intended "wedged, not slow" signal.
Cost and blast radius
test_rendering_cartpole.pycarries noPER_TEST_TIMEOUTSentry, so it runs onDEFAULT_TIMEOUT(1000 s) plus the 700 s cold-cache buffer it earns as the firstenable_cameras=Truefile — roughly 28 minutes before the dump fires, against the job's 120-minute budget. Left at the defaults deliberately so the probe exercises the real configuration.The orchestrator runs each file in its own process, so only
test_rendering_cartpole.pyis affected; the other six rendering files in the job run normally afterward.test-rendering-correctnessiscontinue-on-erroron pull requests, so a red result here does not block.Before
The same hang on
developproduces:No stack, anywhere. That is the gap #7152 closes.
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there