Skip to content

SIGUSR1 live observer dump: ask a running process how it's progressing (fixes #660)#696

Merged
InauguralPhysicist merged 2 commits into
InauguralSystems:mainfrom
Nitjsefnie-OSC:observer-dump-660
Jul 22, 2026
Merged

SIGUSR1 live observer dump: ask a running process how it's progressing (fixes #660)#696
InauguralPhysicist merged 2 commits into
InauguralSystems:mainfrom
Nitjsefnie-OSC:observer-dump-660

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Implements #660 v1: kill -USR1 <pid> on a running program prints one row per live binding to stderr at the next loop safepoint, then the program continues unchanged:

step_count | 3213 | when=3214 | entropy=0.00407362 | dH=-1.12811e-06 | converged
epsilon | 0.9683805288358108 | when=3214 | entropy=0.999814 | dH=-1.15838e-07 | equilibrium
# scope=frame
n | 2000000 | when=1 | entropy=- | dH=- | unobserved
loss | 1.1606843800047104 | when=3214 | entropy=0.996007 | dH=7.40942e-06 | equilibrium

Shape (per the issue's sketch)

  • Handler installed once at CLI startup (sigaction, SA_RESTART); it ONLY sets a volatile sig_atomic_t — no allocation or I/O in the handler. Embedders via eigs_embed don't get it.
  • The flag is tested at the four existing per-iteration safepoints (interpreter LOOP_STALL_CHECK/LOOP_CAP_CHECK cases + the shared cores the JIT/AOT call), mirroring the eigs_loop_cap_step cost profile. CAS test-and-clear → exactly one dump per signal under MT.
  • Dump covers module scope + the dumping thread's live frame (+ its loop child); runtime __* bindings and builtins filtered; containers as <type:count> (dict-held config is invisible to the observer by design — issue Scope §2); strings capped. The dump path allocates nothing.
  • MT: module-scope walk holds the existing Data race: module-env binding creation (main) vs worker name resolution — worker can read freed names/values/bucket arrays #607 g_module_env_lock; frame envs are the dumping thread's own view (the issue's allowed option). No new locking scheme.
  • Not built (issue: "Later"): --observer-socket, periodic auto-report.

One honest deviation: when= is max(assign_counts, obs_age)

The issue says Env.assign_counts backs when — verified on main, that's only half true: the SET_LOCAL fast path never bumps assign_counts (the compiler keeps when is x correct only by forcing interrogated names onto the slow paths — interrogated NameSet, compiler.c). A plain fn-local assigned 300k times reads assign_counts == 0, which would be exactly the "confident wrong answer" Scope §1 forbids. obs_age IS bumped once per compiled assignment and resets per call env, so the dump merges both. Verified live: a parameter prints when=1 next to loss | ... | when=3214 in the same frame.

Requirements from the issue's "Done when"

  • Dump + continue: test-asserted (program prints DONE, exit 0 after dump).
  • Every row carries its assign count; when=1 visibly distinct from settled bindings.
  • ASan+UBSan+leaks clean, including a spawn-task dump (a worker-thread dump was also exercised live during review).
  • Idle cost: bench_dmg_shape.eigs n=5 medians 166ms (main) → 163ms (branch), re-measured independently at 164 vs 163-164 interleaved — no measurable cost, so no EIGS_OBSERVE gate (issue: gate only "if it measures").
  • tests/test_sigusr1_dump.sh (suite block [99d], 13 checks): drives a real SIGUSR1 against a live process, waits on observable READY/DONE markers (no sleeps), asserts row shapes incl. when= counts, single-threaded and with a spawned task. Mutation-verified: neutering the safepoint check fails 6 of its checks.

Suites: release 3143/3143, make asan 3147/3147, freestanding gate green.

Pre-existing findings surfaced while in there (filed, not fixed here)

Inherent bound worth stating: a program stuck in a long syscall or loop-free straight-line code only dumps at its next loop safepoint — that's the issue's safepoint design, not a gap. And SIGUSR1 no longer terminates eigenscript CLI processes (that's the feature).


Generated by Claude Fable 5 (brief, review), Kimi K3 (implementation)

InauguralSystems#660)

kill -USR1 <pid> prints one row per live binding to stderr at the next
loop safepoint:

    name | value | when=<assigns> | entropy=<H> | dH=<dH> | trajectory

The sigaction handler (SA_RESTART, installed by the CLI) only sets a
volatile sig_atomic_t; the dump runs in normal thread context at the
loop-cap per-iteration check (interpreter LOOP_STALL/CAP_CHECK cases +
the shared eigs_loop_*_step cores the JIT/AOT call), so idle cost is one
flag test — unmeasurable on bench_dmg_shape (166ms -> 163ms n=5 medians),
so no EIGS_OBSERVE gate. First thread to see the flag dumps (atomic
test-and-clear), covering module scope (root of the env chain) plus the
dumping thread's live call frame; the module walk holds the existing InauguralSystems#607
g_module_env_lock under MT — no new locking scheme.

Every row carries its assign count so a fresh per-call binding (when=1)
reads differently from a settled one — the issue's silent-tolerance
failure mode. env->assign_counts alone can't back that: the SET_LOCAL
fast path never bumps it (the compiler only keeps it consistent for
interrogated names), so the dump prints max(assign_counts, obs_age),
which vm_park_call_env resets per call. Unobserved slots say
"unobserved" rather than borrowing a window-less verdict.

Test: tests/test_sigusr1_dump.sh (suite block [99d]) drives the signal
against a live process — observable READY/DONE barriers, no sleeps —
asserts the row shapes incl. when= counts single-threaded and with a
spawned task, and requires DONE + exit 0 afterward.

Suite: 3143/3143 release (baseline 3130/3130), 3147/3147 under
make asan (ASan+UBSan+detect_leaks); freestanding-check green.

Co-Authored-By: Kimi K3 <noreply@kimi.com>
Comment thread src/vm.c Fixed
Comment thread src/vm.c Fixed
…cation marker

Addresses the CodeQL annotation on InauguralSystems#696; no behavior change.

Round-trip check compares bit patterns (memcmp) — exact, -0.0-preserving.

Co-Authored-By: Kimi K2.7 Code <noreply@kimi.com>
@Nitjsefnie

Copy link
Copy Markdown
Contributor Author

Addressed both CodeQL annotations in b991531 (src/vm.c only, no behavior change):

  • Offset-before-range-check (the high-severity one): the string-scan loop now tests i < 100 before dereferencing v->data.str[i]. The old order was functionally safe only because the string is NUL-terminated; the new order doesn't rely on that. Also bounded the "..." truncation-marker append explicitly (o + 5 <= nbuf) — unreachable with the current 160-byte buffer, purely defensive.
  • Float equality: the shortest-round-trip loop's strtod(buf) == n was the intended exact comparison (round-trip formatting is defined by exact equality), but I've switched it to a bitwise memcmp of the doubles — same loop behavior, strictly more precise (-0.0 keeps its sign in the dump; NaN falls through to the widest precision exactly as before), and it says what it means.

Suites re-run on the amended head: release 3143/3143, ASan+leaks 3147/3147, [99d] 13/13.

@InauguralPhysicist
InauguralPhysicist merged commit b886ef7 into InauguralSystems:main Jul 22, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants