Live observer introspection: ask a long-running process how it's progressing from outside

[InauguralPhysicist]

Agent-era debugging gap from a real session: Tidepool's train.eigs ran ~10 minutes with no output, and there was no way to distinguish "slow but progressing" from "hung" without killing it. The runtime already knew — every tracked binding's trajectory is live in the observer — but nothing outside the process could ask.

The language's thesis is that programs are inspectable from the inside. This extends it: let the outside ask the same interrogatives.

Proposal (v1)

Signal-triggered observer dump:

kill -USR1 <pid>
# -> stderr, one line per dumped binding:
#    name | value | when (assigns) | entropy | dH | trajectory

Sketch:

• Handler sets a volatile sig_atomic_t only (no allocation in a handler); the dump happens at the next safe point.
• Precedent to mirror: the per-iteration loop-cap check (eigs_loop_cap_step, vm.c:1446), not "existing error checks" — there is currently no signal handler or safepoint flag in the VM, so this adds the first one. Cost should be one flag test; gate behind EIGS_OBSERVE=1 if it measures.
• The data is already there: ObserverSlot (eigenscript.h:243) holds entropy, last_entropy, dH, prev_dH, obs_age, last_value and the dH/value windows; Env.assign_counts backs when.
• MT: dump under the collector's locks, or dump the requesting thread's view with a note.

Scope — three things that must be right, or v1 lies

These are the difference between a useful dump and a confident wrong answer. All three verified against main.

1. Function-locals carry PER-CALL trajectories, not progress

Observer state is keyed to the (env, slot) binding, and each call gets a fresh env. So "walk the current call stack's envs" does not rescue the innermost hot bindings — it gets a one-assignment-deep window. Probe:

define step(v) as:
    loss is v
    return when is loss
r is 0
for v in [100.0, 50.0, 25.0, 12.5, 6.25, 3.1]:
    r is step of v
print of r     # -> 1     (after SIX calls)

And with report of loss over a cleanly converging 12-value sequence: "equilibrium" — which does not mean "converged", it means "I have seen one sample". train.eigs's loss (:274-283) is exactly this shape.

Requirement: print each binding's assign count next to its verdict, so when=1 reads as "this call" rather than "settled". A trajectory word with no sample count behind it is the silent-tolerance failure of this feature.

2. Dict-held state is invisible — including all config

A dict field is never observed (bookkeeping is gated on the named env path, eigenscript.c:1765; vm.c:939 calls the field "untracked" — see #655). train.eigs's hp is a dict, so the config cannot appear in the dump. Worth stating so nobody files "the dump is missing hp" as a bug: it is not a gap in the dumper, it is what the observer tracks.

3. The bindings that answer the question are already long-lived

For train.eigs the useful ones are module-scope and genuinely have trajectories:

binding	answers
step_count (:491)	monotonically increasing — "hung or just slow", the original question
epsilon (:490)	anneals — position in the schedule
best_avg_score (:492)	improving or flat

Trap: loss_sum (:500) is module-level but reset to 0.0 every episode (:640), so its trajectory is a sawtooth and the observer reads it as oscillating. Listing it unqualified would actively mislead.

So v1's honest framing is "is this process progressing", not "is loss converging". The former is what was actually wanted at ten minutes of silence, and step_count nails it today. The latter needs the program to hoist a running metric to a long-lived binding first — see the follow-up below.

Done when

• kill -USR1 <pid> on a long-running program dumps module-scope + live-frame bindings to stderr and the program continues correctly
• Each row carries its assign count; a when=1 binding is visibly distinguishable from a settled one
• Dumping mid-run is safe under ASan + the leak gate, and under make asan with a program spawning tasks
• No measurable cost on bench_dmg_shape.eigs with the feature idle (n=5 medians), or it is gated behind EIGS_OBSERVE=1
• A test drives the signal against a live process and asserts the dump's shape

Later (not v1)

• --observer-socket for structured queries ("why is loss") against a running process — the out-of-process REPL for interrogatives.
• Periodic auto-report every N seconds, so a long job self-reports unprompted.

Follow-up worth its own issue

A cross-call trajectory is currently inexpressible: a function-local's observer resets per call, so nothing in the language tracks "loss across training steps" without the author manually hoisting an accumulator to module scope. That is the forcing-function question behind this one — should the language make cross-call trajectories a first-class thing, and if so keyed to what, since the binding is by construction fresh each call?

Why this fits

report of answers "has this settled?" from inside, at places the author anticipated. This answers it at places nobody anticipated — which is where debugging actually happens, and where agents (which cannot attach debuggers) live entirely on what the runtime volunteers.