From 64b8c1f687942fab446e7a9773b75b0e1621d4af Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Fri, 24 Jul 2026 23:01:07 -0500 Subject: [PATCH 1/3] =?UTF-8?q?release:=20cut=20v0.33.0=20=E2=80=94=20the?= =?UTF-8?q?=20silent-wrong-answer=20release?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two threads. The first is a sweep of the highest-yield failure class: a plausible-but-wrong value returned at rc=0, where nothing warns and nothing crashes. regex_match deleted every capture after a non-participating group (#629); json_decode truncated long number tokens and parsed a wrong-magnitude prefix, off by 50 orders of magnitude (#628); four stdlib math functions returned confidently wrong numbers (#638-#641); a module's top-level assignment rebound its IMPORTER's global instead of creating module state (#673); a comprehension variable and a same-named local split into two bindings, so program text after a statement changed that statement's result (#633/#642); and the observer indicted itself twice — report_value called a converging sequence diverging (#674), while the learning-rate gate measured 100% "stable" across 2,597 steps with the loss demonstrably still falling. The second is the primitive layer, every item requested by a consumer that had hit the wall rather than speculated: four DEFLATE codecs behind `make zlib` (#684 — .xlsx and .gz readable from script), a tape-captured clock_unix (#683), and four C-backed list operations (#543/#544). Plus `kill -USR1` live observer dumps (#660) and a test runner that no longer trusts its own binary, liveness, or arithmetic (#681/#648/#654 — its hand-synced tallies were under-reporting by 169 asserts). Carries the sites the cut ritual needs: VERSION, the CHANGELOG section retitled from [Unreleased] (with the ~20 landed changes that never got an entry written in — the drift this cut found), the docs/llms.txt version stamp, CLAUDE.md's "Latest release" line, and the ROADMAP sync. doc_drift_check rules 1/3/4/5 pass; rule 2 fires locally until the tag exists, which is the expected pre-tag state (CI's depth-1 clone has no tags, so the guard skips there — verified empirically at the v0.32.0 cut). Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 244 +++++++++++++++++++++++++++++++++++++++++++++++++- CLAUDE.md | 46 +++++----- ROADMAP.md | 11 ++- VERSION | 2 +- docs/llms.txt | 2 +- 5 files changed, 277 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8df6db8..537b6fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,9 +2,79 @@ All notable changes to EigenScript are documented here. -## [Unreleased] +## [0.33.0] - 2026-07-24 + +The **silent-wrong-answer release**, plus the primitives the app fleet forced. +Two threads run through it. The first is a sweep of the highest-yield failure +class — a plausible-but-wrong value returned at rc=0, where nothing warns and +nothing crashes: `regex_match` deleting every capture after a non-participating +group (#629), `json_decode` silently truncating long numbers by 50 orders of +magnitude (#628), four stdlib math functions returning confidently wrong numbers +(#638-#641), a module's top-level assignment rebinding its *importer's* global +(#673), a comprehension variable and a same-named local splitting into two +bindings so program text after a statement changed that statement's result +(#633/#642), and the observer itself calling a converging sequence `diverging` +(#674) while its learning-rate gate certified a still-moving system as settled. +The second is the primitive layer: four DEFLATE codecs (#684), a tape-captured +`clock_unix` (#683), and four C-backed list operations (#543/#544) — every one +of them requested by a consumer that had hit the wall (`.xlsx` import, `TODAY()`, +interpreter-speed list scans) rather than speculated. `kill -USR1` now makes a +running process report how each of its bindings is progressing (#660), and the +test runner learned to distrust itself: it fingerprints the binary mid-run, +bounds every block against a runaway, and derives its own tally from block +output after the hand-synced literals were caught under-reporting by 169 +asserts (#654). ### Added +- **DEFLATE codecs: `inflate` / `deflate` / `zlib_inflate` / `zlib_deflate` + (#684).** Four thin wrappers over the system zlib, two dual pairs. `inflate`/ + `deflate` are raw DEFLATE (windowBits -15) — the ZIP member format, so + `.xlsx`/`.ods` entries are readable from script; `zlib_inflate`/`zlib_deflate` + are zlib-wrapped, and the inflate side auto-detects zlib *and* gzip headers + (windowBits 15+32), which is what makes a plain `.gz` readable. Byte + representation mirrors `read_bytes`/`write_bytes` exactly: a list of ints + 0-255 or a buffer in, a fresh list of ints 0-255 out. Corrupt or truncated + input raises a catchable `value` error rather than crashing, and decompressed + output is capped at 256 MiB — a zip-bomb bound matching the `sandbox_run` + default budget. The zero-dependency minimal build is preserved through the + existing `EIGENSCRIPT_EXT_*` gate (default OFF, `make zlib` opts in, same + pattern as `make http`): compiled without zlib the four names stay + *registered*, so scripts can feature-detect with try/catch and the + fail-closed sandbox allowlist can name real builtins, but every call raises + "compiled without zlib support". All four are pure bytes-to-bytes and join + `SANDBOX_ALLOW`. `tests/test_inflate.eigs` (22 checks) covers round-trip + identity for both pairs including empty and 8 KB inputs, plus fixed known + vectors; CI runs them for real on a `make zlib` leg (#697) instead of the + probe-gated stub. +- **`clock_unix` — a wall clock that still replays (#683).** Seconds since the + Unix epoch, routed through the trace-tape nondeterminism seam + (`eigs_trace_record_nondet` / `eigs_replay_take`), so a program that reads + the time replays byte-for-byte from its tape. The standing tape-first rule + applied to a new nondeterministic source: no untraced clock. +- **`list_index_of` and `list_contains` (#543); `list_insert_at` and + `list_slice` (#544).** C-backed value search and range operations over lists, + using the VM's structural `value_equals` — the same comparison `==` uses — so + membership and index lookup no longer route a per-element closure call + through the interpreter. `list_insert_at of [list, i, v]` is the dual of + `list_remove_at`, `list_slice of [list, start, end]` the dual of `copy_into`, + so reorder and range-copy stop being hand-rolled interpreter-speed loops. +- **`kill -USR1 ` dumps the live observer state (#660).** Ask a running + process how it is progressing: one row per live binding to stderr — + `name | value | when= | entropy= | dH= | trajectory` — at the + next loop safepoint. The `sigaction` handler (SA_RESTART, installed by the + CLI) only sets a `volatile sig_atomic_t`; the dump itself runs in normal + thread context at the loop-cap per-iteration check, so the idle cost is one + flag test — unmeasurable on `bench_dmg_shape` (166ms -> 163ms, n=5 medians), + hence no `EIGS_OBSERVE` gate. The first thread to see the flag dumps (atomic + test-and-clear), covering module scope plus that thread's live call frame; + the module walk takes the existing #607 `g_module_env_lock` under MT, no new + locking scheme. Every row carries its assign count, so a fresh per-call + binding (`when=1`) reads differently from a settled one — the silent-tolerance + failure mode the issue was about. `env->assign_counts` alone cannot back that + (the SET_LOCAL fast path never bumps it), so the dump prints + `max(assign_counts, obs_age)`; unobserved slots say `unobserved` rather than + borrowing a window-less verdict. Driven against a live process by + `tests/test_sigusr1_dump.sh` with observable barriers rather than sleeps. - **`native_train_step` batched causal-mask training path (~10.75x faster).** The transformer trainer (model extension, used by iLambdaAi) re-ran a full forward+backward over the growing prefix for *every* target position in a @@ -84,7 +154,6 @@ All notable changes to EigenScript are documented here. button geometry is now defined once (`_dialog_btn_rect`) instead of being recomputed by render and click separately. -### Added - **`eigen_generate` accepts an optional 4th argument: `top_p` (nucleus sampling).** Keeps the smallest set of tokens whose cumulative probability reaches `p`, so the candidate set adapts to the model's confidence, rather @@ -115,6 +184,26 @@ All notable changes to EigenScript are documented here. untrained model must score ln(vocab_size) — measured 7.004 against a predicted 7.011 at vocab=1109 — which simultaneously catches a missing max-subtraction (overflow to inf), a mis-indexed target, and sign errors. +- **`build_corpus`: lossless slot-mode identifier encoding (#675) and exact + tokens for small integer literals (#678).** Both are optional args that stop + the corpus builder from erasing the information a model would need. Frequency + mode promoted the top-N identifiers and collapsed every other one onto a + single fallback token — 45.7% of all identifier occurrences at top_n=256 on + the 2026-07 ecosystem corpus. That is not a quality loss but an + *expressiveness* loss: `total is total + items[i]` encodes as ` is + + []`, so "assign the variable I just read" — the + property separating well-formed output from correct output — is not statable, + and no amount of training recovers it. Slot mode (6th arg) encodes local + names losslessly by position. Separately, every numeric literal tokenized to + one `TOK_NUM`, so `[1,2,3]` and `[0,0,0]` were indistinguishable; the 7th arg + `int_count=N` gives exact tokens to integers `[0, N)`. That erasure was + costly twice over: a specification written as assertions carried no + information (`(f of 4) == 1` and `(f of 3) == 0` arrive identical, so a model + graded on satisfying them has nothing to learn), and it manufactured + degenerate repetition — 5.3% of the corpus sits inside a short literal cycle + repeated 3+ times, essentially all of it routed through the collapsed literal + tokens. Integers 0-31 are 76% of numeric-literal occurrences and 7.7% of the + whole corpus. ### Changed - **`+` on two lists now names the fix (#680).** `[1,2] + [3,4]` reported @@ -146,8 +235,159 @@ All notable changes to EigenScript are documented here. release threshold (default 1e-3). This is the observer's own failure mode — a meter blind to slow motion certifying a moving system as converged — found by pointing the instrument at itself. +- **The test runner no longer trusts its own binary, its own liveness, or its + own arithmetic.** Three integrity gaps closed in one pass, all of the shape + "the suite reports success it did not earn": + - **Mid-run rebuild detection (#681).** Every build variant writes to the + same `src/eigenscript` path, so rebuilding while a suite runs silently + swaps the binary underneath it. The runner now records cksum+size+mtime at + start, re-checks at section boundaries and at the end, and aborts loudly + on mismatch. Section `[99d]` swaps the binary from a background step and + asserts the guard fires. + - **Runaway guards (#648, #616, #651, #656).** `check_eigs_suite` ran each + file with no timeout, so one infinite loop hung the whole suite until the + CI hard limit instead of reporting a failure. A shared `$EIGS_TMO` + (timeout -> gtimeout -> none, mirroring `[97]`) now wraps it and reports + rc=124 as a named failure that lets subsequent blocks run; the budget is a + deliberately generous 180s (`EIGS_TEST_TIMEOUT`) — a backstop, not a + latency assertion, so it never fires on a slow-but-working test on the + N3350. `[97]`'s own too-tight `timeout 60` is fixed in the same pass + (`invariant_weak.eigs` is ~60.5s standalone under ASan). Section `[99c]` + drives a genuinely non-terminating fixture through the real + `check_eigs_suite` to prove the guard fires *and* is tallied exactly. The + last unguarded interpreter call — `[60]` Terminal Builtins — got the same + shape plus `= 0`. POSIX sets `rm_so = -1` for a + group that did not participate — an unmatched optional `(x)?` — so the first + such group terminated emission and dropped every later capture: + `regex_match of ["ab", "(x)?(a)(b)"]` returned `["ab"]` instead of the full + match plus three groups. Independently, groups past 15 vanished silently (a + 17-group pattern returned 16 elements). Both broke the docs/BUILTINS.md + invariant that group *n* sits at index *n*. The array is now sized from the + compiled pattern (`re.re_nsub + 1`) through the overflow-guarded + `xmalloc_array` — `re_nsub` is attacker-controlled through the pattern but + POSIX-bounded — that count is passed to `regexec`, the allocation is freed on + every path, and the participation test moved into the loop body, so a + non-participating group emits null and the ones after it still arrive. +- **`json_decode` silently truncated long number tokens (#628).** A number + token was capped at 63 chars while the scanner kept going, so the dropped + tail vanished and `atof` parsed a wrong-magnitude prefix at rc=0: a + zero-padded `1.<60 zeros>e50` decoded to `1` — off by 50 orders of magnitude + — and a 70-digit integer to 1e62 instead of 1e69, diverging from the + runtime's own lexer. The `numbuf` is gone: the scanner already validates the + decimal JSON number grammar, so the token span is recorded and `strtod` runs + directly over it, correct at any length. The existing early returns for + malformed tails (`1e`, `1e+`, `1.`) are unchanged. `strtod` could only + overrun the scanned token via a C99 hex-float `0x` prefix — JSON has no hex + numbers, so the scanner stops at `x` — and that is guarded by verifying + `endptr` landed on the scanned end, re-parsing the isolated span on any + mismatch so the value stays tied to the grammar the scanner accepted. +- **`pad_left` / `pad_right` hung forever on an empty pad char (#637).** Both + looped until the string reached `width`, growing it by the pad char each + pass; an empty pad char never grows it, so `pad_left of ["x", 5, ""]` never + terminated. The degenerate case now returns the string unchanged — the honest + reading of "pad with nothing" — matching the empty-arg guards `capitalize` + and `replace_all` already carry in the same file. Contract documented in + docs/STDLIB.md, with regression tests plus the previously-missing `pad_left` + normal-path coverage. +- **Four stdlib functions returned confidently wrong numbers (#638, #639, + #640, #641).** Each produced a plausible-but-wrong value at rc=0 — the + highest-yield failure class — and each fix ships with a regression test + verified to fail on the pre-fix library. + - `punnett_square`: Aa x Aa read 1:1:1:1 instead of the canonical 1:2:1. The + heterozygous boxes "Aa" and "aA" are the same genotype but were keyed + distinctly; genotypes are now canonicalized so they fold. + - `power_iteration`: returned |lambda|, not lambda — it read the eigenvalue + off `|w| = |A v|`, which drops the sign, so `diag(-5, 1)` reported 5, a + number that is not an eigenvalue of that matrix at all. Now the Rayleigh + quotient over the pre-update vector. Positive spectra are unaffected. + - `correlation`: with mismatched lengths it paired `min(len)` elements but + centered them on the **full**-list means, so unpaired tail elements + corrupted the covariance. Centers on the paired prefix now. + - Two degenerate-input contracts: `factorial` of a negative returned 1 (via + `n <= 1`) though it is undefined — it now returns the 0 sentinel, + impossible for a real factorial and matching `combinations`' convention — + and `time_of_flight` could return a **negative** time when both roots of + `x = v0 t + ½ a t²` are negative; it returns the smallest non-negative + root, or the same 0 sentinel it already used for the no-real-root case. +- **An imported module's top-level assignment rebound the importer's global + (#673).** A module's top-level `name is expr` walked `mod_env`'s parent chain + and rebound a same-named global in the **importer's** scope instead of + creating module state, so the module's own functions — and any later use in + the importer — resolved to the wrong binding. This is the top-level + counterpart of #373's function-write boundary, one level up: `import`'s own + compile of a module's top-level statements now forces `OP_SET_NAME_LOCAL`, + binding strictly within `mod_env`, gated by a compile-time flag set only + around that call. `load_file` is untouched — it shares #373's + function-boundary flag but never sets the new one — so its documented + "top-level statements execute directly in the current scope" contract still + holds, and docs/SPEC.md's module write boundary now states both rules + explicitly. +- **A comprehension/`catch` name and a same-named assignment split into two + bindings (#633, #642).** A comprehension variable and a `catch` name bind by + name into the function env via `OP_SET_NAME_LOCAL`, but a plain `x is ...` of + the same name is slot-promoted to `OP_SET_LOCAL`. The two never met: the env + held one binding while the slot kept a stale value, so a slot read returned + the pre-comprehension value, and #642's half corrupted the comprehension's + own result — the body resolved `i` to the shadowed slot every iteration, so + `[-2, -2, -2]` came back instead of `[2, 4, 6]`, rc=0 with every element + wrong. Worse, `c->captured` was the only thing suppressing slot promotion, so + adding any closure or interrogative that merely *mentioned* the name — + anywhere in the body, never executed — flipped which binding an earlier read + saw: program text after a statement changed that statement's result. + Listcomp vars and catch names are now collected into a per-scope env-bound + name set (the mechanism that already keeps for-loop vars off the slot path), + routing a same-named plain assignment to `OP_SET_FN_NAME_LOCAL`. Thanks to + @Nitjsefnie for the fix. Its tests could not see #642 — all four read the + name *after* the comprehension — so SS20/SS20b now pin the body's own result + and SS21 is the control in the other direction, since a fix that de-promotes + too eagerly would break the ordinary comprehension. +- **Undefined behavior in the float-to-`long long` cast in `value_to_string` + (#695).** The `VAL_NUM` case tested `n == (long long)n && fabs(n) < 2^53`, + evaluating the cast in the left conjunct *before* the magnitude guard; for + |n| >= 2^63 that conversion is UB (float-cast-overflow, flagged by + UBSan/CodeQL). The conjuncts are swapped so the range check short-circuits + and guards the cast; in-range behavior is unchanged. The observer dump's + string scan got the same treatment — range-check before dereference, and a + bounded truncation marker. +- **`test_proc_stream` 8a raced the child instead of waiting for it (#604).** A + 100k spin loop only *hoped* the child had been scheduled; under load it had + not, and the write returned 5. It now waits for child exit — `proc_wait` + waitpid()s the pid and leaves the parent's pipe fds open, so waiting before + writing pins the intended property: EPIPE from a dead child, not EBADF from a + closed handle. +- **Three stale documented contracts a reader would have coded against + (#645).** `type`'s doc listed `"null"` as a return value, which + `builtin_type` never produces — it returns `"none"` — so `if type of x == + "null":`, written straight from the doc, is silently and permanently false + (SPEC.md's CI-gated example had pinned the truth while BUILTINS.md + contradicted it); the four omitted types are now enumerated too. `buf_get`'s + doc promised "0 on out-of-bounds" long after #502 deliberately made it raise + `index_range`, so an unguarded scan loop written on that promise gets a hard + abort. And SPEC.md's error-kind table, introduced as a **closed** set, + omitted `deadlock` — which SPEC.md itself tells you to catch 350 lines later. - **LSP `import ` completion offered 34 of 75 stdlib modules (#692).** The completion list was a hand-maintained array in `src/eigenlsp.c` that had fallen 41 modules behind `lib/` — every `ui_w_*` widget module, the whole diff --git a/CLAUDE.md b/CLAUDE.md index 42462d1..77970e0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -146,28 +146,30 @@ bash tools/embed_stack_soak.sh # embed REPL soak inside a 64 KiB stack rlimit ( ## Current state & where the detail lives -- **Latest release: v0.32.0** (2026-07-16) — the desktop-shell release: - lib/ui's gap series (#561–#577, #594), surfaced by DeslanStudio's port — - the toolkit's first real consumer — and closed. The toolkit now owns what - the shell hand-rolled: `handle_key` (#563, keyboard through `dispatch` - instead of trapped inside `app_loop`, priority pinned in one place), - `request_quit` (#564, ending the loop from a callback instead of `exit of - N` skipping `gfx_close`), clickable combobox dropdown items (#577 — - hit-test before closing, select by the click's own y, and exempt popups - from double-click detection), self-measuring `label` w/h + `auto_size` - (#561), `grid.owns_cells` + a configurable row-label gutter (#572), - piano-key release wherever the pointer lands (#570), hover feedback on - checkbox/toggle (#594), and **`menu_bar`** (#565 — owning its popup's - z-order via the `_render_popups` overlay pass + `_find_open_popup_at`, - since rendering above the tree is only half of z-order). Three of the ten - filed issues had already been fixed and were closed with probe output, not - re-fixed. (v0.31.0, 2026-07-14: DAW audio-I/O + live-synthesis — bulk - audio-I/O kernels #602/#603, the live audio-streaming primitive - `audio_stream_*` #612, `waveform_view` selection markers #610/#611. - v0.30.0, 2026-07-13: debugging-and-distribution + first-consumer — - `--step` tape stepper #418, tape format v2 #539, error carets/LSP ranges - #407, `--bundle` #413, entropy-walk cycle detection #571, gfx audio - capture #579 + buffer playback #578, the lib/ui input trio #567–#569.) +- **Latest release: v0.33.0** (2026-07-24) — the silent-wrong-answer release, + plus the primitives the app fleet forced. A sweep of the plausible-but-wrong- + at-rc=0 class: `regex_match` dropping every capture after a non-participating + group (#629), `json_decode` truncating long numbers by 50 orders of magnitude + (#628), four stdlib math functions returning confidently wrong values + (#638–#641), a module's top-level assignment rebinding its *importer's* + global (#673), a comprehension var and a same-named local splitting into two + bindings so later program text changed an earlier statement's result + (#633/#642), `report_value` calling a converging sequence `diverging` (#674), + and an observer LR gate that measured 100% "stable" while the loss was still + falling. Primitives, each requested by a consumer that hit the wall: four + DEFLATE codecs behind `make zlib` (#684 — `.xlsx`/`.gz` readable from + script), tape-captured `clock_unix` (#683), and C-backed `list_index_of` / + `list_contains` / `list_insert_at` / `list_slice` (#543/#544). Plus + `kill -USR1` live observer dumps (#660) and a test runner that fingerprints + its own binary mid-run, bounds every block against a runaway, and derives its + tally from block output after the hand-synced literals were caught + under-reporting by 169 asserts (#654). (v0.32.0, 2026-07-16: the + desktop-shell release — lib/ui's gap series #561–#577/#594, surfaced by + DeslanStudio's port and closed, incl. `menu_bar` #565 and `handle_key` #563. + v0.31.0, 2026-07-14: DAW audio-I/O + live-synthesis — bulk audio-I/O kernels + #602/#603, `audio_stream_*` #612, `waveform_view` markers #610/#611. + v0.30.0, 2026-07-13: debugging-and-distribution — `--step` tape stepper #418, + tape format v2 #539, error carets/LSP ranges #407, `--bundle` #413.) Unreleased work on `main`: see CHANGELOG.md `[Unreleased]`. Full version history: **CHANGELOG.md** (don't re-narrate it here — tools/doc_drift_check.sh diff --git a/ROADMAP.md b/ROADMAP.md index 4ddb23b..19598e9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,8 +1,8 @@ # Roadmap -Current version: **0.32.0** +Current version: **0.33.0** -Recently shipped (0.28.0 → 0.32.0, 2026-07-08 → 07-16): +Recently shipped (0.28.0 → 0.33.0, 2026-07-08 → 07-24): - **0.29.0** — the strategic headline landed: the **deterministic cooperative task layer on the tape** (#408, seeded scheduler, zero @@ -22,6 +22,13 @@ Recently shipped (0.28.0 → 0.32.0, 2026-07-08 → 07-16): #594) surfaced by DeslanStudio's port and closed, including `menu_bar` (#565), `handle_key`/`request_quit` (#563/#564), self-measuring `label` (#561), and `dialog` children + `file_dialog` (#575). +- **0.33.0** — the silent-wrong-answer release plus the primitives the app + fleet forced: the plausible-but-wrong-at-rc=0 sweep (`regex_match` #629, + `json_decode` #628, four stdlib math bugs #638–#641, import top-level scope + #673, the listcomp/local double binding #633/#642, `report_value` #674), the + DEFLATE codecs behind `make zlib` (#684), tape-captured `clock_unix` (#683), + four C-backed list operations (#543/#544), `kill -USR1` live observer dumps + (#660), and the test-runner integrity pass (#681/#648/#654). Full per-version detail lives in [CHANGELOG.md](CHANGELOG.md) — this file is forward-looking. diff --git a/VERSION b/VERSION index 9eb2aa3..be386c9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.32.0 +0.33.0 diff --git a/docs/llms.txt b/docs/llms.txt index cc052a2..cf148cb 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -1,6 +1,6 @@ # EigenScript — single-file reference for language models -> EigenScript v0.32.0. A small dynamically-typed language with one distinctive +> EigenScript v0.33.0. A small dynamically-typed language with one distinctive > feature: an **observer** tracks every value's entropy/trend on each > assignment, so convergence, stability, and oscillation are first-class. This > file is the whole surface you need to generate correct `.eigs`. Models From 650c8da15d0127bb86ee0900d0aaccc2a46cf683 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Fri, 24 Jul 2026 23:23:52 -0500 Subject: [PATCH 2/3] changelog: flag #630's operand widening as an ABI break for external bytecode producers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The original fix said 'no opcode number or on-disk format changed' — true, and the wrong frame: the operand WIDTH is the bytecode ABI for anything that builds a chunk and hands it to vm_run_bytecode. The pre-bump consumer sweep found ouroboros producing empty output for every behavioral-parity program (bootstrap fixed point still green — it compares bytes, it does not execute) and iLambdaAi's grading ladder unable to award its top grade, both silently at rc=0. Also flags #629's regex_match change, which is observable to anyone who consumed the truncated shape (EigenRegex's compat shim mirrored it on purpose). Gate for the class filed as #704. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 537b6fa..2278ee1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,15 @@ bounds every block against a runaway, and derives its own tally from block output after the hand-synced literals were caught under-reporting by 169 asserts (#654). +**Upgrade notes.** Two changes are observable from outside. (1) `OP_LINE`'s +operand widened from 16 to 32 bits (#630), which breaks any external producer +of bytecode — see the entry below; `ouroboros` and `iLambdaAi` needed a +downstream emitter fix. (2) `regex_match` now emits null for a +non-participating capture group and continues, where it previously truncated +the result there (#629) — correct per the documented "group n at index n" +invariant, but code that consumed the truncated shape (such as EigenRegex's +`compat_match` shim, which mirrored it deliberately) needs updating. + ### Added - **DEFLATE codecs: `inflate` / `deflate` / `zlib_inflate` / `zlib_deflate` (#684).** Four thin wrappers over the system zlib, two dual pairs. `inflate`/ @@ -463,8 +472,20 @@ asserts (#654). assignments 65536 lines apart collapsed onto one stamp and `what is x at L` returned the wrong value at rc=0; runtime-error/stack-trace lines and the trace-tape line context were also silently wrong past line 65535 (the - regime generated programs reach). Widened to 32-bit; no opcode number or - on-disk format changed. + regime generated programs reach). Widened to 32-bit; no opcode number + changed. **Breaking for external bytecode producers**, which the original + fix did not call out: any program that hands the VM a chunk it built itself + — via `vm_run_bytecode` — must now emit a 4-byte `OP_LINE` operand. A 2-byte + emission leaves the VM reading the next instruction's first two bytes as the + high half of the line number and resuming misaligned, so the chunk executes + as garbage **silently, at rc=0**. Caught by the pre-bump consumer sweep, not + by CI: `ouroboros`'s self-hosted codegen (every behavioral-parity program + produced empty output while the bootstrap fixed point still passed) and + `iLambdaAi`'s grading ladder (every "runs" rung scored 1 instead of 2 — a + grader that quietly lost the ability to award its top grade). Both are fixed + downstream by emitting u32. Nothing upstream gates this class: consumers pin + a release, so a mid-release operand-width change cannot fail their CI until + the bump. - **`relu`/`leaky_relu`/`softmax`/`log_softmax` returned a silent null on a scalar (#632).** `tensor_to_flat` reports 0 dims for a `VAL_NUM`, and the four did `if (!flat) return make_null()`, so the null poisoned downstream From b9d475241b84abcb1c62fbab7239c17d3eb72ed3 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Fri, 24 Jul 2026 23:29:23 -0500 Subject: [PATCH 3/3] tests: pin OP_LINE's 32-bit operand in the external-producer canary (#704) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The canary for vm_run_bytecode already existed and is thorough — 34 checks covering opcodes, the verifier, the sandbox, and #348 — but it emitted no OP_LINE at all. So when #630 widened that operand from 16 to 32 bits, nothing here failed, and the break surfaced only at the next consumer bump: ouroboros producing empty output for all 44 parity programs, iLambdaAi's grading ladder unable to award its top rung, both at rc=0 with no error. Four checks close the hole: - a well-formed 4-byte OP_LINE keeps the stream aligned (2 + 3 == 5) - the pre-#630 3-byte shape must NOT still evaluate to 5 — whatever it does (verifier rejection or misaligned garbage), it is no longer the producer's intended arithmetic - a line number past 65535 survives the operand, which is the regime #630 exists for - a truncated operand is rejected rather than read off the end Validated red-then-green rather than assumed: against a v0.32.0 worktree build the first two fail ("expected 5, got null"); against main they pass 34/34. So this file now fails in the SAME PR as any future opcode-number or operand-width change, instead of at bump time in someone else's repo. The header comment said "16-bit operands are little-endian", which #630 made false; it now states the exception and names this file as the ABI contract for external producers. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_vm_run_bytecode.eigs | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/tests/test_vm_run_bytecode.eigs b/tests/test_vm_run_bytecode.eigs index 8b07e89..20ec7a7 100644 --- a/tests/test_vm_run_bytecode.eigs +++ b/tests/test_vm_run_bytecode.eigs @@ -1,7 +1,14 @@ # vm_run_bytecode — the self-hosting bootstrap bridge. Assemble a chunk from # EigenScript values (code bytes + constant pool) and run it on the C VM. -# Opcodes/operands are exactly what compile_ast emits (16-bit operands are -# little-endian); this is what an EigenScript-written compiler will produce. +# Opcodes/operands are exactly what compile_ast emits (operands are +# little-endian, 16-bit except OP_LINE's, which is 32-bit since #630); this is +# what an EigenScript-written compiler will produce. +# +# This file is the ABI contract for external bytecode producers (ouroboros' +# self-hosted codegen, iLambdaAi's vendored copy of it). Changing an opcode's +# NUMBER or its OPERAND WIDTH breaks every one of them silently — a misaligned +# chunk runs as garbage at rc=0 — and their CI cannot see it, because they pin +# a release. So any such change belongs here, in the same PR (#704). load_file of "lib/test.eigs" @@ -153,4 +160,25 @@ catch es348: assert_eq of [sl348, 1, "SET_LOCAL out-of-range slot raises (#348)"] assert_eq of [vm_run_bytecode of [[GET_LOCAL,255,255, RETURN], []], null, "GET_LOCAL out-of-range reads null (missing-param semantics)"] +# --- #630/#704: OP_LINE's operand is 32-bit, and that width IS the ABI --- +# #630 widened this operand from 16 to 32 bits. No opcode number changed, so +# nothing here failed — this file emitted no OP_LINE at all — and the break +# only surfaced at the next consumer bump, where ouroboros produced empty +# output for every program and iLambdaAi's grader lost its top rung. Both were +# silent: rc=0, no error, just a chunk executing misaligned garbage. +LINE is 68 +# well-formed: OP_LINE then 2 + 3. A 4-byte operand keeps the stream +# aligned, so this is 5. Emit three bytes instead and ADD lands on the wrong +# byte — which is exactly what every external producer did on the v0.33.0 bump. +assert_eq of [vm_run_bytecode of [[LINE,7,0,0,0, CONST,0,0, CONST,1,0, ADD, RETURN], [2, 3]], 5, "OP_LINE 32-bit operand keeps the stream aligned"] +# the pre-#630 shape — OP_LINE + 2 operand bytes — must NOT still evaluate to 5. +# It is either verifier-rejected or misaligned garbage; either way the contract +# is that it is no longer the arithmetic the producer intended. +old_shape is vm_run_bytecode of [[LINE,7,0, CONST,0,0, CONST,1,0, ADD, RETURN], [2, 3]] +assert of [old_shape != 5, "pre-#630 16-bit OP_LINE emission no longer silently works"] +# a line number past 65535 — the regime #630 exists for — survives the operand. +assert_eq of [vm_run_bytecode of [[LINE,161,134,1,0, CONST,0,0, RETURN], [99]], 99, "OP_LINE carries a line past 65535 (#630)"] +# truncated 32-bit operand is rejected by the verifier, not read off the end. +assert_eq of [vm_run_bytecode of [[LINE,7,0,0], []], null, "verify: truncated OP_LINE operand rejected"] + test_summary of null