callouts: build + bake m-stdlib YDB call-out .so's into the engine image#4
Open
rafael5 wants to merge 2 commits into
Open
callouts: build + bake m-stdlib YDB call-out .so's into the engine image#4rafael5 wants to merge 2 commits into
rafael5 wants to merge 2 commits into
Conversation
m-stdlib's three optional modules (STDCOMPRESS / STDCRYPTO / STDHTTP) reach native code on YottaDB via $&pkg.fn -> a .so registered through a .xc descriptor. The running container had no such libraries, so the optional suites couldn't run — and with a .xc descriptor present but its .so missing, STDCOMPRESSTST hung the engine. This bakes the libraries in and wires them. How: - New multi-stage Dockerfile `callout-builder` stage: installs gcc + the -dev headers (zlib1g-dev, libzstd-dev, libssl-dev, libcurl4-openssl-dev), compiles m-stdlib's src/callouts/*.c via tools/build-callouts.sh. The toolchain lives ONLY in this stage; the final image carries just the .so's and the already-present runtime libs (libz 1.3 / libzstd 1.5.5 / libcrypto 3 / libcurl 4.8.0) — staying minimal. - Final stage: COPY the compiled .so's -> /opt/stdlib/lib, the .xc descriptors -> /opt/stdlib/xc, and export STDLIB_LIB + ydb_xc_stdcompress / _stdcrypto / _stdhttp via /etc/profile.d/stdlib-callouts.sh (sourced by `bash -lc`, the m-cli DockerEngine transport). A build-time assert fails the image if any of the three .so's is absent — enforcing the hang-guard invariant (never a descriptor without its library). - Makefile: `callouts-stage` copies the C sources + .xc from m-stdlib into the build context (gitignored docker/_callouts/ — single-sourced in m-stdlib, never vendored here); `up` depends on it. New `test-optional` runs the four callout-backed suites in byte mode. - dist/: lifecycle.json gains the callout env vars + a stdlib_callouts block + the test-optional target; verified_on bumped to 2026-05-30 (re-verified with the new image). Verification (via `m test`, no hand docker exec): - make smoke: healthy (mumps ok, OCI labels, healthcheck, mte status). - make test-optional (YDB byte mode, --chset m): 4 suites, 151 assertions, 0 failed, no hang — STDCRYPTOTST 23, STDCRYPTODOCTST 1, STDCOMPRESSTST 59, STDHTTPTST 68. The deployed .so's resolve the Stage-A STDCOMPRESS hang (which was the missing-.so case). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2.07
The stdhttp $ZF call-out was dark on 0.1.0 — every $&stdhttp.* ref failed
to compile/resolve on YottaDB r2.07, so STDS3 egress could not leave YDB.
Root-caused live (see m-stdlib discoveries.md G-HTTP-YDB): two latent bugs
in the YDB http_perform path (never exercised before — the package never
loaded and IRIS uses %Net.HttpRequest):
1. underscore in the M-side call-out labels in std_http.xc (_=concat →
%YDB-E-EXPR/ZCRTENOTF) — fixed in m-stdlib (perform:/available:).
2. http_perform missing the leading `int argc` ABI param → SIGSEGV —
fixed in m-stdlib.
The image fix is purely a rebake: `make callouts-stage` single-sources the
corrected http.c + std_http.xc from m-stdlib; nothing in the bake logic
changes. This commit:
- Dockerfile: ydb-version LABEL r2.02 -> r2.07 (matches the baked release).
- dist/m-test-engine.json: default_tag 0.1.0 -> 0.2.0; ydb_version
r2.02 -> r2.07; verified_on -> 2026-06-20.
- tools/gen-skill.py: refresh the drift example; skill regenerated (r2.07).
- docs/m-engine-tracker.md: 0.2.0 rebake entry.
Verified on the rebaked image (both engines): make smoke ✓; in-image
$&stdhttp.available()=1; STDHTTPTST 67/67 + STDS3MINIOTST 11/11 on YDB
(unchanged on IRIS); v-stdlib VSLS3E2ETST 6/6 YDB; crypto/compress/sigv4
unregressed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
m-stdlib's three optional modules (STDCOMPRESS / STDCRYPTO / STDHTTP) reach
native code on YottaDB via $&pkg.fn -> a .so registered through a .xc
descriptor. The running container had no such libraries, so the optional
suites couldn't run — and with a .xc descriptor present but its .so missing,
STDCOMPRESSTST hung the engine. This bakes the libraries in and wires them.
How:
callout-builderstage: installs gcc + the-dev headers (zlib1g-dev, libzstd-dev, libssl-dev, libcurl4-openssl-dev),
compiles m-stdlib's src/callouts/*.c via tools/build-callouts.sh. The
toolchain lives ONLY in this stage; the final image carries just the .so's
and the already-present runtime libs (libz 1.3 / libzstd 1.5.5 / libcrypto 3
/ libcurl 4.8.0) — staying minimal.
-> /opt/stdlib/xc, and export STDLIB_LIB + ydb_xc_stdcompress / _stdcrypto /
_stdhttp via /etc/profile.d/stdlib-callouts.sh (sourced by
bash -lc, them-cli DockerEngine transport). A build-time assert fails the image if any of
the three .so's is absent — enforcing the hang-guard invariant (never a
descriptor without its library).
callouts-stagecopies the C sources + .xc from m-stdlib into thebuild context (gitignored docker/_callouts/ — single-sourced in m-stdlib,
never vendored here);
updepends on it. Newtest-optionalruns the fourcallout-backed suites in byte mode.
the test-optional target; verified_on bumped to 2026-05-30 (re-verified with
the new image).
Verification (via
m test, no hand docker exec):0 failed, no hang — STDCRYPTOTST 23, STDCRYPTODOCTST 1, STDCOMPRESSTST 59,
STDHTTPTST 68. The deployed .so's resolve the Stage-A STDCOMPRESS hang
(which was the missing-.so case).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com