build(kernel): 2-mode kernel build recipe + tagged CI job#395
Draft
mani-mathur-arch wants to merge 3 commits into
Draft
build(kernel): 2-mode kernel build recipe + tagged CI job#395mani-mathur-arch wants to merge 3 commits into
mani-mathur-arch wants to merge 3 commits into
Conversation
de9c139 to
38c283f
Compare
This was referenced Jul 12, 2026
Add a reproducible build recipe for the opt-in SEA-via-kernel backend and a CI job that exercises it, so the databricks_kernel-tagged path is built and unit-tested in CI instead of being dead code. - KERNEL_REV pins the kernel commit (PR #163 head) to build against. - cgo.go drops the machine-specific link line; per-platform cgo_<os>.go files carry ${SRCDIR}-relative LDFLAGS and force static linking (-l:...a). - build/kernel-lib.sh builds the kernel .a from source at KERNEL_REV (cargo --no-default-features --features tls-rustls), with a KERNEL_LOCAL_A override; Makefile adds kernel-lib / build-kernel / test-kernel and a kernel-lib-download stub for the future published-.a path. - go.yml gains build-and-test-kernel (CGO + rust + cargo-via-JFrog); the pure-Go CGO_ENABLED=0 jobs are untouched. Built artifacts are gitignored. Co-authored-by: Isaac
- build-and-test-kernel: drop the "no warehouse creds → live e2e/parity self-skip" note (goes stale once creds are added to the job). - kernel-token step: retirement trigger is the repo going public, not a published .a — the source clone needs auth regardless of the download path, and the source-build option stays alongside the prebuilt .a. Co-authored-by: Isaac
38c283f to
77e0231
Compare
…chain, job limits Addresses three build-hygiene findings from the PR-395 review: - F1 platform guard: add cgo_unsupported.go, compiled only on OS/arch combos with no LDFLAGS file (e.g. linux/arm64, darwin/amd64). The Makefile host== target guard passes there and builds a host .a, but the link then fails with an opaque "undefined reference to kernel_*". A build-time constant fails earlier, at compile, naming the supported targets. - F2 reproducible .a: pin rustc via rust-toolchain.toml (governs the kernel build under build/kernel-src/; kernel repo pins none), match the CI Rust step to it, pass cargo --locked (build against the kernel's Cargo.lock), and fold rust-toolchain.toml into both kernel cache keys so a toolchain bump busts the stale archive/build-tree caches. - F6 job limits: add timeout-minutes 30 and a per-ref concurrency group with cancel-in-progress to build-and-test-kernel, so a hung cargo can't park a protected-runner slot and repeated pushes don't stack heavy builds. Verified: pure-Go CGO_ENABLED=0 build stays clean; cgo_unsupported.go is excluded on linux/amd64 + darwin/arm64 and compiled on the unsupported combos; go.yml is valid YAML; the recipe script passes a shell syntax check. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a reproducible build recipe for the opt-in SEA-via-kernel backend and a CI job that exercises it, so the
databricks_kernel-tagged path is built and unit-tested in CI instead of being dead code. This is the distribution follow-up that lets #393 leave draft.KERNEL_REVpins the kernel commit to build against (currently PR 163 head in the kernel, which carries the statement canceller the driver links; re-pin to the merge SHA once 163 lands).cgo.godrops the machine-specific link line; per-platformcgo_<os>.gofiles carry${SRCDIR}-relative LDFLAGS and force static linking (-l:…a). linux/amd64 is real; darwin/windows are documented stubs pending native runners. Acgo_unsupported.goguard fails at compile time (naming the supported targets) on any other OS/arch, instead of an opaqueundefined referenceat link.build/kernel-lib.shbuilds the kernel.afrom source atKERNEL_REV(cargo --release --locked --no-default-features --features tls-rustls), with a rev-stamped cache short-circuit, aKERNEL_LOCAL_Aprebuilt-archive override, and a fail-loud guard against source cross-builds. The Rust toolchain is pinned viarust-toolchain.toml(it governs the checkout underbuild/kernel-src/), so the archive is reproducible under a fixedKERNEL_REV.Makefileaddskernel-lib/build-kernel/test-kerneland akernel-lib-downloadstub for the future published-.apath.go.ymlgainsbuild-and-test-kernel(CGO + rust + cargo-via-JFrog), withtimeout-minutesand a per-refconcurrencygroup so the heavy source build can't park a protected-runner slot or stack redundant runs; the pure-GoCGO_ENABLED=0jobs are untouched. Built artifacts are gitignored.Multi-OS support comes via native per-OS runners (host == target) or staging a prebuilt
.a, not cross-compiling from source — matching the settled distribution design.Known items from review (deferred / pre-merge)
Tracked here rather than as code comments, since these are sequencing/scope notes that would go stale in the source.
Must resolve before merge:
push:trigger.go.yml'sbranches: [main, mani/sea-kernel-build-recipe]exists only to run CI on this stacked branch (its PR base is the feature branch, notmain). It must be removed before merge somaindoesn't carry a dead personal-branch trigger that would mint anINTEGRATION_TEST_APPtoken + clone the private kernel on any recreation of that branch.KERNEL_REVto a merged commit. It currently points at 163's PR-head SHA (9b90406), reachable only viarefs/pull/163/head— GC-able if that PR force-pushes/deletes, which would break cold-cache CI. Re-pin to the 163 merge SHA once it lands, sequenced with the kernel merge. (Merge order overall: feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393 → main first, then this PR, since these files live in the kernel package that only exists in feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393.)Deferred (out of scope for this PR):
kernel_e2e_test.go,SELECT 1→ scan) is env-gated and self-skips withoutDATABRICKS_HOST/DATABRICKS_HTTP_PATH/DATABRICKS_TOKEN, which this job doesn't set. It passes locally against staging (90 pass / 84 skip). Wiring those three secrets onto the job (un-skipping the e2e test) is deferred to the CI-dispatch / creds follow-up.cgo_darwin.go/cgo_windows.goLDFLAGS are never compiled. M0 is linux; per-OS runners are the multi-OS follow-up.Testing
make kernel-libbuilds the.a(reproducible sha256) from a clean clone, a cache-restored tree, and a leftover-source tree.make test-kernellinks the archive with CGO and passes thedatabricks_kernel-tagged unit tests (ok …/internal/backend/kernel).CGO_ENABLED=0 go build ./...still succeeds — default pure-Go build unaffected.GOOS≠ host) fails loud.cgo_unsupported.goguard verified: excluded on linux/amd64 + darwin/arm64 (kernel builds normally), compiled on the unsupported combos (clear compile error).Co-authored-by: Isaac