Skip to content

ci: dispatch the SEA-via-kernel integration leg behind kernel labels + kernel CLOSE_STATEMENT telemetry & auth-rejection sentinel#403

Open
mani-mathur-arch wants to merge 3 commits into
mani/sea-kernel-nightly-e2efrom
mani/sea-kernel-c2-dispatch
Open

ci: dispatch the SEA-via-kernel integration leg behind kernel labels + kernel CLOSE_STATEMENT telemetry & auth-rejection sentinel#403
mani-mathur-arch wants to merge 3 commits into
mani/sea-kernel-nightly-e2efrom
mani/sea-kernel-c2-dispatch

Conversation

@mani-mathur-arch

@mani-mathur-arch mani-mathur-arch commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Context for reviewers

Two changes for the SEA-via-kernel backend (the alternative backend that talks to Databricks SQL through the Rust databricks-sql-kernel static lib over a C ABI, behind the databricks_kernel build tag):

  1. CI: extend the databricks-driver-test dispatch so the Go integration suite can run against the SEA-via-kernel backend, selected by label.
  2. Kernel backend: record CLOSE_STATEMENT telemetry from kernelRows.Close(), and make the unsupported-authenticator rejection programmatically detectable via the ErrNotSupportedByKernel sentinel.

The two are unrelated in subject but small and both review-driven; they share this PR because the telemetry/sentinel commit was folded onto this branch. Reviewable independently — the CI change touches only .github/workflows/, the kernel change only internal/backend/kernel/ + kernel_config*.go.

Stack position (top of the stack):

#399  OAuth / namespace / metric-view / params / types / query-id   (base)
  └─ #400  richer TLS options
       └─ #401  kernel logging → driver log level
            └─ #402  nightly E2E workflow
                 └─ #403  CI dispatch label + CLOSE_STATEMENT telemetry   ← THIS PR

1. CI: kernel integration-test dispatch

Extend the databricks-driver-test dispatch so the Go integration suite can run against the SEA-via-kernel backend, selected by label — the analogue of the existing Thrift labels, not a copy.

Label Backend Proxy mode
integration-test Thrift replay
integration-test-full Thrift passthrough
integration-test-kernel sea (kernel) passthrough
  • The label resolves both proxy_mode and a new go_mode (thrift vs sea).
  • go_mode rides in the repository_dispatch client_payload. The receiver in driver-test reads it to decide whether to build the kernel static lib and run the databricks_kernel-tagged leg. Thrift labels send go_mode: thrift and are unchanged.
  • Only one kernel label, and it's passthrough: there are no committed Go-sea proxy recordings yet, so a sea+replay label would have nothing to replay and produce a red, misleading run. A sea replay label can be added later once recordings are captured.
  • The new label is added to the on-new-commit label-drop list and the skip-stub guidance.
  • The required merge-queue gate stays Thrift-only (go_mode pinned to thrift): the kernel leg is previewable on demand but not a required gate until the SEA backend ships in a released driver.

Two-repo change

This is the driver-side half. The receiver-side half (read go_mode, sea-gated kernel build prerequisites, thread DATABRICKS_GO_MODE, guard sea+replay) is a separate PR in databricks-driver-test. The runner + harness already handle DATABRICKS_GO_MODE=sea end-to-end. On the receiver side the review gate caught (and fixed before merge) two real defects worth knowing about: a backend-aware RECORDING_NAME (databricks-go-${go_mode}) so a sea run can't clobber/serve the committed thrift recordings, and a narrowed git-token insteadOf scope (exact databricks-sql-kernel repo, not the whole databricks/ org) so the kernel App-token isn't applied to unrelated org fetches.


2. Kernel backend: CLOSE_STATEMENT telemetry + auth-rejection sentinel

Two review-driven, Go-only fixes on the kernel backend.

CLOSE_STATEMENT telemetry

conn records CLOSE_STATEMENT / close latency / statement success-or-failure only when the rows' OnClose callback fires (the Thrift path fires it). kernelRows.Close() never called it, so kernel queries emitted no close telemetry — including failures — a production observability blind spot.

  • Close() now times the teardown and fires OnClose(latencyMs, chunkCount, iterationErr, nil). closeErr is nil because the kernel teardown (C stream/statement close) has no fallible close RPC to report, unlike Thrift's CloseOperation.
  • Next() is split into a thin wrapper over next() that records the first non-EOF error as iterationErr (io.EOF is normal drain, not a failure), which Close() passes to the callback — mirroring the Thrift path's iterationErr.
  • The callback is armed only after newKernelRows finishes constructing successfully, so the cleanup Close() on a schema-fetch/import failure does not record a falsely-successful CLOSE_STATEMENT (matching the Thrift NewRows, which sets the close callback just before returning).
  • Idempotent: the existing closed guard means a double-Close fires OnClose exactly once.

Auth-rejection sentinel

resolveKernelAuth's unsupported-authenticator branch (token-provider / external / federated) returned a bare errors.New, so fallback logic ("kernel can't honor this auth → use Thrift") could only fragile substring-match. It now wraps ErrNotSupportedByKernel via %w so callers can errors.Is it, matching the other kernel-config rejections.

Deliberately not wrapped: the "requires a personal access token" (empty-token) error — that is missing-required-config, not an unsupported feature; wrapping it would wrongly signal "fall back to Thrift" for a forgotten token.


Testing

  • Workflow YAML validated; label resolution dry-run: integration-test→thrift/replay, integration-test-full→thrift/passthrough, integration-test-kernel→sea/passthrough.
  • TestKernelRowsCloseFiresOnClose (tagged): success path (nil errs), iterationErr propagation, idempotent double-close, nil-callbacks-safe, construction-failure-no-success-close.
  • TestValidateKernelConfig: strengthened the non-PAT authenticator rejection to assert errors.Is(err, ErrNotSupportedByKernel).
  • Default CGO_ENABLED=0 + tagged databricks_kernel: build + vet + test — 24 packages ok, 0 fail each.

Stacking

Stacked on #402 (mani/sea-kernel-nightly-e2e).

This pull request and its description were written by Isaac.

@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from 5f940a0 to 90122c9 Compare July 14, 2026 17:01
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-nightly-e2e branch from 2a43d24 to ebedfc7 Compare July 14, 2026 18:43
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from 90122c9 to 2a9715b Compare July 14, 2026 18:43
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-nightly-e2e branch from ebedfc7 to 23de162 Compare July 14, 2026 18:49
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from 2a9715b to 21a18d7 Compare July 14, 2026 18:49
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-nightly-e2e branch from 23de162 to fb098a1 Compare July 14, 2026 18:53
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from 21a18d7 to c9ff45d Compare July 14, 2026 18:53
@mani-mathur-arch mani-mathur-arch changed the title ci: dispatch the SEA-via-kernel integration leg behind kernel labels ci: dispatch the SEA-via-kernel integration leg behind kernel labels + kernel CLOSE_STATEMENT telemetry & auth-rejection sentinel Jul 15, 2026
Comment thread internal/backend/kernel/kernel_test.go Outdated
Comment thread internal/backend/kernel/rows.go Outdated
@mani-mathur-arch mani-mathur-arch marked this pull request as ready for review July 15, 2026 05:18
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from d49b653 to 1d357ba Compare July 15, 2026 05:37
@mani-mathur-arch mani-mathur-arch added the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Jul 15, 2026
@github-actions

Copy link
Copy Markdown

Go integration tests triggered (sea / passthrough). View workflow runs.

@mani-mathur-arch mani-mathur-arch added integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) and removed integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) labels Jul 15, 2026
@github-actions

Copy link
Copy Markdown

Go integration tests triggered (sea / passthrough). View workflow runs.

@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-nightly-e2e branch from 6376422 to 4512646 Compare July 15, 2026 08:46
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from 1d357ba to 2b09947 Compare July 15, 2026 08:51
@github-actions github-actions Bot removed the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Jul 15, 2026
@github-actions

Copy link
Copy Markdown

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test-kernel were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: 2b09947

@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-nightly-e2e branch from 4512646 to 091accc Compare July 15, 2026 09:20
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from 2b09947 to a3d9c92 Compare July 15, 2026 09:22
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-nightly-e2e branch from 091accc to ac112d4 Compare July 15, 2026 09:49
Extend the driver-test dispatch so the Go integration suite can run against the
SEA-via-kernel backend, selected by label — the analogue of the existing Thrift
labels, not a copy. Adds one kernel-namespaced label:

  - integration-test-kernel  → sea backend, passthrough (real warehouse)

The label resolves both proxy_mode and a new go_mode (thrift vs sea), and go_mode
rides in the repository_dispatch client_payload. databricks-driver-test reads it to
decide whether to build the kernel static lib and run the tagged (databricks_kernel)
leg; the Thrift labels send go_mode=thrift and are unchanged. The new label is added
to the on-new-commit label-drop list and the skip-stub guidance.

The kernel label is passthrough (not replay): the sea leg has no committed
recordings to replay yet, so there is no sea replay label until those are captured.
The required merge-queue gate stays Thrift-only (go_mode pinned to thrift): the
kernel leg is previewable on demand but not a required gate until the SEA backend
ships in a released driver.

Co-authored-by: Isaac
Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
…h rejection

Two review-driven fixes on the kernel backend, both Go-only.

CLOSE_STATEMENT telemetry: kernelRows.Close() now fires the OnClose callback, so
kernel queries record close latency / statement success-or-failure the same way the
Thrift path does (conn gates that recording on OnClose firing). Previously it never
fired, so kernel traffic emitted no close telemetry — including failures — a
production observability blind spot. Next() is split into a thin wrapper that
records the first non-EOF error as iterationErr (io.EOF is normal drain), which
Close() passes to the callback; closeErr is nil since the kernel teardown has no
fallible close RPC. The callback is armed only after newKernelRows finishes
constructing (mirroring the Thrift NewRows), so a schema-fetch/import failure's
cleanup Close() does not record a falsely-successful CLOSE_STATEMENT.

Auth rejection sentinel: the unsupported-authenticator branch of resolveKernelAuth
wrapped a bare errors.New, so fallback logic could only substring-match. It now
wraps ErrNotSupportedByKernel via %w so callers can errors.Is it, matching the other
kernel config rejections. The missing-personal-access-token error stays a plain
error (missing-required-config, not an unsupported feature — it must not signal
"fall back to Thrift").

Tests: TestKernelRowsCloseFiresOnClose (success, iterationErr propagation, idempotent
double-close, nil-callbacks, construction-failure-no-success-close); strengthened the
non-PAT authenticator rejection to assert errors.Is(ErrNotSupportedByKernel).

Co-authored-by: Isaac
Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
…x Thrift comment

Addresses two review comments on the CLOSE_STATEMENT telemetry change:

- The "construction-failure Close must not fire a success OnClose" subtest was a
  no-op: `fired` was never assignable to true, so it duplicated the nil-callbacks
  case and asserted nothing. Replaced with a test that drives the real
  newKernelRows cleanup path — a nil result stream makes
  kernel_result_stream_get_schema return a defined InvalidArgument error (the
  kernel null-checks the handle, never UB), so newKernelRows takes its r.Close()
  cleanup branch and returns an error, and the supplied OnClose must not fire.
  Verified by mutation: arming the callback before the schema import makes the
  test fail as intended.

- Reworded the newKernelRows comment that claimed the deferred callback assignment
  matches Thrift's NewRows "just before returning" — Thrift actually assigns
  closeCallback during construction. Dropped the inaccurate comparison and state
  the invariant directly.

Co-authored-by: Isaac
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-c2-dispatch branch from a3d9c92 to cf06659 Compare July 15, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant