ci(longhaul): add build/deploy/monitor workflows + lint & functional smoke gates

[WentingWu666666]

Summary

Builds on PR #405 (long-haul driver core, merged).

Adds the GitHub Actions plumbing + Kubernetes manifests to build, deploy,
monitor, and auto-upgrade the long-haul test driver on the dedicated
AKS cluster, plus two CI gates (static lint + functional smoke) that keep the
long-haul workflows and driver honest on every PR.

Note: The plan originally split this into two PRs (PR-3 workflows + PR-4 upgrade). Since the upgrade logic is contained entirely in one file (longhaul-monitor.yaml) and all the Go-side dependencies (operations/upgrade.go, K8sClusterClient.UpgradeDocumentDB, schemaVersion: "auto" in setup.yaml) already shipped in PR #405, splitting added review overhead with no benefit. Consolidated into this PR — long-haul story now lands in 3 PRs instead of 5.

What this PR adds

Workflows (.github/workflows/)

Workflow	Trigger	Purpose
longhaul-image-build.yml	push to main (paths: test/longhaul/**), workflow_dispatch	Build test/longhaul/Dockerfile, push to GHCR with :sha-<short> (immutable) + :main tags
longhaul-deploy.yml	workflow_run on successful build (auto-deploys :sha-<short>), workflow_dispatch (manual rollback)	kubectl apply Deployment manifest, set image, wait for rollout
longhaul-monitor.yaml	hourly cron + workflow_dispatch	(1) Poll Deployment health, report ConfigMap freshness (≤2h), test result ≠ FAIL. (2) Auto-upgrade operator via Helm if a newer GHCR tag is available, with post-upgrade verification + CRD re-apply. (3) Publish the latest DocumentDB GHCR tag into the longhaul-versions ConfigMap; the driver performs the in-band DocumentDB upgrade as a load-aware operation so continuous writers/verifiers can catch any data-integrity regressions.
lint-longhaul-workflows.yml	pull_request (paths: long-haul workflows + test/longhaul/** + test/shared/**), workflow_dispatch	Static gate. actionlint + yamllint on the long-haul workflows, go vet/go build/go test on the driver module, kubeconform + Helm-template smoke on the manifests. Catches YAML/lint/compile regressions before they reach the cluster.
longhaul-smoke.yml	pull_request (paths: test/longhaul/**, test/shared/**, the build workflow, .github/actions/**), workflow_dispatch	Functional gate. Reuses test-build-and-package.yml to build operator + sidecar from PR source (documentdb/gateway stay on public images via the registry probe), stands up kind + cert-manager + a real DocumentDB through the shared setup-test-environment action, then runs the actual long-haul driver (real deploy/*.yaml manifests, only bounded via ConfigMap overrides) and asserts the driver exits 0 and writes longhaul-report result=PASS.

Manifests (test/longhaul/deploy/)

• deployment.yaml — single-replica Deployment + tunable ConfigMap; image fields templated (__OWNER__/__IMAGE_TAG__) for the deploy workflow.
• rbac.yaml — namespace-scoped ServiceAccount/Role/RoleBinding (pods, documentdb.io/dbs, configmaps) + ClusterRole for metrics.k8s.io.

Driver hardening (test/longhaul/workload/)

• verifier.go — stream the verification scan (cursor-based) instead of buffering the full result set, keeping the driver comfortably within its 256Mi limit under sustained load (measured ~35Mi RSS). Prevents the OOM the earlier buffered scan hit as the collection grew.

CI gates: how they complement each other

	lint-longhaul-workflows.yml	longhaul-smoke.yml
Kind	Static	Functional (live cluster)
Speed	Fast (seconds–minutes)	~20 min (build + kind bring-up)
Proves	YAML valid, Go compiles/tests, manifests schema-valid	Driver actually builds, connects to DocumentDB, runs writers+verifier, reports PASS

Setup required before the deploy/monitor workflows can run

Cluster admin one-time bootstrap (the deployer ServiceAccount is namespace-scoped by design):

1. kubectl apply -f test/longhaul/deploy/setup.yaml (already on main via PR test(longhaul): add long-haul test driver core #405; namespace + DocumentDB CR + credentials placeholder)
2. kubectl apply -f test/longhaul/deploy/rbac.yaml
3. Create longhaul-documentdb-credentials secret with key uri (see test/longhaul/README.md)
4. Issue a long-lived kubeconfig for the longhaul-test ServiceAccount, store as repo secret LONGHAUL_KUBECONFIG

Test plan

• Lint Long-Haul Workflows — green on PR (actionlint + yamllint + driver go checks + manifest schema/Helm smoke).
• Long-Haul Smoke Gate — verified green end-to-end on this PR: driver exit 0, longhaul-report result PASS, with the database images reused (not rebuilt) via registry mode. Evidence posted in a PR comment.
• Manual dry-run of build/deploy/monitor on the long-haul AKS cluster (pending LONGHAUL_KUBECONFIG secret provisioning).

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: CI/CD, enhancement
Project fields suggested: Component ci · Priority P2 · Effort L · Status In Progress
Confidence: 0.30 (deterministic)

Reasoning

component from path globs (ci); effort from diff stats (672+0 LOC, 5 files); LLM failed: Invalid response body while trying to fetch https://api.anthropic.com/v1/messages: Premature close

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

[WentingWu666666]

✅ End-to-end verification evidence

All four long-haul workflows have been exercised against a live AKS cluster (longhaul-aks, eastus2) and are green. Runs below are on my fork (public, so links are viewable).

Workflow	Result	Run
Lint Long-Haul Workflows (PR gate: lint + image-build-smoke)	✅ pass	checks on this PR
LONGHAUL – Build Test Driver Image	✅ pass	run 28875853574
LONGHAUL – Deploy Test Driver to AKS	✅ pass	run 28876113977
Long Haul Monitor (check-longhaul + upgrade-if-needed)	✅ pass	run 28876022267
DCO	✅ pass	—

What each run proves

• Build – multi-stage image builds from repo root context and pushes to GHCR (longhaul-test:sha-01e5032, :main).
• Deploy – applies the driver Deployment to AKS, waits for rollout, verifies pod health.
• Monitor – reads the driver's longhaul-report ConfigMap, validates freshness + result, and the upgrade-if-needed job reconciles the operator and publishes the desired DocumentDB version via server-side apply (the --force-conflicts fix in 01e5032 is exercised here).

Live cluster state at time of run

• DocumentDB CR: "Cluster in healthy state", documentdb-cluster-1 2/2 Running.
• Latest driver report: Result: PASS — Writes 14,735 / Ack 14,735 (100% success), 0 gaps, 0 checksum errors.

_{Note: the monitor's check-longhaul job correctly failed on a first attempt when the report was stale (>2h), demonstrating the staleness guard works; it passed once a fresh report was published.}

[WentingWu666666]

🪶 Follow-up: driver is now genuinely lightweight (256Mi, was 1Gi)

While gathering evidence I hit repeated OOMKills of the driver and initially bumped its limit to 1Gi. That was masking a symptom, not fixing the cause — a write-load driver shouldn't need 1Gi. Root-caused and fixed properly:

Root cause

workload/verifier.go:fetchDocs materialised an entire per-cycle scan window into a []WriteDocument slice before auditing it. Steady-state that window is tiny, but after a disruption pause the verifier catches up on tens of thousands of docs at once; the slice + BSON-decode buffers spike the heap and OOM against a hard 256Mi cgroup limit — even though the driver's own bookkeeping is only a few MB.

Fix (commit d032131)

• Stream the scan: extracted an incremental auditor (step/finish) and feed cursor-decoded docs through it one at a time, so verify memory is O(1) per cycle instead of O(window). auditDocs is retained as a thin wrapper, so all existing table tests are unchanged.
• Also fixed a latent bug: now surface cursor.Err() so a truncated read is reported instead of being silently mistaken for tail loss (a false positive).
• Restore 256Mi limit / 128Mi request, and add GOMEMLIMIT=200MiB as a soft GC ceiling below the hard limit (belt-and-suspenders).
• Dropped the temporary memory-floor lint guard — unnecessary once the cause is fixed.

Evidence (live AKS, streaming image sha-d032131 at 256Mi)

• Driver Running, 0 restarts, GOMEMLIMIT=200MiB applied, limit 256Mi.
• kubectl top steady at ~35Mi RSS (≈14% of the 256Mi limit) under active 5-writer load.
• All PR checks green (lint, go-test, image-build-smoke, DCO); build + deploy workflows green against the live cluster.

_{Steady-state is proven here; the disruption-catch-up spike is eliminated at the source (O(1) scan) and further guarded by GOMEMLIMIT. The post-merge AKS long-haul remains the endurance/scale proof.}

[WentingWu666666]

✅ Long-Haul Smoke Gate — verified green on this PR

The new longhaul-smoke.yml gate ran end-to-end on this PR's HEAD (050ab45) and passed:

Run: https://github.com/documentdb/documentdb-kubernetes-operator/actions/runs/28883342342 → Long-Haul Smoke job ✅

Final verdict (Assert PASS step):

Driver exit code : 0
Report result    : PASS
✅ Long-haul smoke gate passed (exit 0, report PASS).

What this proves (real, not static):

• The reusable build (test-build-and-package.yml) built operator + sidecar from this PR's source, while the database images were NOT rebuilt — the registry probe found the public documentdb/gateway images and Build DocumentDB Packages was skipped (registry mode).
• The shared setup-test-environment action stood up kind + cert-manager + the built operator + a real DocumentDB cluster (use-external-images: false, artifacts loaded into kind).
• The actual long-haul driver (the same test/longhaul/deploy/{rbac,deployment}.yaml manifests shipped for the long-haul cluster — only the image ref substituted and a few ConfigMap knobs patched for a bounded 3-minute run) connected to DocumentDB over the gateway (TLS), ran its writers + verifier, exited 0, and wrote longhaul-report with result=PASS.

Build stage (registry mode) job results:

Job	Result
Probe Released DocumentDB Images	✅ success
Build Operator Images (amd64)	✅ success
Build Sidecar Images (amd64)	✅ success
Build DocumentDB Packages	⏭️ skipped (public images reused)

This complements the static Lint Long-Haul Workflows gate with a functional check that the driver + deploy manifests actually work against a live operator build.

[Copilot]

Pull request overview

This PR adds the CI/CD and Kubernetes plumbing to build, deploy, validate, and continuously monitor the long-haul test driver, alongside a workload verifier change to reduce memory usage by streaming Mongo cursor results.

Changes:

• Add GitHub Actions workflows for long-haul image build, AKS deploy, health monitoring + auto-upgrade, PR-time linting, and a kind-based smoke gate.
• Add long-haul Deployment + RBAC manifests for running the driver on the dedicated cluster.
• Refactor the verifier’s audit path to support O(1) memory scanning via a streaming auditor.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
test/longhaul/workload/verifier.go	Refactors verification to stream cursor results through an incremental auditor, avoiding buffering large doc windows.
test/longhaul/deploy/rbac.yaml	Introduces ServiceAccount/Role(+Binding) and metrics ClusterRole(+Binding) for the long-haul driver.
test/longhaul/deploy/deployment.yaml	Adds ConfigMap-driven configuration and a single-replica Deployment for the long-haul driver.
.github/workflows/longhaul-smoke.yml	Adds a PR gate that runs the driver end-to-end on kind for a short bounded window.
.github/workflows/longhaul-monitor.yaml	Adds an hourly monitor that checks driver health, auto-upgrades the operator via Helm, and publishes the desired DocumentDB version.
.github/workflows/longhaul-image-build.yml	Adds a main-branch workflow to build/push the long-haul driver image to GHCR with immutable and rolling tags.
.github/workflows/longhaul-deploy.yml	Adds auto-deploy (post-build) and manual deploy/rollback to the AKS long-haul cluster.
.github/workflows/lint-longhaul-workflows.yml	Adds PR-time static validation (actionlint/yamllint/kubeconform/helm template), long-haul Go tests, and a no-push image build smoke test.

[WentingWu666666]

Good catch — fixed in dd49e83. Comment now reads "ServiceAccount for the long-haul test driver Deployment" (the driver runs as a Deployment, not a Job).

[WentingWu666666]

Correct — there is no PVC (the manifest mounts no volumes). Fixed in dd49e83: the comment now says the driver resumes each writer's sequence from the DocumentDB workload collection and checkpoints to the longhaul-report ConfigMap (no PVC).

[WentingWu666666]

Agreed — reframed in dd49e83: "never run two driver pods concurrently against the same DocumentDB cluster / workload collection" (the Recreate rationale is about avoiding concurrent drivers, not a PVC).

[WentingWu666666]

Fixed in dd49e83 — the LONGHAUL_RESET_DATA comment now points at the actual persistence model: restarts resume durability state from the existing DocumentDB workload collection and longhaul-report ConfigMap checkpoints, not a PVC.

[WentingWu666666]

Valid — for the :main rolling tag a node could serve a stale cached image. Fixed in dd49e83: imagePullPolicy: Always. Pull cost is negligible for a single long-lived driver and it guarantees mutable tags are refreshed.

[WentingWu666666]

Fixed in dd49e83 — step summary now says "Deploy by setting the driver Deployment image to the immutable tag."

[WentingWu666666]

Fixed in dd49e83 — header now reads "Check health: Deployment status, report staleness, test result."

[WentingWu666666]

Fixed in dd49e83 — the "Fails if" line now describes it accurately: fails if the driver Deployment is unhealthy, the report ConfigMap reports FAIL, the report is stale (>2h), or the report ConfigMap is not found.

[WentingWu666666]

Confirmed and fixed in dd49e83. charts/ and Chart.lock are gitignored, so the cloudnative-pg dependency isn't vendored — a fresh CI checkout would fail the helm upgrade. Added helm dependency update operator/documentdb-helm-chart immediately before the upgrade.

[alaye-ms]

Wouldn't the || true here make this pass even if you couldn't access the cluster?

[WentingWu666666]

Great catch — yes, || true made the gate pass even against an unreachable cluster. Fixed in dd49e83: dropped || true so the step fails fast on a bad/expired kubeconfig before the apply/rollout steps run.