test(longhaul): add kill-operator-pod and kill-primary-pod chaos operations

[WentingWu666666]

Long-haul chaos operations (kill-operator-pod + kill-primary-pod)

Adds the first two Chaos / HA operations from the long-haul design (docs/designs/long-haul-test-design.md → Operations) to the driver introduced in #405, and reworks the outage-budget model they depend on. Scope: fault-inject the control plane and the database primary on the canary cluster, and assert — over a multi-day run — that the data plane keeps serving and the cluster reconverges to its declared topology.

What this PR adds

Two new operations registered with the scheduler:

Operation	Fault	Asserts
kill-operator-pod	Deletes the operator pod	The data plane keeps serving throughout (control-plane restart must not drop writes); the operator Deployment returns to Available.
kill-primary-pod	Deletes the CNPG primary pod	CNPG promotes a standby (automatic failover); writes resume and the cluster returns to full topology. Requires HA (instancesPerNode>=2).

The continuous workload verifier independently catches any data loss caused by a failover.

Outage policy: duration-based, writer-count-independent

The chaos ops needed a principled disruption budget, so OutagePolicy moved from a raw write-failure count to a wall-clock write-outage duration:

• OutagePolicy.MaxWriteOutage time.Duration replaces AllowedWriteFailures int64. The journal converts the observed failure count into an estimated outage using the workload's aggregate write rate (WriteFailures / WritesPerSecond), so budgets no longer scale with LONGHAUL_NUM_WRITERS. The rate is computed once at startup from NumWriters and the fixed writer cadence (workload.AggregateWriteRate) and stamped onto each disruption window.
• Two orthogonal guardrails, both fail independently (ExceededPolicy trips if either is exceeded):
  • MaxWriteOutage — client-visible write availability (data plane).
  • MustRecoverWithin — the managed cluster's return to full declared topology (all pods Ready, CR Ready). Because a failed op is only logged (not scored), this is the sole mechanism that fails the run when the cluster never reconverges — e.g. a failover where writes resume fast but a replacement standby never rejoins.
• Shared budgets to prevent drift:
  • journal.NoOutagePolicy (~300ms cushion) for operations that keep the write path up throughout: kill-operator-pod and the pre-existing scale ops (which only add/remove a standby). One fully-failed write tick maps to ~100ms of estimated outage regardless of writer count, so the cushion absorbs unrelated noise without tolerating a real outage.
  • journal.PrimaryHandoverPolicy (30s) shared by kill-primary-pod and upgrade-documentdb: both interrupt writes for exactly one primary handover (an ungraceful failover vs. a graceful switchover). A rolling upgrade's longer whole-topology restart is bounded by MustRecoverWithin, not the write-outage budget.

Design notes

• Operator discovery. The operator Deployment name is a chart-pinned literal (documentdb-operator, singleton install), so it is a safe constant; the namespace is install-overridable, exposed as LONGHAUL_OPERATOR_NAMESPACE (default documentdb-operator). There is no back-reference from the DocumentDB CR to the operator, so cluster-derived discovery isn't viable. kill-operator-pod builds its pod selector from the Deployment's own spec.selector.matchLabels rather than a hardcoded label (the operator label is release-name dependent).
• Primary discovery. kill-primary-pod reads the CNPG Cluster.Status.CurrentPrimary (CNPG cluster name == DocumentDB CR name) to target the primary pod, then waits for steady state.
• HA guard. kill-primary-pod skips (does not fail) when instancesPerNode<2 — killing the sole instance is guaranteed downtime with no failover target. Skips don't consume the scheduler cooldown.

Changes

• operations/kill_operator.go, operations/kill_primary.go (+ unit tests): the two chaos ops.
• journal/policy.go, journal/journal.go: duration-based OutagePolicy, WritesPerSecond per window + EstimatedWriteOutage, SetWriteRate, and the shared NoOutagePolicy / PrimaryHandoverPolicy helpers.
• workload/writer.go: AggregateWriteRate(numWriters).
• monitor/{health,k8sclient}.go: GetPrimaryInstance / DeletePod on ClusterClient; CNPG scheme registration.
• operations/scale.go, operations/upgrade.go: adopt the shared policies (scale ops were previously over-lenient count budgets).
• config/config.go: LONGHAUL_OPERATOR_NAMESPACE + validation.
• report/report.go: disruption-window table gains an Est. Write Outage column.
• cmd/longhaul/main.go: supply the write rate to the journal; register both ops.
• go.mod: promote cloudnative-pg to a direct dependency.

Follow-up dependency

The driver ServiceAccount needs, in deploy/rbac.yaml (added by the CI/CD PR): get/list on clusters.postgresql.cnpg.io + delete on pods (kill-primary), and get on deployments + get/list/delete on pods in the operator namespace (kill-operator). Documented in test/longhaul/README.md.

Verification

cd test/longhaul
go build ./...   # clean
go vet ./...     # clean
go test ./...    # all packages pass

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: test, documentation, dependencies, enhancement
Project fields suggested: Component test · Priority P2 · Effort XL · Status In Progress
Confidence: 0.88 (mixed)

Reasoning

component from path globs (test, docs, dependencies); effort from diff stats (746+78 LOC, 23 files); LLM: Adds new chaos fault-injection operations (kill-operator-pod, kill-primary-pod) to the long-haul test suite, touching multiple files across the test infrastructure including health monitoring, k8s client, config, and operation implementations.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.