Force-apply topology on reconcile and add a per-cluster sync endpoint (#395)

[bocharov]

Problem

Fixes #395. Kvrocks cluster nodes do not gossip — the controller is the sole source of topology and must push a complete, identical view to every node. Today a node can silently diverge and never recover:

• The probe loop re-pushes CLUSTERX SETNODES only when a node's epoch is behind the stored version. A node that has drifted at an equal epoch (a dropped push, a partial view, a stale/phantom node entry) is never repaired.
• SETNODES is sent without force, so even a re-push at an equal version is a no-op on the server (the version gate rejects a lower version and no-ops an equal one). The only prior recovery was CLUSTER RESET, which requires an empty DB and so is unusable once a node holds data. Controller's cluster_nodes push silently drops when a data node is unreachable, leaving cluster meta stale forever #395 reports exactly this: divergent cluster_nodes snapshots that never converge, with no operator-facing way to force a re-push.

Changes

1. SyncClusterInfo(ctx, cluster, force) — when force is set, send the SETNODES force flag so the topology applies unconditionally. Because SETNODES replaces the node's whole topology, a forced push also clears stale/phantom entries without a CLUSTER RESET and without data loss. Wrapped in a bounded retry-with-backoff (a push dropped by a transient blip is otherwise not retried until the next tick).

2. Divergence detection at an equal epoch — the reconcile loop parses cluster_known_nodes and cluster_slots_ok from CLUSTER INFO and force-pushes a node whose peer count or slot coverage disagrees with the desired topology, not only one whose epoch lags. Coverage is compared against the desired covered-slot count (not a hardcoded 16384), so a cluster that is intentionally mid-scale is not seen as drifted; a node that omits these fields falls back to epoch-only reconcile. The existing "node is ahead" adopt branch is preserved.

3. POST /namespaces/{namespace}/clusters/{cluster}/sync — force-push the stored topology to every node and report per-node results. This is the operator-facing re-push Controller's cluster_nodes push silently drops when a data node is unreachable, leaving cluster meta stale forever #395 asks for. It performs no CLUSTER RESET, so it is safe to run on a populated cluster.

4. Reject blank/port-less node addresses in CheckNewNodes (the create/add-node boundary) and Shard.ToSlotsString (serialization), so a half-resolved address fails loudly instead of registering a phantom, unreachable node.

Safety

The controller is the single writer of topology (failover routes through it), so "newest complete view wins" holds. Force fires only when a node's version is not greater than the store's; a node legitimately ahead keeps the existing adopt-from-node branch. Verified against Apache Kvrocks 2.16.0 that CLUSTERX SETNODES <str> <ver> force bypasses the equal-version no-op.

Tests

make test passing. New unit tests cover the CLUSTER INFO parsing (incl. sentinel/malformed values), the divergence check, address validation, the blank-address guard, and the /sync handler. A behavioral test asserts the reconcile fix by exercising the real probe path.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.30435% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.77%. Comparing base (6c56470) to head (807123c).
⚠️ Report is 109 commits behind head on unstable.

Files with missing lines	Patch %	Lines
store/cluster_node.go	89.65%	3 Missing ⚠️
controller/cluster.go	91.30%	0 Missing and 2 partials ⚠️
server/api/shard.go	0.00%	0 Missing and 2 partials ⚠️
server/route.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable     #400      +/-   ##
============================================
+ Coverage     43.38%   50.77%   +7.39%     
============================================
  Files            37       45       +8     
  Lines          2971     4169    +1198     
============================================
+ Hits           1289     2117     +828     
- Misses         1544     1832     +288     
- Partials        138      220      +82     

Flag	Coverage Δ
unittests	50.77% <91.30%> (+7.39%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bocharov]

Addressed the Codecov patch-coverage feedback in 807123c. Added unit tests for the previously-uncovered branches:

• SyncClusterInfo bounded retry/backoff — retry exhaustion and context-cancel during backoff.
• Cluster.SyncToNodes — force-push to every node and node-level error propagation.
• Reconcile probe error paths — node restoring-from-backup is skipped, an unreachable node increments its failure count, and an uninitialized (CLUSTERDOWN) node is force-pushed rather than counted as a failure.
• The /sync handler's cluster-not-found path.

go test/go vet/-race all clean locally. The workflow run is pending approval (fork PR) — would appreciate a committer approving it so CI + Codecov can re-evaluate. Thanks!

[git-hulk]

@bocharov Thanks for your contribution, a few comments inline.

[bocharov]

@git-hulk thanks for the review — all three addressed in 1e9a79c:

1. validateAddr extracted and shared by ToSlotsString + CheckNewNodes.
2. SyncClusterInfo now takes a SyncPolicy{MaxRetries, RetryDelay, Force} (with DefaultSyncPolicy/ForceSyncPolicy helpers).
3. topologyDiverged is now (*store.ClusterInfo).TopologyDiverged.

go build/go vet/gofmt clean; the reconcile, retry/backoff, topology-divergence, and address-validation unit tests pass locally (the docker-backed integration tests need the compose stack, which CI runs).

[git-hulk]

@bocharov Can help to resolve the CI failure

[bocharov]

Force-pushed to squash the three commits into one; the tree is byte-identical (no code change). The previous red was a flaky TestClusterFailover (shared 7770/7771 nodes — test-isolation, MOVED/topology-version mismatch), unrelated to this diff. Could a maintainer approve the pending run? Thanks!