Skip to content

USHIFT-6795: C2CC: Routes controller#6599

Open
pmtk wants to merge 43 commits intoopenshift:mainfrom
pmtk:c2cc/routes
Open

USHIFT-6795: C2CC: Routes controller#6599
pmtk wants to merge 43 commits intoopenshift:mainfrom
pmtk:c2cc/routes

Conversation

@pmtk
Copy link
Copy Markdown
Member

@pmtk pmtk commented Apr 28, 2026

When reviewing, skip Vendor libovsdb & knftables commit

Summary by CodeRabbit

  • New Features

    • Added Cluster-to-Cluster Connectivity (C2CC) controller integrated into startup: automated cross-cluster route management (policy & service routes), nftables masquerade-bypass rules, OVN static route sync, node SNAT-exclude tracking with reconciliation/cleanup, and a managed NetworkPolicy to allow remote pods.
  • Tests

    • Added extensive unit and integration tests, Robot Framework suites, presubmit scenario, and test assets for end-to-end connectivity, reconciliation, sanity, and cleanup.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 28, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Apr 28, 2026

@pmtk: This pull request references USHIFT-6795 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 28, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 28, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 28, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds a new Cluster-to-Cluster (C2CC) route manager to MicroShift that implements OVN NB route management, Linux policy/service routing, nftables bypass rules, node SNAT annotation management, a C2CC NetworkPolicy, service registration, unit tests, and Robot Framework end-to-end scenarios.

Changes

C2CC Route Manager (single cohesive change DAG)

Layer / File(s) Summary
Module deps
go.mod
Adds direct dependencies for OVN OVSDB client, knftables, backoff, and related transitive modules.
Service wiring & config
pkg/cmd/run.go, pkg/config/c2cc.go
Registers C2CCRouteManager with ServiceManager and adds C2CC.AllRemoteCIDRs() helper.
Top-level manager
pkg/controllers/c2cc/controller.go
Introduces exported C2CCRouteManager, constructor, Name/Dependencies, Run lifecycle: best-effort cleanup when disabled, full init when enabled, shared reconcile channel, periodic ticker, fullReconcile and cleanupAll orchestration.
OVN NB client & model
pkg/controllers/c2cc/nbdb.go, pkg/controllers/c2cc/nbdb_test.go
Implements NB connect/monitor flow with unix socket polling, exponential reconnect/backoff, DB model for logical router and static route, and tests for the model.
OVN route manager
pkg/controllers/c2cc/ovn.go, pkg/controllers/c2cc/ovn_test.go
Deterministic UUID builder, desired static-route construction from resolved remotes, list/diff transaction to create/delete Logical_Router_Static_Route rows and router refs, subscribe to NB cache events, cleanup, and tests.
Linux policy routing helpers
pkg/controllers/c2cc/policy_routes.go, pkg/controllers/c2cc/policy_routes_test.go
Provides policyRouteTable abstraction: list/compare/apply/delete kernel routes, ip rule helpers, netlink subscribe, ipFamilyOf, and tests.
Linux route manager
pkg/controllers/c2cc/routes.go, pkg/controllers/c2cc/routes_test.go
Implements linuxRouteManager: desired routes from remotes, reconcile kernel routes and ip rules in configured table/priority, cleanup, and unit tests.
Service route manager
pkg/controllers/c2cc/service_routes.go, pkg/controllers/c2cc/service_routes_test.go
Implements serviceRouteManager: management-port gateway discovery, service CIDR route reconciliation and paired ip rule reconciliation across remote/local service CIDRs, gateway computation, cleanup, and tests.
nftables manager
pkg/controllers/c2cc/nftables.go, pkg/controllers/c2cc/nftables_test.go
Manages nft table/chain rules annotated with CIDR comments, adds/removes rules to match desired CIDRs, subscribes to knftables/netlink events with debounce, cleanup, and tests including IPv4/IPv6 cases.
Node annotation manager
pkg/controllers/c2cc/annotation.go, pkg/controllers/c2cc/annotation_test.go
Manages OVN SNAT-exclude and tracking annotations on Node: deterministic sorted CIDR lists, merge semantics preserving foreign entries, reconcile/cleanup, node watch with backoff and non-blocking reconcile signals, and tests.
NetworkPolicy
pkg/controllers/c2cc/networkpolicy.go
Creates/updates/deletes a C2CC NetworkPolicy allowing ingress from remote pod CIDRs; reconciliation preserves ResourceVersion and delete ignores NotFound.
Test helpers & unit tests
pkg/controllers/c2cc/helpers_test.go, other *_test.go
Adds test helpers and many unit tests covering OVN model, annotation parsing, nftables, routes, policy helpers, and manager desired-state logic.
Robot test assets & resources
test/assets/c2cc/*, test/resources/c2cc.resource
Adds Pod/Service manifests and a Robot Framework resource file with keywords for multi-cluster SSH/oc operations, verifications for routes/rules/nft/OVN annotations/networkpolicy, and annotation-manipulation helpers.
Scenarios & suites
test/bin/scenario.sh, test/scenarios-bootc/...el98-src@c2cc.sh, test/suites/c2cc/*
Extends scenario variable population to include sibling VMs, adds presubmit to provision/configure two VMs for C2CC, and adds Robot suites: sanity, infrastructure, connectivity, reconciliation (fault-injection), and cleanup.

Sequence Diagram(s)

sequenceDiagram
    participant MS as MicroShift
    participant CM as C2CCRouteManager
    participant OVN as OVN_NB
    participant K8s as Kubernetes_API
    participant NL as Linux_Netlink
    participant NFT as nftables

    MS->>CM: Start manager (NewC2CCRouteManager / Run)
    CM->>OVN: Connect & Monitor NB (unix socket, backoff)
    CM->>K8s: Init kube client
    CM->>NL: Subscribe routes/rules events
    CM->>NFT: Subscribe nft events
    CM->>CM: Start reconcile loop (periodic + event-driven)

    alt On reconcile (tick or event)
        CM->>OVN: List/Create/Delete Logical_Router_Static_Route
        CM->>K8s: Get/Patch Node annotations, Reconcile NetworkPolicy
        CM->>NL: Reconcile kernel policy routes and ip rules
        CM->>NFT: Reconcile nftables bypass rules (add/remove)
        OVN-->>CM: state / events
        K8s-->>CM: state / events
        NL-->>CM: state / events
        NFT-->>CM: state / events
    end

    MS->>CM: Shutdown
    CM->>OVN: Cleanup routes
    CM->>NL: Cleanup routes/rules
    CM->>NFT: Cleanup nft rules
    CM->>K8s: Delete NetworkPolicy / clear annotations
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 10 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 3.57% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning Go unit tests lack meaningful failure messages on 55% of assertions, reducing debugging value in CI/CD environments. Add descriptive failure messages to all assertions, especially those validating complex state transformations and subtle logic errors.
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR uses standard Go testing and Robot Framework, not Ginkgo framework, so naming convention check is not applicable.
Microshift Test Compatibility ✅ Passed The custom check targets Ginkgo e2e tests, but this PR contains only standard Go unit tests and Robot Framework test suites. No Ginkgo e2e tests are present.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests added. PR contains only Robot Framework integration tests and Go unit tests using testing.T, which are not SNO-incompatible.
Topology-Aware Scheduling Compatibility ✅ Passed PR introduces topology-agnostic C2CC route manager controller and test assets with no scheduling constraints or node role assumptions.
Ote Binary Stdout Contract ✅ Passed PR introduces no violations of the OTE Binary Stdout Contract. All klog statements use standard Kubernetes APIs defaulting to stderr; no new process-level entry points added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR contains only Go unit tests and Robot Framework tests, no Ginkgo e2e tests. Code properly handles IPv4 and IPv6 with adaptive address family checking.
Title check ✅ Passed Title accurately describes the main change: adding a C2CC Routes controller implementation, which is the core of this comprehensive PR.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 28, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pmtk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 28, 2026
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 11

🧹 Nitpick comments (4)
pkg/controllers/c2cc/nbdb_test.go (1)

16-41: FieldTags tests are currently tautological.

These assertions only re-check values set in the same struct literal, so they won’t catch ovsdb tag/schema regressions. Please validate tag mappings directly (e.g., reflection on struct tags or model metadata checks).

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/controllers/c2cc/nbdb_test.go` around lines 16 - 41, The tests
TestLogicalRouterStaticRoute_FieldTags and TestLogicalRouter_FieldTags are
tautological because they only re-assert values set in struct literals; replace
these with reflection-based assertions that verify the actual struct field tags
(e.g., the ovsdb/model tags) on the types instead of field values. Use
reflect.TypeOf(LogicalRouterStaticRoute{}) and reflect.TypeOf(LogicalRouter{})
to inspect fields UUID, IPPrefix, Nexthop, ExternalIDs, Policy and UUID, Name,
StaticRoutes respectively, and assert that Tag.Get("ovsdb") (or the correct tag
key used by your OVSDB mapping) equals the expected tag/schema name for each
field; update the test expectations to the correct tag strings so the tests fail
on tag/schema regressions. Ensure you keep the original test names and replace
value assertions with these tag checks.
test/suites/c2cc/reconciliation.robot (1)

19-21: 30s timeout may be tight for some reconciliation scenarios.

The controller's periodic reconcile is 10s (reconcileInterval). With 5s retry, you get ~6 attempts. Should be sufficient, but event-driven reconciliation should trigger faster.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/suites/c2cc/reconciliation.robot` around lines 19 - 21, The test's
reconciliation timeout (${RECONCILE_TIMEOUT}) is likely too short for slower
environments; increase ${RECONCILE_TIMEOUT} (or adjust ${RECONCILE_RETRY}) so
the suite allows more attempts given the controller's reconcileInterval (~10s).
Update the Variables block to set a larger ${RECONCILE_TIMEOUT} (for example to
60s or more) or reduce ${RECONCILE_RETRY} gap to ensure enough reconciliation
attempts relative to reconcileInterval and event-driven triggers.
pkg/controllers/c2cc/ovn.go (1)

25-28: Consider URL-safe encoding for named UUIDs.

The replacer handles common characters but ., /, :, - may not be exhaustive for all CIDR formats. IPv6 CIDRs could introduce other characters.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/controllers/c2cc/ovn.go` around lines 25 - 28, The current buildNamedUUID
uses strings.NewReplacer to sanitize a few characters but misses other
characters (e.g., from IPv6 CIDRs); change buildNamedUUID to produce a URL-safe
deterministic token by encoding the combined prefix+suffix with a safe encoder:
e.g., compute a SHA-256 (or other stable) hash of prefix+suffix and return its
URL-safe base64 (base64.RawURLEncoding.EncodeToString(hash[:])) or, if
readability is desired, return url.PathEscape(prefix+suffix) to percent-encode
all unsafe characters; update the buildNamedUUID function to use one of these
approaches so all possible characters are handled safely.
test/resources/c2cc.resource (1)

174-180: Python inline script for JSON parsing works but is fragile.

Consider using oc get node -o jsonpath with the annotation key directly, similar to line 186. The current approach handles edge cases but adds complexity.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/resources/c2cc.resource` around lines 174 - 180, Replace the fragile
Python inline in the "Verify Node SNAT Annotation" check: instead of piping oc
JSON to python3 -c, call Oc On Cluster to run oc get node with a jsonpath that
directly extracts the annotation key
'k8s.ovn.org/node-ingress-snat-exclude-subnets' (for example using oc get node
-o
jsonpath="{.items[0].metadata.annotations['k8s.ovn.org/node-ingress-snat-exclude-subnets']}")
so ${stdout} contains the annotation value reliably; update the command that
currently uses python3 -c in the test to use this jsonpath form and keep the
subsequent Should Contain assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/controllers/c2cc/annotation.go`:
- Around line 74-97: The node watch loop currently can leak watches and spin on
reconnects; ensure each watcher is explicitly stopped and that unexpected
channel closure backs off: after successfully creating watcher (from
a.kubeClient.CoreV1().Nodes().Watch with FieldSelector
"metadata.name="+a.nodeName) defer or call watcher.Stop() when you exit that
watch scope, iterate using: ch, ok := <-watcher.ResultChan() and if !ok break
the inner loop and sleep/backoff (e.g., time.Sleep(10*time.Second)) before
continuing, and keep the existing ctx.Done() checks and reconcileCh send
(reconcileCh <- "node-annotation-changed") behavior unchanged so the loop won't
rapidly spin or leak goroutines.

In `@pkg/controllers/c2cc/nbdb.go`:
- Around line 56-58: The current loop around os.Stat(ovnNBSocketPath) treats
every error as "not found" and keeps retrying; update the check so that if
os.Stat returns an error that is not os.IsNotExist(err) you immediately return
that error (or wrap/propagate it) instead of retrying, and only continue retry
logic when os.IsNotExist(err) is true; locate the os.Stat(ovnNBSocketPath) call
in nbdb.go and change the conditional to explicitly handle non-`not exists`
errors (using os.IsNotExist) and return them promptly.

In `@pkg/controllers/c2cc/networkpolicy.go`:
- Around line 27-53: newNetworkPolicyManager currently builds an ingress rule
from ingressPeers which, if remotePodCIDRs is empty, results in an ingress rule
with an empty From (matching all sources); change the logic so you only add the
ingress rule and set networkingv1.PolicyTypeIngress when ingressPeers is
non-empty: compute ingressPeers as before, then conditionally populate
policy.Spec.Ingress and policy.Spec.PolicyTypes only when len(ingressPeers) > 0
(otherwise leave Spec.Ingress nil/empty and omit PolicyTypeIngress) so the
NetworkPolicy does not accidentally allow all ingress; update references to
ingressPeers, newNetworkPolicyManager, policy.Spec.Ingress, and PolicyTypes
accordingly.
- Around line 65-96: The reconcile/update and cleanup paths are mutating a
fixed-named NetworkPolicy without verifying ownership; before
Create/Update/Delete, fetch the existing object (existing from client.Get in the
reconcile flow and the object returned by Delete error handling in cleanup) and
check its labels (specifically "app.kubernetes.io/managed-by") against the
controller's expected manager value (compare
existing.Labels["app.kubernetes.io/managed-by"] to the value used for your
managed resources—e.g., the label on m.desired); if the label exists and does
not match, do not Update or Delete (instead log and return nil or a
non-destructive result), and when creating ensure you set the managed-by label
on m.desired so ownership is explicit; apply this logic inside the
reconcile/update code paths where client.Get, client.Create, client.Update are
used and inside networkPolicyManager.cleanup before deleting the named policy.

In `@pkg/controllers/c2cc/nftables.go`:
- Around line 161-206: The debounce goroutine is leaked because rawCh is never
closed when unsubscribing; change the returned unsubscribe to a closure that
closes rawCh and then calls sock.Close (use a sync.Once to ensure idempotence)
so the receive goroutine unblocks and the debounce goroutine can exit (refer to
rawCh, reconcileCh, the anonymous debounce goroutine, and sock.Close in your
change).

In `@pkg/controllers/c2cc/policy_routes.go`:
- Around line 41-45: The route reconcile code currently only logs
netlink.RouteReplace/RouteDel failures and always returns nil; update
reconcileRoutes to collect and propagate errors so callers
(linuxRouteManager.reconcile and serviceRouteManager.reconcile) see failures:
inside reconcileRoutes (and the block handling netlink.RouteReplace and
netlink.RouteDel) accumulate any failures (e.g., append to a slice or use
multierror) and return a non-nil error if any netlink operation fails, and
adjust callers that expect reconcileRoutes to return error so they propagate
that error instead of treating reconciliation as success.
- Around line 38-45: The equality check that skips RouteReplace only compares
gateways (actual.Gw == route.Gw) but must also compare the interface index;
update the condition that uses actualByDst/actual to require both
actual.Gw.Equal(route.Gw) AND actual.LinkIndex == route.LinkIndex (or
equivalent) before skipping; this ensures stale routes with a changed ifindex
(e.g., ovn-k8s-mp0 recreated) will be replaced by netlink.RouteReplace and
matches how routes are programmed in service_routes.go.

In `@pkg/controllers/c2cc/service_routes.go`:
- Around line 57-72: The loop is using the single IPv4 gateway returned by
getMgmtPortGateway() for every svc CIDR in m.localSvcCIDRs, which breaks
IPv6/dual‑stack; update the logic to pick a gateway that matches the address
family of each svc CIDR (or have getMgmtPortGateway return gateways per family)
before creating netlink.Route entries: for each svcCIDR in m.localSvcCIDRs,
determine whether dst is IPv4 or IPv6 and use the matching gateway IP (or leave
Gw nil if no same‑family gateway exists) when constructing the netlink.Route
(fields Dst, Gw, Table, Protocol, LinkIndex), so only same‑family next hops are
applied.

In `@test/bin/scenario.sh`:
- Line 1242: The local declaration combines with command substitution (local
var_name="${var_prefix}_$(echo "${prop}" | tr '[:lower:]' '[:upper:]')") which
can hide failures under set -e; fix by splitting into two statements: first
declare the variable (local var_name) and then perform the assignment to
var_name using the command substitution (var_name="${var_prefix}_$(echo
"${prop}" | tr '[:lower:]' '[:upper:]')"), referencing the same var_prefix and
prop variables and the var_name identifier so command-substitution failures
correctly propagate.

In `@test/scenarios-bootc/el9/presubmits/el98-src`@c2cc.sh:
- Around line 132-133: The script copies the cluster-admin kubeadmin kubeconfig
to /tmp and makes it world-readable (chmod 644), which exposes credentials;
change the flow so the kubeconfig remains owned by redhat and only readable by
its owner (mode 0600) or avoid the shared temp file entirely. Update the
run_command_on_vm call that creates /tmp/kubeconfig-b to either set ownership
and permissions (chown redhat && chmod 600) instead of chmod 644, or skip
writing to /tmp and have the scp call pull directly from the source path
(/var/lib/microshift/resources/kubeadmin/${host2_ip}/kubeconfig) so you remove
the temporary shared file; reference the run_command_on_vm invocation and the
scp line that uses "${kubeconfig_b}" when making the change.

In `@test/suites/c2cc/reconciliation.robot`:
- Around line 117-123: Replace the non-disruptive lookup with a disruptive
command to avoid a race between discovery and deletion: change the step that
sets ${handle} which currently uses "Command On Cluster" (running `nft list
chain inet ovn-kubernetes ovn-kube-pod-subnet-masq -a | grep
'c2cc-no-masq:${cidr}' | awk '/# handle/{print $NF}'`) to use "Disruptive
Command On Cluster" instead, so the handle discovery and the subsequent
"Disruptive Command On Cluster    ...    nft delete rule ... handle ${handle}"
both use the disruptive execution path (and consider adding a check for an empty
${handle} before attempting deletion).

---

Nitpick comments:
In `@pkg/controllers/c2cc/nbdb_test.go`:
- Around line 16-41: The tests TestLogicalRouterStaticRoute_FieldTags and
TestLogicalRouter_FieldTags are tautological because they only re-assert values
set in struct literals; replace these with reflection-based assertions that
verify the actual struct field tags (e.g., the ovsdb/model tags) on the types
instead of field values. Use reflect.TypeOf(LogicalRouterStaticRoute{}) and
reflect.TypeOf(LogicalRouter{}) to inspect fields UUID, IPPrefix, Nexthop,
ExternalIDs, Policy and UUID, Name, StaticRoutes respectively, and assert that
Tag.Get("ovsdb") (or the correct tag key used by your OVSDB mapping) equals the
expected tag/schema name for each field; update the test expectations to the
correct tag strings so the tests fail on tag/schema regressions. Ensure you keep
the original test names and replace value assertions with these tag checks.

In `@pkg/controllers/c2cc/ovn.go`:
- Around line 25-28: The current buildNamedUUID uses strings.NewReplacer to
sanitize a few characters but misses other characters (e.g., from IPv6 CIDRs);
change buildNamedUUID to produce a URL-safe deterministic token by encoding the
combined prefix+suffix with a safe encoder: e.g., compute a SHA-256 (or other
stable) hash of prefix+suffix and return its URL-safe base64
(base64.RawURLEncoding.EncodeToString(hash[:])) or, if readability is desired,
return url.PathEscape(prefix+suffix) to percent-encode all unsafe characters;
update the buildNamedUUID function to use one of these approaches so all
possible characters are handled safely.

In `@test/resources/c2cc.resource`:
- Around line 174-180: Replace the fragile Python inline in the "Verify Node
SNAT Annotation" check: instead of piping oc JSON to python3 -c, call Oc On
Cluster to run oc get node with a jsonpath that directly extracts the annotation
key 'k8s.ovn.org/node-ingress-snat-exclude-subnets' (for example using oc get
node -o
jsonpath="{.items[0].metadata.annotations['k8s.ovn.org/node-ingress-snat-exclude-subnets']}")
so ${stdout} contains the annotation value reliably; update the command that
currently uses python3 -c in the test to use this jsonpath form and keep the
subsequent Should Contain assertions unchanged.

In `@test/suites/c2cc/reconciliation.robot`:
- Around line 19-21: The test's reconciliation timeout (${RECONCILE_TIMEOUT}) is
likely too short for slower environments; increase ${RECONCILE_TIMEOUT} (or
adjust ${RECONCILE_RETRY}) so the suite allows more attempts given the
controller's reconcileInterval (~10s). Update the Variables block to set a
larger ${RECONCILE_TIMEOUT} (for example to 60s or more) or reduce
${RECONCILE_RETRY} gap to ensure enough reconciliation attempts relative to
reconcileInterval and event-driven triggers.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 66502359-d58b-4535-b110-837f5171a8be

📥 Commits

Reviewing files that changed from the base of the PR and between 6add7bd and 2834fb9.

⛔ Files ignored due to path filters (179)
  • go.sum is excluded by !**/*.sum
  • vendor/github.com/cenkalti/backoff/v4/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/backoff.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/context.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/exponential.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/retry.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/ticker.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/timer.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/backoff/v4/tries.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/hub/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/hub/.travis.yml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/hub/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/hub/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/hub/hub.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/.travis.yml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/client.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/codec.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/debug.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/jsonrpc/jsonrpc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/server.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/cenkalti/rpc2/state.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/.gitattributes is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/.golangci.yml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/charset/charset.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/csv/parser.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/json/parser.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/archive.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/audio.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/binary.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/database.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/document.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/font.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/ftyp.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/geo.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/image.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/magic.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/ms_office.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/netpbm.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/ogg.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/text.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/text_csv.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/video.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/magic/zip.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/markup/markup.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/internal/scan/bytes.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/mime.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/mimetype.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/supported_mimes.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/gabriel-vasile/mimetype/tree.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/.travis.yml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/currency/currency.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/logo.png is excluded by !**/*.png, !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/locales/rules.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/.travis.yml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/Makefile is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/errors.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/import_export.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/logo.png is excluded by !**/*.png, !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/translator.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/universal-translator/universal_translator.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/.golangci.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/MAINTAINERS.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/Makefile is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/baked_in.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/cache.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/country_codes.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/currency_codes.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/doc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/errors.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/field_level.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/logo.png is excluded by !**/*.png, !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/options.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/postcode_regexes.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/regexes.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/struct_level.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/translations.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/util.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/validator.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/go-playground/validator/v10/validator_instance.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/kind.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/machine.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/machine.go.rl is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/makefile is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/options.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/parsing_mode.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/scim.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/scim/schema/type.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/urn.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/leodido/go-urn/urn8141.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/NOTICE is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/cache/cache.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/cache/doc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/cache/uuidset.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/api.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/api_test_model.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/client.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/condition.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/config.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/doc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/metrics.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/monitor.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/options.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/client/validation.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/database/database.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/database/doc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/database/references.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/mapper/info.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/mapper/mapper.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/model/client.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/model/database.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/model/model.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/bindings.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/condition.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/error.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/map.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/monitor_select.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/mutation.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/named_uuid.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/notation.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/row.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/rpc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/schema.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/serverdb/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/serverdb/database.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/serverdb/gen.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/serverdb/model.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/set.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/update3.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/updates.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/updates2.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/ovsdb/uuid.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/updates/difference.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/updates/doc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/updates/merge.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/updates/mutate.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/updates/references.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/ovn-kubernetes/libovsdb/updates/updates.go is excluded by !**/vendor/**, !vendor/**
  • vendor/golang.org/x/crypto/sha3/hashes.go is excluded by !**/vendor/**, !vendor/**
  • vendor/golang.org/x/crypto/sha3/legacy_hash.go is excluded by !**/vendor/**, !vendor/**
  • vendor/golang.org/x/crypto/sha3/legacy_keccakf.go is excluded by !**/vendor/**, !vendor/**
  • vendor/golang.org/x/crypto/sha3/shake.go is excluded by !**/vendor/**, !vendor/**
  • vendor/modules.txt is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/.gitignore is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/CHANGELOG.md is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/CONTRIBUTING.md is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/LICENSE is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/Makefile is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/OWNERS is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/SECURITY_CONTACTS is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/code-of-conduct.md is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/error.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/exec.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/fake.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/nftables.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/objects.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/transaction.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/types.go is excluded by !**/vendor/**, !vendor/**
  • vendor/sigs.k8s.io/knftables/util.go is excluded by !**/vendor/**, !vendor/**
📒 Files selected for processing (29)
  • go.mod
  • pkg/cmd/run.go
  • pkg/config/c2cc.go
  • pkg/controllers/c2cc/annotation.go
  • pkg/controllers/c2cc/annotation_test.go
  • pkg/controllers/c2cc/controller.go
  • pkg/controllers/c2cc/helpers_test.go
  • pkg/controllers/c2cc/nbdb.go
  • pkg/controllers/c2cc/nbdb_test.go
  • pkg/controllers/c2cc/networkpolicy.go
  • pkg/controllers/c2cc/nftables.go
  • pkg/controllers/c2cc/nftables_test.go
  • pkg/controllers/c2cc/ovn.go
  • pkg/controllers/c2cc/ovn_test.go
  • pkg/controllers/c2cc/policy_routes.go
  • pkg/controllers/c2cc/policy_routes_test.go
  • pkg/controllers/c2cc/routes.go
  • pkg/controllers/c2cc/routes_test.go
  • pkg/controllers/c2cc/service_routes.go
  • pkg/controllers/c2cc/service_routes_test.go
  • test/assets/c2cc/curl-pod.yaml
  • test/assets/c2cc/hello-microshift.yaml
  • test/bin/scenario.sh
  • test/resources/c2cc.resource
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
  • test/suites/c2cc/connectivity.robot
  • test/suites/c2cc/infrastructure.robot
  • test/suites/c2cc/reconciliation.robot
  • test/suites/c2cc/sanity.robot

Comment thread pkg/controllers/c2cc/annotation.go
Comment thread pkg/controllers/c2cc/nbdb.go Outdated
Comment thread pkg/controllers/c2cc/networkpolicy.go Outdated
Comment thread pkg/controllers/c2cc/networkpolicy.go Outdated
Comment thread pkg/controllers/c2cc/nftables.go
Comment thread pkg/controllers/c2cc/policy_routes.go
Comment thread pkg/controllers/c2cc/service_routes.go Outdated
Comment thread test/bin/scenario.sh Outdated
Comment thread test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh Outdated
Comment thread test/suites/c2cc/reconciliation.robot
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test ?

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-el9

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test e2e-aws-tests-bootc-arm-el9

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
pkg/controllers/c2cc/nftables.go (1)

187-201: Use a resettable timer for the debounce loop.

time.After creates a new timer for every event and can't be canceled when a newer event arrives. A reusable time.Timer would cut timer churn and make shutdown cleaner.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/controllers/c2cc/nftables.go` around lines 187 - 201, Replace the
time.After-based debounce in the goroutine that watches rawCh/reconcileCh with a
reusable time.Timer: create a timer variable (e.g., debounceTimer *time.Timer)
initially nil, on receiving a rawCh event start the timer if nil
(time.NewTimer(2*time.Second)) or call debounceTimer.Reset(2*time.Second) if
already exists, on the timer channel fire send the "nftables-change" to
reconcileCh, and on goroutine shutdown stop the timer and drain its channel if
needed (use debounceTimer.Stop() and drain to avoid races). Ensure you reference
the existing rawCh and reconcileCh identifiers and replace uses of the local
debounce <-chan time.Time with this resettable timer logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/controllers/c2cc/nftables.go`:
- Around line 164-167: The current handler tears down the nftables watcher by
returning on a single sock.Receive() error; change it to keep the watcher alive
by replacing the immediate return with a retry/backoff loop that either
continues on transient errors or attempts to recreate the netlink socket (the
sock used for sock.Receive()) and resume receiving; update the logic around
msgs, _, err := sock.Receive() so errors are logged, waited-on with exponential
backoff, and the socket is reinitialized when necessary instead of returning,
ensuring the watcher goroutine does not exit on the first read error.

In `@pkg/controllers/c2cc/policy_routes.go`:
- Around line 67-98: cleanupRoutes and cleanupRules currently swallow deletion
errors (they log but always return nil); change both functions to collect errors
from netlink.RouteDel and netlink.RuleDel (e.g., store the first error or
aggregate multiple errors) and return a non-nil error when any deletion fails.
Specifically, in cleanupRoutes iterate over routes and on netlink.RouteDel
append or wrap the error (include context like route.Dst, t.table) instead of
only klog.Errorf, and similarly in cleanupRules capture errors from
netlink.RuleDel for rules matching t.priority and t.table and return an
aggregated/wrapped error at the end so callers see cleanup failures.

---

Nitpick comments:
In `@pkg/controllers/c2cc/nftables.go`:
- Around line 187-201: Replace the time.After-based debounce in the goroutine
that watches rawCh/reconcileCh with a reusable time.Timer: create a timer
variable (e.g., debounceTimer *time.Timer) initially nil, on receiving a rawCh
event start the timer if nil (time.NewTimer(2*time.Second)) or call
debounceTimer.Reset(2*time.Second) if already exists, on the timer channel fire
send the "nftables-change" to reconcileCh, and on goroutine shutdown stop the
timer and drain its channel if needed (use debounceTimer.Stop() and drain to
avoid races). Ensure you reference the existing rawCh and reconcileCh
identifiers and replace uses of the local debounce <-chan time.Time with this
resettable timer logic.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ca3c2996-c214-452b-92cd-c6c0af3d6b18

📥 Commits

Reviewing files that changed from the base of the PR and between 2834fb9 and dbf1c5f.

📒 Files selected for processing (5)
  • pkg/controllers/c2cc/annotation.go
  • pkg/controllers/c2cc/nbdb.go
  • pkg/controllers/c2cc/nftables.go
  • pkg/controllers/c2cc/policy_routes.go
  • pkg/controllers/c2cc/service_routes.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • pkg/controllers/c2cc/annotation.go
  • pkg/controllers/c2cc/service_routes.go

Comment thread pkg/controllers/c2cc/nftables.go
Comment thread pkg/controllers/c2cc/policy_routes.go
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 29, 2026

/test e2e-aws-tests-bootc-arm-el9

pmtk and others added 17 commits April 29, 2026 14:34
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 30, 2026
@openshift-ci openshift-ci Bot requested review from pacevedom and vanhalenar April 30, 2026 17:48
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/scenarios-bootc/el9/presubmits/el98-src`@c2cc.sh:
- Around line 58-63: The calls to wait_for_greenboot_on_hosts and
wait_for_microshift_to_be_ready currently rely on global errexit; change each
invocation (e.g. wait_for_greenboot_on_hosts "c2cc_pre_greenboot",
wait_for_greenboot_on_hosts "c2cc_greenboot", and
wait_for_microshift_to_be_ready) to perform an explicit return-code check: run
the function, test its exit status with if ! <call>; then emit a clear error
message (e.g. via echo or logger) and exit or return a non-zero status (exit 1)
to fail fast; keep configure_c2cc_host calls unchanged but ensure subsequent
waits follow this explicit-check pattern.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c46ac14a-0f05-4c99-b70d-121c97d50990

📥 Commits

Reviewing files that changed from the base of the PR and between 41e3ab2 and 22dbfdd.

📒 Files selected for processing (2)
  • pkg/controllers/c2cc/ovn.go
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
✅ Files skipped from review due to trivial changes (1)
  • pkg/controllers/c2cc/ovn.go

Comment thread test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
pkg/controllers/c2cc/ovn.go (1)

230-252: 💤 Low value

Consider using a backoff for reconnect polling.

The 1-second polling loop at lines 238-244 is functional but will generate constant CPU wake-ups during prolonged outages. An exponential backoff would be gentler, though not critical for MicroShift's typical deployment.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/controllers/c2cc/ovn.go` around lines 230 - 252, The reconnect loop
spawned in the goroutine using m.nbClient.DisconnectNotify() currently polls
m.nbClient.Connected() every 1s; replace that tight fixed-interval loop with an
exponential backoff retry (e.g., start at ~1s, double on each failed try up to a
max like 30s) and reset the backoff when Connected() returns true. Update the
inner select that checks ctx.Done() and the time.After to use the backoff
duration, and keep sending "ovn-reconnected" on reconcileCh as before; reference
the goroutine, m.nbClient.DisconnectNotify(), m.nbClient.Connected(),
ctx.Done(), and reconcileCh when making the change.
test/resources/c2cc.resource (1)

205-217: 💤 Low value

Inline Python JSON manipulation could fail silently on malformed input.

If ${current} is empty or malformed JSON, the Python one-liner will raise an exception that may produce unclear test failures. Consider adding a sanity check or fallback.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/resources/c2cc.resource` around lines 205 - 217, The inline Python
one-liner that builds ${new_value} from ${current} can fail if ${current} is
empty or malformed; update the Command On Cluster invocation that generates
${new_value} so the Python snippet defensively handles bad input (check if stdin
is empty or not valid JSON and default to an empty list), then append
${foreign_cidr} and print the JSON-sorted result; reference the variables and
keywords: Get Node SNAT Annotation (to obtain ${current}), ${current},
${new_value}, and the python3 one-liner used in Command On Cluster, and ensure
the patched oc annotate step still receives the JSON string output.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/controllers/c2cc/ovn.go`:
- Around line 230-252: The reconnect loop spawned in the goroutine using
m.nbClient.DisconnectNotify() currently polls m.nbClient.Connected() every 1s;
replace that tight fixed-interval loop with an exponential backoff retry (e.g.,
start at ~1s, double on each failed try up to a max like 30s) and reset the
backoff when Connected() returns true. Update the inner select that checks
ctx.Done() and the time.After to use the backoff duration, and keep sending
"ovn-reconnected" on reconcileCh as before; reference the goroutine,
m.nbClient.DisconnectNotify(), m.nbClient.Connected(), ctx.Done(), and
reconcileCh when making the change.

In `@test/resources/c2cc.resource`:
- Around line 205-217: The inline Python one-liner that builds ${new_value} from
${current} can fail if ${current} is empty or malformed; update the Command On
Cluster invocation that generates ${new_value} so the Python snippet defensively
handles bad input (check if stdin is empty or not valid JSON and default to an
empty list), then append ${foreign_cidr} and print the JSON-sorted result;
reference the variables and keywords: Get Node SNAT Annotation (to obtain
${current}), ${current}, ${new_value}, and the python3 one-liner used in Command
On Cluster, and ensure the patched oc annotate step still receives the JSON
string output.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e8939fdf-a12c-4c50-8eba-742f89423f79

📥 Commits

Reviewing files that changed from the base of the PR and between 41e3ab2 and 5338603.

📒 Files selected for processing (4)
  • pkg/controllers/c2cc/ovn.go
  • test/resources/c2cc.resource
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
  • test/suites/c2cc/cleanup.robot
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/suites/c2cc/cleanup.robot
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh

pmtk added 2 commits May 4, 2026 10:53
c2cc blocks readiness waiting for OVN socket, which might result in
MicroShift restart:
- MicroShift starts, wants to create kubepods.slice
- Systemd queue waits in sequence: first MicroShift, then kubepods.slice
- MicroShift does not become ready, no kubepods.slice, no Node, no Pods
- No Pods means no OVN socket
@ggiguash
Copy link
Copy Markdown
Contributor

ggiguash commented May 4, 2026

Code Review: Critical Issues

1. BUG: Double close(ready) will panic — controller.go

The Run() method calls close(ready) unconditionally early (for the "early readiness" design), then calls it again in the error paths for initKubeClient, connectOVNNB, and initSubsystems. Closing an already-closed channel panics in Go.

close(ready)  // always runs

if err := c.initKubeClient(); err != nil {
    close(ready)  // PANIC: already closed
    return ...
}

Fix: Remove the close(ready) calls from the three error-return blocks after the initial unconditional close.

2. Unnecessary API writes every reconcile cycle — networkpolicy.go

Every reconcile cycle (every 10s + events) triggers a Kubernetes Update call on the NetworkPolicy even when nothing changed. This generates unnecessary API server load and Kubernetes events.

Fix: Compare the existing NetworkPolicy spec with the desired one and skip the Update when they already match.

3. Cleanup functions silently swallow errors — routes.go, service_routes.go

Both linuxRouteManager.cleanup() and serviceRouteManager.cleanup() assign errors to _ and always return nil:

func (m *linuxRouteManager) cleanup(ctx context.Context) error {
    _ = m.cleanupRoutes()
    _ = m.cleanupRules()
    return nil  // always nil
}

The upstream cleanupAll in controller.go logs errors but will never see them because cleanup always reports success. Either propagate the errors or log them at the cleanup site.

@ggiguash
Copy link
Copy Markdown
Contributor

ggiguash commented May 4, 2026

Code Review: Medium Issues

4. parseCIDRAnnotation silently drops malformed data — annotation.go

If another component writes a malformed SNAT annotation, parseCIDRAnnotation returns nil instead of logging a warning. The reconciler then treats the annotation as empty and overwrites it, potentially clobbering the other component's data.

func parseCIDRAnnotation(value string) []string {
    if value == "" {
        return nil
    }
    var cidrs []string
    if err := json.Unmarshal([]byte(value), &cidrs); err != nil {
        return nil  // silently swallows parse errors
    }
    return cidrs
}

Suggestion: Log a warning on parse failure so operators can diagnose annotation conflicts.

5. Annotation patch built via string interpolation — annotation.go

The cleanup patch uses nested fmt.Sprintf with %q to construct JSON:

patch := fmt.Sprintf(`{"metadata":{"annotations":{%s:%s,%q:null}}}`,
    fmt.Sprintf("%q", ovnNodeDontSNATSubnets), snatValue,
    c2ccSNATTrackingAnnotation)

This works but is fragile — any unexpected characters in annotation keys or values could produce invalid JSON.

Suggestion: Use json.Marshal to build the patch object instead of string interpolation.

6. NetworkPolicy scope may be too narrow — networkpolicy.go

The NetworkPolicy is created only in the default namespace with an empty PodSelector (matching all pods in that namespace). This means:

  • Pods in default get very permissive ingress from all remote cluster CIDRs
  • Pods in any other namespace get no C2CC ingress allowance at all

If users deploy workloads in other namespaces, they'll need to create their own NetworkPolicies. This may be intentional, but it should be documented — either in code comments or in the feature documentation.

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented May 5, 2026

/retest

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented May 6, 2026

/retest

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented May 6, 2026

/verified by @pmtk

@openshift-ci-robot
Copy link
Copy Markdown

@pmtk: This PR has been marked as verified by @pmtk.

Details

In response to this:

/verified by @pmtk

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label May 6, 2026
@pacevedom
Copy link
Copy Markdown
Contributor

/assign


// Declaring ready even before init because many of the components it tries to communicate with are not up yet
// and excessive waiting before readiness can cause them to never become ready resulting in MicroShift restart.
close(ready)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any other way to signal readiness? I imagine a pod coming up requiring this before it becomes ready. Is that a possibility?

Copy link
Copy Markdown
Member Author

@pmtk pmtk May 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be readiness custom resource which will report true status of C2CC reconciliation and connectivity


run_command_on_vm "${host}" "sudo mkdir -p /etc/microshift/config.d"
run_command_on_vm "${host}" "sudo tee /etc/microshift/config.d/50-c2cc.yaml > /dev/null << EOF
clusterToCluster:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to update the enhancement to match this name instead of c2cc

)

const (
c2ccSvcRouteTable = 201
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the enhancement this is configurable, right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet. Enhancement is about feature as whole, configuring this will come later.

)

const (
c2ccRouteTable = 200
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the enhancement this is configurable, right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet. Enhancement is about feature as whole, configuring this will come later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldnt we have a rhel10 test too?

${NAMESPACE} c2cc-test


*** Test Cases ***
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we adding the DNS tests later on, or should they be in this PR?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Later, together with the implementation

Comment thread pkg/controllers/c2cc/service_routes.go Outdated
Comment on lines +181 to +193
if addr.IP.To4() != nil {
ip4 := addr.IP.To4()
gwIP := make(net.IP, len(ip4))
copy(gwIP, ip4)
gwIP = gwIP.Mask(addr.Mask)
gwIP[len(gwIP)-1] = 1
gateways[netlink.FAMILY_V4] = mgmtPortGateway{ip: gwIP, linkIdx: linkIdx}
} else if addr.IP.To16() != nil {
ip6 := make(net.IP, len(addr.IP.To16()))
copy(ip6, addr.IP.To16())
ip6 = ip6.Mask(addr.Mask)
ip6[len(ip6)-1] = 1
gateways[netlink.FAMILY_V6] = mgmtPortGateway{ip: ip6, linkIdx: linkIdx}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we grab the gateway address from the routing table instead? That would be automatic and not subject to changes in ovnk.

Comment thread pkg/controllers/c2cc/ovn.go Outdated
Comment on lines +47 to +48
allCIDRs := append([]*net.IPNet{}, rc.ClusterNetwork...)
allCIDRs = append(allCIDRs, rc.ServiceNetwork...)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are four places where we append cluster network and service network cidrs, is it possible to have a function?

@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label May 6, 2026
@pmtk pmtk changed the title USHIFT-6795: C2CC: Router controller USHIFT-6795: C2CC: Routes controller May 6, 2026
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented May 6, 2026

/retest

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 6, 2026

@pmtk: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-tests-periodic d2dc1af link true /test e2e-aws-tests-periodic
ci/prow/e2e-aws-tests-periodic-arm d2dc1af link true /test e2e-aws-tests-periodic-arm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants