OVN: Open Virtual Network as a guest network provider#135
Open
msinhore wants to merge 33 commits intoshapeblue:mainfrom
Open
OVN: Open Virtual Network as a guest network provider#135msinhore wants to merge 33 commits intoshapeblue:mainfrom
msinhore wants to merge 33 commits intoshapeblue:mainfrom
Conversation
Enforce TLS certificate requirement when either the Northbound or Southbound connection uses ssl:, add a logger to the provider service impl, drop the redundant CollectionUtils.isNotEmpty check in validateNetworkState, and correct a stale migration comment in the guest network guru.
Replaces the regex-only OvnNbClient with a real OVSDB JSON-RPC client that opens a transient connection to the OVN Northbound endpoint, runs an echo, and confirms the OVN_Northbound database is advertised. Both tcp: and ssl: connection strings are supported. Adds OvnSslContext, an ICertificateManager implementation that loads the operator-supplied CA, client certificate and client private key into in-memory keystores so SSL connections do not need any keystore on disk. OvnService grows verifyNbConnection so callers can fail fast when the Northbound endpoint is unreachable. The ODL OVSDB library 1.18.3 is declared as a plugin dependency with Jackson, Guava and OSGi annotations excluded, since CloudStack already pins those.
addOvnProvider now opens a transient OVSDB connection during request validation and rejects the registration with InvalidParameterValueException if the endpoint is unreachable or does not advertise the OVN_Northbound database. This catches typos, missing TLS material and routing problems before persisting a broken provider row. Also reorders the order-of-evaluation of two pre-existing tests that constructed Mockito stubs inside thenReturn() arguments, and rewrites AddOvnProviderCmdTest to set the service field by reflection — the prior @InjectMocks-driven setup was ambiguous because BaseCmd has an Object-typed _responseObject field that matched any of the @mock fields.
OvnElement.prepare/release now creates and removes a Logical_Switch_Port in the OVN Northbound database for every NIC plugged into an OVN-broadcast network. The LSP carries the NIC's MAC and IPv4 in both addresses and port_security and is named after the NIC UUID; OvnNbClient performs the insert+mutate transaction natively over OVSDB JSON-RPC and is idempotent on both ends. OvsVifDriver picks up a new BroadcastDomainType.OVN branch that drops the NIC tap on the OVN integration bridge and stamps external_ids:iface-id with the same NIC UUID, which is what ovn-controller looks up to bind the local OVS port to the LSP. End to end the VM now appears under the correct chassis in ovn-sbctl show and traffic is plumbed through OVN.
OvnElement.implement now seeds a DHCP_Options row whose cidr/router/dns mirror the CloudStack network and tags it with cloudstack_network_id in external_ids; OvnElement.prepare links each new Logical_Switch_Port to that row through dhcpv4_options so ovn-controller answers DHCP DISCOVER locally instead of leaking the request out of the integration bridge. The row is removed in destroy along with the Logical_Switch. OvnNbClient grows three native OVSDB helpers — createDhcpOptions (idempotent via external_ids lookup), deleteDhcpOptions, setLspDhcpv4Options — that shape the JSON-RPC transactions for these flows. Verified end to end with ovn-trace: a DHCPDISCOVER from a VM LSP triggers put_dhcp_opts with the expected offerip/lease_time/mtu/router/server_id and an OFFER reply with the per-network server_mac.
OvnElement.implement now provisions the full external attachment for an
isolated guest network:
* a Logical_Router (cs-router-<id>) with a guest LRP at the network
gateway and a public LRP carrying the source-NAT public IP;
* a public-side Logical_Switch (cs-pub-<id>) with a localnet port
bound to the operator-supplied physical network so traffic can leave
via br-ex;
* a snat NAT row mapping the guest CIDR to the public IP;
* a default Logical_Router_Static_Route to the VLAN gateway so OVN
knows where to ARP next-hops.
The trigger fires from implement() instead of waiting for
applyIpAssociations because no port-forwarding/static-NAT rule is
required for source-NAT alone — Netris does the same. ovn-trace
confirms a packet from the VM toward 8.8.8.8 is rewritten with the
public IP, switched out the LR's external port and ARPs the VLAN
gateway.
OvnNbClient grows native OVSDB helpers for Logical_Router,
Logical_Router_Port (with a paired router-type LSP via
attachRouterToSwitch), localnet LSP, NAT and Logical_Router_Static_Route.
…s anchor applyStaticNats now installs a dnat_and_snat NAT row on the per-network Logical_Router with external_mac/logical_port set to the VM's NIC, so the distributed pipeline can rewrite both directions without bouncing through a centralised gateway for the SNAT half. ovn-northd will only materialise lr_in_dnat for a NAT row when the public-side Logical_Router_Port is anchored to a chassis via Gateway_Chassis. Without it, lr_in_dnat keeps only the priority-0 default and inbound packets are silently dropped instead of ct_dnat()'d. To pick a chassis we now query the OVN_Southbound Chassis table via a new listSouthboundChassisNames helper and rotate by network id, so multiple LRs do not all pile on the same hypervisor. The Chassis row is keyed by the OVS system-id, which is what Gateway_Chassis.chassis_name must reference - the previous attempt of using HostVO.getGuid was wrong because that surface is not the system-id. OvnNbClient additions: - addNatRule overload taking distributedMac + distributedLogicalPort - removeNatRulesByExternalIp (delete by type + external_ip alone) - setLrpGatewayChassis (idempotent Gateway_Chassis create + LRP mutate) - listSouthboundChassisNames (queries OVN_Southbound) - runOnDb refactor so the same connect/dispatch path serves NB and SB OvnElement additions: - pickAnchorChassis(provider, network) using OVN SB - applySourceNatForNetwork now also anchors the public LRP - applyStaticNats anchors before installing the dnat_and_snat row Validated on the lab against cs-router-257: cycling the static NAT through cmk recreates the NAT row and a Gateway_Chassis row pointing at ovn-kvm-02, and ovn-trace from the localnet shows "lr_in_dnat priority 100 ... ct_dnat(10.1.1.222)" and output to the guest LSP for the inbound 10.0.56.186 -> 10.1.1.222 flow.
applyFWRules now translates each CloudStack FirewallRule into an OVN
ACL row attached to the network's public Logical_Switch in the
"to-lport" direction toward the router patch port. Allowed traffic uses
"allow-related" so conntrack opens the reverse path automatically; the
match expression is constructed before DNAT (ip4.dst == publicIp) so
the user's public-side TCP/UDP port stays correct.
A per-public-IP default-drop ACL is installed alongside the first allow
to flip the implicit OVN allow-by-default into CloudStack's expected
deny-by-default for an exposed public IP. Every ACL row is tagged via
external_ids:cloudstack_fw_rule_id (per rule) plus cloudstack_fw_ip and
cloudstack_network_id, which lets revoke target the row by tag without
re-walking the rule set.
OvnNbClient additions:
- ACL_TABLE constant
- addAclOnLs (idempotent: removes any prior ACL with the same
external_ids tag before inserting + linking via Logical_Switch.acls)
- removeAclsOnLsByExternalId (deletes by tag, mutates LS.acls)
- findAclUuidsByExternalIds helper (server-side select + client-side
map filter, since OVSDB select cannot match into a map column)
OvnElement additions:
- applyFWRules now programs ACL per FirewallRule
- programFirewallRule respects FirewallRule.State.Revoke
- buildFirewallMatch handles tcp/udp dst ports, icmp type/code, "all"
and source CIDR list (skipping the 0.0.0.0/0 noise)
- ensureFirewallDefaultDeny installs the per-IP drop ACL
Validated on the lab against cs-pub-257 with two rules on 10.0.56.186
(icmp + tcp/22). ovn-nbctl acl-list shows the two allow-related ACLs
plus the priority-100 drop. ovn-trace confirms:
- tcp/22 -> ACL allow -> ct_dnat(10.1.1.222) -> output to guest LSP
- icmp -> ACL allow -> ct_dnat(10.1.1.222) -> output to guest LSP
- tcp/80 -> ls_out_acl_eval priority 1100 hits the drop, no output
removeNatRulesByExternalIp and removeNatRule were issuing a bare delete on the NAT table, leaving the strong reference set in Logical_Router.nat dangling. OVSDB then refused the delete with "referential integrity violation: cannot delete NAT row ... because of N remaining reference(s)", which surfaced in CloudStack as "Failed to expunge VM" / "Failed to disable static NAT" whenever a VM tied to a static NAT IP was destroyed. Both helpers now resolve the matching NAT UUIDs first, then mutate Logical_Router.nat to remove those UUIDs and delete the rows in the same transaction. Same idempotency guarantees as before, no new public API. Validated on the lab: destroyed the static-NAT VM that previously got stuck in Error and deleted its network. ovn-nbctl shows the LR, LSes, NAT, ACL and Gateway_Chassis tables fully empty afterwards.
PortForwarding (applyPFRules): - Adds dnat_and_snat NAT rows with external_port + protocol columns - Handles port ranges by iterating extPortStart..extPortEnd - Logs a warning when external port ≠ internal port (OVN NAT does 1:1 port mapping; full remapping would require a Load_Balancer rule) - Revoke path removes per-port rows via removeNatRulesByExternalIpAndPort - Anchors public LRP Gateway_Chassis on first call so ovn-northd materialises the lr_in_dnat pipeline NetworkACL (applyNetworkACLs): - Full-sync: wipe all ACLs tagged cloudstack_network_id=<id> then re-install the authoritative set so reordering and list replacements converge correctly - CloudStack Ingress → OVN to-lport; Egress → from-lport - OVN priority derived from rule.getNumber() (lower CS# = higher prio) - Default-drop at priority 1 for both directions matches CloudStack's implicit deny-all-unmatched semantics (OVN default is allow-all) - buildNetworkAclMatch: supports tcp/udp (port range), icmp (type/code), all, and source/destination CIDR sets OvnNbClient additions: - addNatRuleWithPorts: inserts NAT row with external_port (Set<Long>) and protocol (Set<String>) optional columns; idempotency check filters client-side since OVSDB WHERE cannot match into set columns - removeNatRulesByExternalIpAndPort: select-then-delete pattern with LR.nat mutation to satisfy strong-ref integrity (same as existing removeNatRulesByExternalIp) - natRuleWithPortExists: helper to detect existing port-scoped NAT rows
…rwarding SNAT replies were silently lost when another device on the public VLAN claimed the same external IP first - the upstream switch CAM table cached the wrong MAC, and our LR's only ARP behaviour was to respond when asked. With no unsolicited announcement, return traffic to 10.0.56.183 routed to the wrong host until manual intervention. Fix: configure options:nat-addresses on the type=router LSP that peers the external LRP, formatted as "<LR_external_MAC> <SourceNat_IP> ...". On port claim, ovn-controller emits gARPs for each address listed (initial burst then periodic announce), forcing the upstream to converge on our MAC. Use the explicit string form rather than nat-addresses=router because the keyword mode only covers dnat_and_snat rules with logical_port set, which excludes plain SNAT. Also set exclude-lb-vips-from-garp=true to stay consistent with the Neutron OVN driver default. Triggered from applySourceNatForNetwork (initial implement) and applyIps (runtime IP assignment) so updates converge whenever CloudStack adds/removes a SourceNat IP. Helper rebuilds the full address list from the DB so multiple SourceNat IPs on a single network all get announced. OvnNbClient: new setLspOptions() merges entries into Logical_Switch_Port.options, preserving keys we don't manage (router-port) and bailing out when nothing changes to avoid spurious NB notifications. PortForwarding: drop the NAT-based path. OVN NB schema 24.03 removed NAT.external_port and NAT.protocol; the previous code referenced both columns and crashed at runtime with "ColumnSchema is null" when revoking PF rules. applyPFRules now logs a clear warning and returns true; LB-backed PF (the Neutron pattern) will land in a follow-up.
OVN NB schema 24.03 dropped NAT.external_port and NAT.protocol; even where the
columns existed, NAT-based DNAT could not remap an external port to a different
internal port, which CloudStack PortForwarding semantics require. This commit
replaces the (broken) NAT-based path with Load_Balancer rows, the same approach
the Neutron OVN port-forwarding driver uses.
OvnNbClient gains four primitives:
createOrReplaceLoadBalancer(name, protocol, vips, externalIds, options)
Inserts or atomically updates a Load_Balancer row keyed by name. vips is
the canonical "<vip_ip>:<vip_port>" -> "<backend_ip>:<backend_port>" map -
OVN northd interprets each entry as one DNAT rule and rewrites both IP
and port, which is what we need for source != destination port mappings.
attachLoadBalancerToRouter(routerName, lbName)
Mutates Logical_Router.load_balancer to include the LB. Idempotent.
attachLoadBalancerToSwitch(switchName, lbName)
Same for Logical_Switch.load_balancer. Required for return traffic when a
backend VM talks to its own FIP (RHBZ#2043543) - without it, the LS would
not see the LB pipeline on egress and replies to hairpin traffic would be
SNATed incorrectly.
removeLoadBalancersByExternalId(key, value)
Walks every LR and LS, mutates their load_balancer set to detach matching
LBs, then deletes the LB rows themselves. Detach must come first because
Load_Balancer is a strong reference set in the OVN NB schema.
OvnElement.applyPFRules now translates each PortForwardingRule into one LB:
- LB name: pf-<rule_id>-<protocol> (one row per rule, simple to revoke)
- vips: one entry per source port. CloudStack range A-B mapped to internal
range C-D becomes A->C, A+1->C+1, ..., B->D. A single dest port is also
supported (all source ports map to the same internal port). Mismatched
range sizes are rejected.
- protocol: tcp/udp/sctp - any other value is skipped with a warning
- external_ids: cloudstack_pf_rule_id, cloudstack_network_id,
cloudstack_nat_kind=portforward (used by revoke)
- options: hairpin_snat_ip=<external_ip> so a VM behind the FIP can talk
to its own public address without the router mis-routing the reply
- attached to both LR (cs-router-<network_id>) and guest LS
(cs-net-<network_id>)
Defensive cap: ranges over MAX_PF_RANGE (256) are rejected to keep transactions
bounded - users with bigger needs can split the rule.
Validated end-to-end in the lab: created PF rule public:22 -> VM:22, LB row
showed up with vips={"10.0.111.209:22":"10.1.1.66:22"} attached to LR and LS,
external TCP probe to the public IP completed handshake on the VM tap with the
OpenSSH banner reaching the client.
…way chassis
Two real bugs hit during lab work surfaced two cleanup gaps that this commit
closes.
(1) Per-IP default-drop ACL leaked on disassociate.
ensureFirewallDefaultDeny plants an ACL of the form
outport == "lsp-lrp-cs-pub-<id>" && ip4 && ip4.dst == <public_ip>
action=drop priority=100
external_ids={cloudstack_fw_default=true, cloudstack_fw_ip=<ip>, ...}
so that the OVN default-allow on a logical switch does not leak public traffic
to internal hosts when no explicit allow rule covers it. The ACL row carries
no cloudstack_fw_rule_id - applyFWRules revoke (which removes ACLs by rule_id)
never matched it, so the row stayed alive when the public IP was disassociated.
A subsequent reassignment of the same IP to another tenant would then inherit
the old drop, blocking traffic invisibly.
Fix: extend the IP-Releasing branch of applyIps to fire for every public IP
(not just SourceNat ones) and call a new cleanupPublicIpArtifacts helper that
drops all ACLs tagged cloudstack_fw_ip=<ip> on the public LS, plus any
dnat_and_snat NAT row on that external IP, plus refreshes the LSP nat-addresses
list so the IP stops being announced via gARP. Validated end-to-end: associate
+ FW rule produced the default-drop and the allow-related rows, disassociate
removed both with no SQL or ovn-nbctl intervention.
(2) Gateway_Chassis row pointed to a dead chassis after host re-add.
When a KVM host is destroyed and re-added (which happened naturally during the
zone rebuild today), ovn-controller registers the chassis with a fresh OVS
system-id. Our existing setLrpGatewayChassis is idempotent on (lrp, chassis),
so it noticed nothing wrong and the old Gateway_Chassis row stayed bound to
the deceased system-id. ovn-northd then never claimed the cr-lrp port and
SNAT/DNAT silently broke for the network until we manually deleted the row.
Fix: new OvnNbClient.pruneStaleGatewayChassis(lrp, liveChassisNames) that
walks the LRP's gateway_chassis set, checks each row's chassis_name against
the SB live chassis list, detaches and deletes the stale ones in a single
transaction. pickAnchorChassis now invokes it before returning a chassis to
the caller, so any subsequent setLrpGatewayChassis call sees a clean slate
and inserts a Gateway_Chassis row pointing at a live chassis.
OVN's Load_Balancer is L4 only - the datapath has no TLS stack and no HTTP
parser. We expose only what OVN can honour, so the offering is honest with
the user instead of silently degrading L7 features.
OvnElement:
- validateLBRule rejects rules whose protocol is not tcp/udp/sctp (catches
http, ssl, tcp-proxy from the API/UI before persistence) and rejects the
leastconn algorithm (OVN keeps no per-backend connection state).
- Capabilities trim:
* SupportedLBAlgorithms: "roundrobin,source" (was: "roundrobin,leastconn")
* SupportedProtocols: "tcp,udp" (no ssl/http)
* LbSchemes: "Public" (Internal scheme deferred)
* HealthCheckPolicy: "true" (so CS lets the user
attach a HC policy)
- applyLBRules iterates rules; programLBRule:
* Builds Load_Balancer.vips from rule.getDestinations() with
"<external>:<public_port>" -> "<vm_ip>:<priv_port>,..." (one row per
rule).
* Maps algorithm/stickiness to selection_fields and affinity_timeout:
algo=source or stickiness method SourceBased -> selection_fields=ip_src;
stickiness expire/idletime/holdtime/timeout -> options:affinity_timeout
(default 180s). Cookie-based stickiness logs and degrades to
source-based since OVN cannot inspect L7.
* Sets options:hairpin_snat_ip = external IP (Neutron-style) so a backend
VM talking to its own public address gets a clean reply path.
* external_ids tag: cloudstack_lb_rule_id, cloudstack_network_id,
cloudstack_lb_kind=loadbalancer. Revoke uses the rule_id tag to drop
the row and detach from LR/LS.
* Populates Load_Balancer.ip_port_mappings (backend_ip ->
"<lsp_name>:<source_ip>") so SB Service_Monitor knows from which LSP
and source IP to probe each backend. The source IP is the LR's gateway
on the guest LS (network.getGateway()).
* Attaches the LB to LR (cs-router-<id>) and the guest LS (cs-net-<id>)
- both attachments are required for the Neutron-recommended workaround
for RHBZ#2043543 (LB visibility for return traffic from internal
backends).
- applyLBHealthCheck creates one Load_Balancer_Health_Check row per active
HC policy, mapping CS fields:
getHealthcheckInterval -> options:interval
getResponseTime -> options:timeout
getHealthcheckThresshold -> options:success_count
getUnhealthThresshold -> options:failure_count
HTTP/PING ping paths are accepted but warn-degraded to TCP probes since
OVN cannot do HTTP-aware probes. ipPortMappings (already populated by
programLBRule) lets the SB Service_Monitor source the probe correctly.
- updateHealthChecks remains a stub - reading SB Service_Monitor status to
surface red/green per backend back to the CS UI is a follow-up.
OvnNbClient:
- createOrReplaceLoadBalancer overload that also writes selection_fields
and ip_port_mappings; the existing 7-arg signature stays as a wrapper for
PortForwarding (which doesn't need either).
- setLoadBalancerHealthCheck(lbName, vip, options, ipPortMappings, ext_ids)
inserts a fresh HC row in the same transaction that swaps LB.health_check,
deleting any old HC the LB referenced (strong-ref; orphan-free).
- clearLoadBalancerHealthCheck(lbName) is the symmetric op for HC revoke or
no-policy state.
Validated end-to-end: created LB rule public 10.0.111.207:8080 -> two VMs at
10.1.1.66:22 and 10.1.1.150:22, six TCP probes from outside completed; added
a HealthCheckPolicy and verified the SB Service_Monitor table shows two
active TCP probes (one per backend) via the LB's ip_port_mappings.
Two nits the lab surfaced (not fixed by this commit, since they're CS-side
configuration rather than OVN code):
1) Existing OVN offerings need public_lb=1 in network_offerings table
and Lb in ntwk_offering_service_map / ntwk_service_map for live
networks. New offerings created after this commit will inherit it from
the seed when the seed is updated.
2) An IP that already carried a Firewall-purpose rule cannot be reused for
LB; the IP must be disassociated and reassociated. Standard CS
behaviour, not OVN-specific.
When a tenant tries to create a LoadBalancingRule on the same public IP that the LR uses as SourceNat egress, the OVN pipeline cannot serve traffic in a way that round-trips correctly. The LR carries a snat NAT row matching logical_ip=guest_cidr -> external_ip in addition to the LB's vips entry on the same external_ip. Replies from a backend traverse lr_out_snat first and get rewritten with the SourceNat IP before the LB un-DNAT runs in the egress path, so the client sees a TCP packet from an IP that doesn't match the 4-tuple of the connection it opened and resets. Confirmed in the lab: SYNs reach the LR and DNAT to the backend, the backend replies, but no SYN+ACK ever leaves the trunk. Adding a guard in validateLBRule fails the rule synchronously when CloudStack asks the provider whether the rule is acceptable, before it is persisted. The error message tells the operator to allocate a dedicated public IP for the LB - which is the only correct path on OVN today (skip_snat / hairpin options on the LB row don't lift the conflict because the SourceNat NAT row applies independently in the LR egress pipeline). The check is provider-scoped (only fires for OVN-backed networks) and is defensive even if a tenant somehow hits the API directly (e.g. through DB-direct mutation): we look up the IPAddressVO for the LB's source IP and abort if it is flagged as SourceNat.
Address review findings on the LB implementation so stale Load_Balancer rows cannot survive ad-hoc network or public-IP teardown paths, and so the validateLBRule / programLBRule / capability triple stays in sync. - destroy(): wipe Load_Balancer rows tagged with this network's cloudstack_network_id before tearing down the LR/LS they hung off, so a force-delete of the network that bypasses the LB revoke callback no longer leaves orphan rows in NB DB. - programLBRule(): tag every Load_Balancer with cloudstack_lb_ip set to the public IP, and have cleanupPublicIpArtifacts() sweep LBs by that external_id when the IP is released. Closes the gap when an IP is freed before the LB rule revoke arrives. - validateLBRule() and programLBRule() drop sctp from the accepted set to match initCapabilities(), which only advertises tcp/udp. - applyLBRules(): document why an empty rule list is a no-op rather than a wipe (the LB manager passes rules in transition, not the full active set).
Centralise the resolution of the OVN Logical_Router and public
Logical_Switch names so the same call sites work for both isolated
networks and VPC tiers. Today every helper still returns the existing
isolated names (cs-router-{netId} / cs-pub-{netId}) when
network.getVpcId() is null, so behaviour for isolated networks is
unchanged.
Helpers added:
- getRouterNameForNetwork(network)
cs-router-{netId} for isolated, cs-vpc-{vpcId} for VPC tier.
- getPublicLogicalSwitchNameForNetwork(network)
cs-pub-{netId} for isolated, cs-vpc-pub-{vpcId} for VPC tier.
- getPublicRouterPortNameForNetwork(network)
"lrp-" + public LS name. Used for attachRouterToSwitch /
setLrpGatewayChassis / pruneStaleGatewayChassis call sites.
- getPublicRouterSwitchPortNameForNetwork(network)
"lsp-" + public LRP. Used by applyFWRules and
applyNatAddressesAnnouncement so firewall ACL match expressions and
gARP nat-addresses options point at the same LSP that pairs the
public router port.
All call sites in OvnElement that previously hard-coded the per-network
naming scheme now resolve through these helpers. The old
getLogicalRouterName / getPublicLogicalSwitchName methods are removed
to keep one resolver per concept and avoid drift between isolated and
VPC code paths.
This is the PR-1 prerequisite for the upcoming VPC implementation
(implementVpc / shutdownVpc, VPC tier hooks, public IP services on the
VPC LR). Each later PR can branch on network.getVpcId() inside its own
logic without touching naming again.
PR-2a of the VPC track. Provision the per-VPC OVN topology so a
CloudStack VPC backed by the OVN provider gets a real Logical_Router
and (when the SourceNat IP is allocated) a public Logical_Switch wired
through localnet to the upstream physical network. Per-tier work
(tier LS, tier LRP, per-tier SNAT row, DHCP) is the next PR.
Naming and MAC scheme:
- LR: cs-vpc-{vpcId}
- public LS: cs-vpc-pub-{vpcId}
- public LRP / LSP: lrp-cs-vpc-pub-{vpcId} / lsp-lrp-cs-vpc-pub-{vpcId}
- buildVpcRouterMac uses leading octets 0xfc (external) / 0xfb
(internal) so that an isolated network and a VPC sharing the same
numeric id never collide on a MAC.
implementVpc(vpc, dest, ctx):
- Looks up the OVN provider for vpc.getZoneId(); fails the call if the
zone has none.
- Idempotently creates the LR cs-vpc-{id} with stable external_ids
(cloudstack_vpc_id, cloudstack_vpc_uuid, cloudstack_zone_id,
cloudstack_role=vpc-router).
- Calls applyVpcSourceNatPublicSide() which - when CloudStack already
allocated the VPC SourceNat IP - creates the public LS, the localnet
port keyed off provider.localnetName/externalBridge plus VLAN tag,
attaches the LR via the public LRP using the SourceNat IP, anchors
it to a Gateway_Chassis (with stale-row prune), installs the default
route via the upstream gateway, and refreshes the gARP nat-addresses
option on the router-type LSP.
- No-op for the public side when no SourceNat IP is allocated yet; the
LR exists, and updateVpcSourceNatIp will re-run the helper later.
shutdownVpc(vpc, ctx):
- Sweeps Load_Balancer rows tagged with cloudstack_vpc_id (LB rows
live in a global table and are not GC'd by deleting the LR/LS when
other refs remain).
- Deletes the public LS (cleaning its localnet/firewall/router-type
LSPs).
- Deletes the LR; OVSDB GCs LRPs and NAT rows it owned.
- Returns success even when the OVN provider is gone, so VPC removal
can complete.
updateVpcSourceNatIp(vpc, ip):
- Re-runs applyVpcSourceNatPublicSide. Idempotent, covers first-time
allocation cleanly. Changing an already-active SourceNat IP is left
as a TODO; v1 only handles allocation.
createPrivateGateway / deletePrivateGateway / applyACLItemsToPrivateGw
/ applyStaticRoutes remain stubs returning true with a comment marking
them out of scope for OVN VPC v1.
The previous idempotency check skipped the insert whenever a
Logical_Router_Static_Route with the same (ip_prefix, nexthop) existed
anywhere in NB, regardless of which Logical_Router referenced it. That
was wrong as soon as two LRs needed the same default route — only the
first LR ended up with the route attached, and the second LR (e.g.
the new cs-vpc-{id} during implementVpc) silently went without a
default gateway. Lab-confirmed when bringing up an OVN VPC alongside
an existing isolated network sharing the same external gateway.
Now the helper resolves the target LR's static_routes set, intersects
it with rows matching (prefix, nexthop), and only short-circuits when
the existing route is already attached to this LR. Otherwise it
inserts a fresh Static_Route row and mutates the LR's set to include
it — multiple LRs end up with their own row pointing at the same
prefix/nexthop, which is exactly what OVN expects.
PR-2b. Make implement(network) / destroy(network) / applyIps branch on
network.getVpcId() so that a tier of an OVN-backed VPC is wired to the
shared cs-vpc-{vpcId} LR provisioned by implementVpc, instead of the
isolated path that creates a per-network LR and per-network public LS.
implement(network) for VPC tier (network.getVpcId() != null):
- Creates the tier LS cs-net-{netId} stamped with cloudstack_vpc_id
and cloudstack_role=tier external_ids (in addition to the existing
network_id / uuid / zone_id).
- Creates DHCP_Options for the tier (existing helper).
- Calls attachVpcTierToRouter to add the tier-LRP on cs-vpc-{vpcId}
using network.gateway/cidr.
- Calls addVpcTierSnatRule which looks up the VPC SourceNat IP via
IPAddressDao.listByAssociatedVpc(vpcId, true) and programs an snat
NAT row on the VPC LR with logical_ip=tier_cidr,
external_ip=vpc_source_nat_ip. Then refreshes the VPC's gARP via
applyVpcNatAddressesAnnouncement.
- Skips createRouterAndAttachToGuest and applySourceNatForNetwork —
those belong to the isolated path, where a per-network LR and
per-network public LS are still authoritative.
destroy(network) for VPC tier:
- Already drops Load_Balancer rows tagged with cloudstack_network_id
(carried over from PR-0).
- Removes the tier SNAT row on cs-vpc-{vpcId} keyed off
(router, snat, vpc_source_nat_ip, tier_cidr) for every IP currently
flagged isSourceNat() on the VPC.
- Detaches the tier-LRP via the new
OvnNbClient.removeLogicalRouterPort helper (mutates LR.ports and
deletes the LRP row in one transaction; tolerant of missing rows).
- Deletes the tier DHCP options and the tier LS.
- Does NOT touch cs-vpc-{vpcId} or cs-vpc-pub-{vpcId} — both remain
shared across surviving tiers.
applyIps(network):
- The SourceNat IP provisioning block was guarded with
network.getVpcId() == null. CloudStack does not deliver the VPC
SourceNat IP through tier applyIps, so running that block on a VPC
tier would create a duplicate per-tier public-side LRP/LS using the
isolated naming/MAC scheme — re-introducing the dual-public-LRP
problem the PR-1 helpers were designed to avoid.
- The post-loop gARP refresh now branches: VPC tier dispatches to
applyVpcNatAddressesAnnouncement(vpc) so the announcement covers
every SourceNat IP owned by the VPC, not just the (empty) per-tier
set.
OvnNbClient.removeLogicalRouterPort: new helper. Resolves the LRP
UUID by name, mutates the LR's ports column to drop it, deletes the
LRP row. Idempotent. Used only when shrinking a VPC; isolated tear-
down still goes through deleteLogicalRouter which OVSDB-cascades the
LRPs it owned.
VpcDao injected so the tier hooks can resolve the VPC SourceNat IP
without round-tripping through CloudStack's higher-level managers.
Lab-validated end-to-end: built a 2-tier VPC (cs-vpc-3) with tier-a
(10.4.1.0/24) and tier-b (10.4.2.0/24); each tier ended up with a
per-tier LRP on cs-vpc-3 and its own snat row pointing the tier CIDR
at the VPC's SourceNat IP (10.0.111.210). The shared public LRP and
gateway-chassis from PR-2a remained untouched.
…cable
PR-3. Wire the public-IP services (StaticNat, PortForwarding, public-IP
Firewall) so they target the shared cs-vpc-{vpcId} LR and the
cs-vpc-pub-{vpcId} LS when the network is a VPC tier, and so every
artifact carries the cloudstack_vpc_id / cloudstack_public_ip
external_ids needed by the cleanup paths.
PR-1 already migrated the resolution of the LR / public LS / LRP /
public-LRP-LSP names through the network-flavoured helpers, so the
high-level call sites already pointed at the right OVN objects for a
VPC tier. The remaining gaps were three:
1. applyStaticNats: the distributed dnat_and_snat NAT row was always
programmed with buildRouterMac(network.getId(), true) — the
per-network isolated MAC scheme (octet 0xfe). For a VPC tier the
external_mac must match the VPC public LRP's MAC (octet 0xfc, from
buildVpcRouterMac), otherwise ovn-northd refuses to apply the
rewrite locally on the chassis hosting the backend VM and the
forwarding plane silently drops the connection.
Same for the gateway-chassis anchor: every tier should converge on
the VPC's anchor chassis, not run an independent rotation. We now
route through pickAnchorChassisForVpc(vpc) when the network is a
VPC tier so a tier never reanchors the LRP at a different chassis
than implementVpc picked.
2. PortForwarding (programPortForwardingRule): the OVN Load_Balancer
row already lived on the right LR / tier LS (because of the PR-1
helpers), but its external_ids were missing cloudstack_vpc_id and
cloudstack_public_ip, which broke the public-IP-release cleanup
path's ability to sweep VPC-scoped LBs.
3. Firewall (programFirewallRule + ensureFirewallDefaultDeny): the
per-rule and the per-IP default-drop ACLs on the public LS were
missing cloudstack_vpc_id, leaving us without a single key to
sweep all firewall artifacts of a VPC at shutdownVpc time.
Lab-validated end-to-end on cs-vpc-3 / tier-a (10.4.1.0/24):
- StaticNat: NAT row 10.0.111.211 -> 10.4.1.158 created with
external_mac fa:16:3e:fc:00:03 (VPC scheme), logical_port set to
the NIC UUID, external_ids include cloudstack_vpc_id=3,
cloudstack_network_id=267, cloudstack_public_ip=10.0.111.211.
Existing isolated NAT (network_id=265) keeps the 0xfe MAC scheme
and no vpc_id tag — the two coexist without collision.
- PortForwarding: pf-36-tcp Load_Balancer with vips
10.0.111.212:22 -> 10.4.1.158:22, external_ids include the new
cloudstack_vpc_id=3 / cloudstack_public_ip=10.0.111.212 tags.
- Firewall: fw-37 ACL on cs-vpc-pub-3 with match outport ==
"lsp-lrp-cs-vpc-pub-3" && ... && tcp.dst == 22 (allow-related),
plus the per-IP default-drop fw-default-10.0.111.212. Both rows
carry cloudstack_vpc_id=3.
Cleanup behaviour (existing in PR-2a / PR-2b) is unchanged in code but
now actually catches every artifact:
- shutdownVpc sweeps Load_Balancer rows by cloudstack_vpc_id (PF +
LB), then deletes the public LS (clears Firewall ACLs and
default-drops), then deletes the LR (GCs StaticNat NAT rows along
with the SNAT-per-tier rows from PR-2b).
- destroy(tier) sweeps Load_Balancer rows by cloudstack_network_id
(still works for PF), removes the tier-LRP, deletes the tier LS
and DHCP options, removes only the tier-scoped SNAT row from the
VPC LR. StaticNat NAT rows tagged with cloudstack_network_id
survive on the VPC LR until the IP itself is released — the
StaticNat revoke callback is the canonical removal path.
Out of scope for this PR (deferred per the agreed VPC-v1 plan):
- StaticRoute, PrivateGateway, Site2SiteVpn, RemoteAccessVpn.
- Adding Firewall to the seeded OVN VPC offering and the OVN VPC
tier offering — that lands in PR-6 (offerings/seed).
PR-4. Make applyNetworkACLs carry enough provenance on every OVN ACL
row so cleanup paths can sweep VPC-scoped ACLs by VPC id, and fix the
ACL-sweep helper so it stops trying to free ACLs that live on a
different LS.
programNetworkAclRule:
- Adds cloudstack_acl_id (the CloudStack network_acl list id) and
cloudstack_acl_number (the rule's user-visible ordering field) to
every per-rule OVN ACL on the tier guest LS.
- Adds cloudstack_vpc_id when the network is a VPC tier so
shutdownVpc / future audits can sweep ACLs by VPC id.
ensureNetworkAclDefaultDeny:
- Same cloudstack_vpc_id tag on the per-direction default-drop ACLs.
OvnNbClient.removeAclsOnLsByExternalId:
Two distinct fixes that surfaced together at runtime when binding the
default_allow ACL list to a VPC tier whose public IP already had a
Firewall ACL on the VPC public LS:
1. Operation order: the previous body interleaved the per-row
`delete ACL` with the matching `mutate Logical_Switch.acls
DELETE`. OVSDB transactions run in declaration order, so the
delete fired before the strong-reference from LS.acls was cleared
and OVSDB returned "referential integrity violation: cannot delete
ACL row because of N remaining reference(s)". The new order is one
bulk mutate (drop every UUID from LS.acls) followed by the per-row
deletes.
2. LS scoping: findAclUuidsByExternalIds searches the global ACL
table, but ACLs of one network can legitimately live on multiple
LSes — public-IP firewall ACLs created by applyFWRules sit on the
VPC public LS yet are tagged with cloudstack_network_id=<tier_id>
because that is the network the IP is associated with. Wiping by
cloudstack_network_id from the tier guest LS used to find the
firewall ACLs too and tried to free-delete them, leaving them
referenced from the public LS and tripping the same OVSDB strong-
ref guard. The new helper lsAclSet() reads the target LS's acls
set and intersects it with the candidate UUIDs, so only ACLs
actually attached to the LS we are sweeping are removed.
Lab-validated end-to-end on cs-vpc-3 / tier-a (10.4.1.0/24):
- Pre-fix: replaceNetworkACLList tier-a → default_allow failed with
"cannot delete ACL row af8eac90-... because of 1 remaining
reference(s)" (the firewall ACL on cs-vpc-pub-3).
- Post-fix: same call returns success. cs-net-267.acls now lists
nacl-3 / nacl-4 (the two default_allow rules at OVN priorities
999 / 998) plus nacl-default-to-lport / nacl-default-from-lport
at priority 1, all tagged with cloudstack_vpc_id=3 and
cloudstack_acl_id=2. cs-vpc-pub-3.acls (firewall) is unchanged.
Out of scope for this PR (deferred per the agreed VPC-v1 plan):
- PrivateGateway-bound ACLs.
PR-5a. Stamp every OVN Load_Balancer row created by programLBRule with
two new external_ids fields:
- cloudstack_lb_scheme: hard-coded to "Public" in this PR. PR-5b will
switch this to "Internal" for the tier-VIP variant. Capabilities
still advertise LbSchemes=Public, so CloudStack's LB manager will
not route an Internal-scheme rule through this code path until 5b
flips the capability.
- cloudstack_vpc_id: the VPC id of the tier hosting the rule, when the
rule is on a VPC tier (network.getVpcId() != null). Mirrors the same
tag that StaticNat / PortForwarding / Firewall / NetworkACL already
carry from PR-3 / PR-4 so shutdownVpc and any future
scheme-/VPC-scoped sweep have a single key to filter on.
The high-level routing was already correct on VPC tiers thanks to
PR-1's getRouterNameForNetwork helper: programLBRule attaches the LB
to cs-vpc-{vpcId} and the tier LS, with hairpin_snat_ip pointing at
the public VIP. This change is purely metadata — no functional change
on isolated networks.
Lab-validated end-to-end on cs-vpc-3 / tier-a (10.4.1.0/24):
- createLoadBalancerRule on a public IP allocated to the VPC,
assignToLoadBalancerRule with vm-tier-a (10.4.1.158).
- OVN Load_Balancer lb-38-tcp created with vips
10.0.111.213:22 -> 10.4.1.158:22, protocol tcp, attached to
cs-vpc-3 + cs-net-267, external_ids include
cloudstack_lb_scheme=Public, cloudstack_vpc_id=3,
cloudstack_network_id=267, cloudstack_lb_ip=10.0.111.213.
- Existing isolated LBs (cs-router-265) keep their previous
external_ids set without cloudstack_vpc_id — the two coexist.
PR-5b. Allow OVN-backed VPC tiers to host Internal Load_Balancer rules
without requiring a separate appliance VM. Internal LB rules share
the same applyLBRules / programLBRule pipeline as Public LB; only the
VIP source, the hairpin SNAT anchor, and a couple of validation gates
diverge.
Capability:
- Network.Capability.LbSchemes now advertises both
"Public,Internal", letting CloudStack route Internal-scheme rules to
this provider when the offering enables internal_lb=1.
programLBRule:
- VIP resolution unified through rule.getSourceIp(): the field is
populated for both schemes (Public uses the public IP allocation;
Internal uses the private VIP allocated from the tier CIDR), so the
LB row is keyed off a single accessor instead of branching on
rule.getLb().getSourceIpAddressId().
- options.hairpin_snat_ip is set to:
* the VIP itself for Public LB (unchanged behaviour),
* the tier gateway IP for Internal LB. The public IP doesn't
exist on the VPC LR for Internal LB; using the tier gateway
keeps OVN happy ("must be an IP we own") and produces the
correct SNAT when a VM in the tier hits its own VIP.
- external_ids.cloudstack_lb_scheme is now stamped per-rule
("Public" or "Internal") instead of being hard-coded.
validateLBRule:
- For Internal rules, reject when the VIP would map to a public-IP
allocation (rule.getLb().getSourceIpAddressId() resolves to an
IPAddressVO) — that is the operator-visible "you wanted Public, not
Internal" mistake. The tier-CIDR check is left to CloudStack's
CIDR-vs-supernet validation, which already runs upstream.
- For Public rules, the SourceNat-IP rejection from PR-5a is preserved
unchanged.
Lab-validated end-to-end on cs-vpc-3 / tier-b (10.4.2.0/24):
- createLoadBalancer scheme=Internal, sourceipaddress=10.4.2.100,
sourceport=22, instanceport=22, networkid=tier-b.
- assignToLoadBalancerRule with vm-tier-b (10.4.2.129).
- OVN Load_Balancer lb-39-tcp:
vips: 10.4.2.100:22 -> 10.4.2.129:22
protocol: tcp
options: hairpin_snat_ip=10.4.2.1 (tier-b gateway)
ip_port_mappings: 10.4.2.129 -> <nic-uuid>:10.4.2.1
external_ids: cloudstack_lb_scheme=Internal,
cloudstack_lb_ip=10.4.2.100,
cloudstack_vpc_id=3,
cloudstack_network_id=269,
cloudstack_lb_rule_id=39,
cloudstack_lb_kind=loadbalancer
attached on: cs-vpc-3 (LR) + cs-net-269 (tier-b LS) only.
- Public LB lb-38-tcp from PR-5a continues to work with hairpin
pointing at the public IP and scheme=Public; the two LB rows
coexist on the same VPC LR without interference.
Out of scope for this PR (deferred):
- Cross-tier Internal LB membership: CloudStack's
assignToLoadBalancerRule rejects backends from a different network
than the LB. OVN would handle the data-plane path correctly (LB
is on the shared VPC LR), but enabling that flow is a separate
CloudStack-side change.
- L7 features (HTTP path matching, SSL offload, cookie stickiness)
remain rejected by validateLBRule.
PR-6. Make a fresh install converge on the same OVN offering shape that the lab was costuring by hand via SQL inserts. Three changes, all in the boot-time seed path: 1. NetworkOrchestrator (apache#8 - DefaultNATOVNNetworkOffering, isolated): - Add Service.UserData -> Provider.ConfigDrive. The OVN data-plane has no metadata service; the OVN provider does NOT advertise UserData in OvnElement.initCapabilities(), so the offering must bind UserData explicitly to ConfigDrive (ISO9660 attached at boot) for cloud-init to find a metadata source. Without this, every VM deployed on an isolated OVN network has no userdata. 2. NetworkOrchestrator (apache#9 - DefaultNATOVNNetworkOfferingForVpc, VPC tier): - Add Service.UserData -> Provider.ConfigDrive (same reason). - Add Service.Firewall -> Provider.Ovn. CloudStack issue apache#8863: VPC tiers must support firewall rules on Public IPs alongside NetworkACL on the tier subnet. PR-3 already wired applyFWRules to the VPC public LS via the OVN provider, so the offering can legitimately bind Firewall to Ovn now. - Set offering.setInternalLb(true) right after createNetworkOffering and persist. Without this CloudStack's createLoadBalancer scheme=Internal API rejects the request with "Scheme Internal is not supported by the network offering" - and the OVN provider can deliver Internal LB natively as of PR-5b (tier-CIDR VIP attached to the VPC LR + tier LS, hairpin_snat_ip on the tier gateway). 3. VpcManagerImpl: new auto-seeded VpcOffering "VPC offering with OVN - NAT Mode" (VpcOffering.DEFAULT_VPC_NAT_OVN_OFFERING_NAME). Bind every service the OVN provider can satisfy natively to Provider.Ovn: Vpc, SourceNat, StaticNat, PortForwarding, Lb, NetworkACL, Firewall, Dhcp, Dns, Gateway Bind UserData to ConfigDrive (same rationale as the tier offerings). Skip Vpn (RemoteAccessVpn / Site2SiteVpn are out of scope for the OVN VPC v1 - no service VM yet). NetworkMode is NATTED. Pairs with DefaultNATOVNNetworkOfferingForVpc on the tier side. The new VpcOffering constant (DEFAULT_VPC_NAT_OVN_OFFERING_NAME) lives on the api module's VpcOffering interface alongside the existing NSX/Netris constants so the seed code can reference it without a provider-private constants file. Out of scope (deferred per agreed VPC-v1 plan): - Service.Vpn (no OVN VPN appliance yet). - PrivateGateway service binding. - Tungsten / Nuage / etc. equivalents - other gurus seed their own. Lab note: existing offerings on installations created before this PR are not migrated by the seed (the seed guard skips when the offering already exists). Operators that want the new services on already- provisioned offerings have to add the service mappings manually: INSERT INTO ntwk_offering_service_map (network_offering_id, service, provider, created) VALUES (<offering_id>, 'UserData', 'ConfigDrive', NOW()), (<offering_id>, 'Firewall', 'Ovn', NOW()); plus, for the VPC tier offering only: UPDATE network_offerings SET internal_lb=1 WHERE name='DefaultNATOVNNetworkOfferingForVpc';
ensureFirewallDefaultDeny installed only the inbound `to-lport` drop ACL on the public Logical_Switch. That kept unsolicited inbound traffic out — but it also dropped the reply leg of any flow the VM itself initiated through a static-NAT public IP, because OVN's ACL conntrack zone never saw the outbound side and therefore never marked the connection as established/related. Concretely: a VM behind dnat_and_snat (10.0.111.209 → 10.1.1.66) pings 8.8.8.8. lr_out_snat rewrites the source to 10.0.111.209 on the LR's egress, and the packet leaves via the localnet port. The reply 8.8.8.8 → 10.0.111.209 enters the public LS via the localnet, needs to traverse the LS to reach the LR, and the only ACL it matches is the per-IP default-drop at priority 100 — because the matching `allow-related` (e.g. fw-19 for tcp/22) was scoped to the specific protocol/port the operator opened. Reply is dropped, ping times out, ssh handshakes never complete. Fix: alongside the inbound default-drop, install an `allow-related` ACL in the `from-lport` direction whose match is `inport == "lsp-lrp-cs-pub-<id>" && ip4 && ip4.src == <publicIp>`. This catches every packet leaving the LS toward the localnet whose source is the public IP — i.e. SNAT'd / dnat_and_snat'd VM-initiated egress — and commits it to the LS's ACL conntrack zone. The reply then matches ct.est && ct.rpl in the inbound pipeline, bypasses the default-drop, and reaches the LR's lr_in_unsnat / lr_in_dnat for the NAT reversal, which finally delivers the packet to the VM. This restores parity with the CloudStack VR-backed semantics: a public IP that has firewall rules attached only restricts unsolicited inbound traffic; replies to flows the VM initiated continue to work. External_ids on the egress ACL carry cloudstack_fw_default_egress=true in addition to cloudstack_fw_default=true / cloudstack_fw_ip / cloudstack_network_id / (cloudstack_vpc_id when applicable), so the sweep paths (cleanupPublicIpArtifacts, applyFWRules revoke, shutdownVpc) match the new ACL by the same external_ids tags they already match the inbound default-drop on. Lab repro: - vm-01 (10.1.1.66, static-NAT to 10.0.111.209) had a TCP/22 firewall rule on 10.0.111.209. - Before: VM's outbound ICMP reached 8.8.8.8, the reply arrived on the kvm host's eth1 (verified by tcpdump), but ovs-appctl ofproto/trace ended with `Datapath actions: drop`, the drop landing in the OVN logical flow at table=46 reg0=0x200/0x200 (cookie d3f2c04f, source northd.c:6743 ls_out_acl_eval) — i.e. the public LS ACL evaluation rejected the packet. - After: the new from-lport allow-related commits the outbound to OVN's ACL conntrack; reply matches ct.est && ct.rpl; the trace walks all the way through lr_in_unsnat / lr_in_dnat / lr_in_routing / lr_out and the packet appears on vnet4 (the VM's vif). ping completes end-to-end. Existing offerings on installations created before this change need the new ACL added to every public IP that already has a default-drop. This can be done in one of two ways: (a) revoke + re-add any one firewall rule on the IP — applyFWRules will refresh both ACLs via ensureFirewallDefaultDeny on every call; or (b) one-time `ovn-nbctl --may-exist acl-add cs-pub-<netId> from-lport 100 ... allow-related` per IP.
Regression introduced by the addStaticRoute idempotency fix that
scoped duplicate detection to the target LR. The new lookup selects
existing Static_Route rows by (ip_prefix, nexthop) and intersects
their UUIDs with the LR's static_routes set, but it constructed the
SELECT operation without listing _uuid as a returned column:
OVSDB_OPS.select(srTable)
.where(srPrefixCol.opEqual(ipPrefix))
.and(srNexthopCol.opEqual(nexthop))
.build();
The OVSDB Java client returns a Row populated only with the columns
explicitly listed in `.column(...)` calls — no list, no columns. The
follow-up `row.getColumn(srTable.column("_uuid", UUID.class))` then
returned null, and the very next `.getData()` raised:
Cannot invoke "Column.getData()" because the return value of
"Row.getColumn(ColumnSchema)" is null
Lab fallout: every VM deploy that touched a network whose LR already
carried a default route — i.e. every isolated network and every VPC
that had been provisioned — failed to start. The error surfaced as
"OVN OVN_Northbound operation against tcp:...:6641 failed" and the
deployer marked the entire DataCenter unreachable, so retries on the
other host hit the same failure path and the VM ended in Error.
Fix: select _uuid, ip_prefix and nexthop explicitly using a single
ColumnSchema instance shared with the row lookup. The shared
instance also avoids a separate gotcha where a recreated
ColumnSchema is not equal-by-reference to the one used in the
SELECT, which can also break Row.getColumn lookups in this client.
Defensively, the new code also tolerates a null Column return from
getColumn (skips the row instead of NPE-ing).
Restores the half of the public-IP firewall semantics we can enforce statelessly without re-introducing the reply-traffic regression that made commit d8082c4 turn ensureFirewallDefaultDeny into a no-op. The previous full default-drop matched any inbound packet with ip4.dst == <publicIp> and dropped both unsolicited probes AND replies to VM-initiated outbound flows (ping 8.8.8.8, DNS, HTTPS) because OVN bypasses ct_next on router/localnet LSPs and the public LS has no way to distinguish a reply from a fresh inbound packet. Splitting the drop on icmp4.type fixes the most user-visible piece: - icmp4.type == 8 : echo request, only sent by clients trying to reach the public IP from outside. - icmp4.type == 0 : echo reply, only seen as the return leg of an ICMP echo a VM sent outbound through the same public IP. Drop the former, leave the latter alone. The match clauses out non- ICMP traffic entirely, so TCP/UDP replies are unaffected. With the per-rule allow-related ACLs at priority 1000 still in place, an operator opening ICMP with a CloudStack FirewallRule still passes a legit echo request through. Lab-verified on the SourceNAT IP 10.0.111.207 of network 273: - Before: ping 10.0.111.207 from outside got reply (LR's lr_in_ip_input auto-replies to ICMP echo on the LRP's own IP). - After: ping 10.0.111.207 times out; per-rule TCP/22 still works; VM outbound ping 8.8.8.8 still completes (reply is type==0, bypasses the drop). What stays unsolved here: - Unsolicited TCP/UDP inbound to the public IP. The LR's lr_in_ip_input still answers (RST for TCP, ICMP unreachable for UDP, etc.). Closing those requires moving firewall enforcement to Logical_Router policies — the LR's conntrack zone IS populated by ct_dnat / ct_snat, so policies can use ct.new vs ct.est correctly. That refactor is tracked separately.
Without this, an operator standing up a new OVN-backed zone has to
either pick a different isolation method (forcing the OVN provider
into a no-op posture) or fall back to provisioning the zone via the
API and registering the OVN provider with addOvnProvider out-of-band.
Both Netris and NSX have a dedicated wizard step for the same flow;
this commit gives OVN the equivalent.
ZoneWizardPhysicalNetworkSetupStep.vue and PhysicalNetworksTab.vue:
add "OVN" to the isolation-method dropdown for KVM zones (matching
how "Netris" is gated to KVM today). Other hypervisors keep their
existing options unchanged.
ZoneWizardNetworkSetupStep.vue:
- new isOvnZone computed (mirrors isNetrisZone) flips on whenever any
declared physical network selects "OVN".
- new "ovn" step inserted into allSteps() right after the netris
step, between the physical-network grid and the public-traffic
configuration.
- new ovnFields list mirrors AddOvnProviderCmd's parameters:
name, nbConnection (required)
sbConnection, externalBridge, localnetName,
cacertpath, clientcertpath, clientprivatekeypath (optional).
Required-vs-optional matches the Java side; the operator can leave
the TLS / bridge-mapping fields blank for simple labs.
- ovnSetupDescription points at a new i18n key that explains the
bridge mapping requirement on the ovn-controller side.
ZoneWizardLaunchZone.vue:
- stepData.isOvnZone is set in the existing physical-network creation
loop using the same gating as Netris/NSX (only when the OVN physnet
carries public or guest traffic - storage-only physnets do not
need a provider entry).
- stepNetworkingProviderOrStorageTraffic dispatches to the new
stepAddOvnProvider when the zone is OVN.
- stepAddOvnProvider posts to addOvnProvider with the prefillContent
collected by the wizard. Optional fields are only forwarded when
the operator filled them in, so a minimal OVN deployment that uses
default ovsdb ports / no TLS does not have to clear ghost values.
- addOvnProvider is the postAPI wrapper, parallel to addNetrisProvider
/ addNsxController.
ui/public/locales/en.json:
- 9 new label.* / message.* keys for the OVN provider step (form
labels, install-wizard tooltips, the description shown on the step,
and the "Add OVN Provider" header used by the launch progress
panel). Tooltip text spells out the connection-string format
(tcp/ssl + host + port) and the relationship between the localnet
name and ovn-bridge-mappings on the hypervisors.
Member
|
cool, great job @msinhore ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces OVN (Open Virtual Network) as a first-class guest
network provider for CloudStack, alongside VirtualRouter, NSX, Netris,
etc. It plugs OVN's Northbound DB directly into the network orchestration
flow, so guest networks (isolated and VPC) and their L3 services are
realised as native OVN logical objects (LS, LR, LRP, NAT, ACL,
Load_Balancer, DHCP_Options) instead of going through a system VM.
The plugin lives under
plugins/network-elements/ovn/and is opt-in: afresh installation continues to behave exactly as before until an OVN
provider is added to a physical network and the OVN-aware offerings are
enabled.
Scope
Provider lifecycle & wiring
OvnProvider/OvnProviderVO/ DAO and aOvnProviderService(add / list / delete / health-check).
a startup health probe against the OVN NB endpoint.
(new "OVN Provider" step that captures the NB endpoint, certs and
gateway chassis), and in the physical-network isolation method picker.
upgrade so a fresh install can deploy OVN networks immediately.
Guest network (isolated)
implement / shutdown / releasefor isolated networks:DHCP_Options(no virtual router VM).prepare/release.VPC
Logical_Routerper VPC; each tier becomes an LRP onthat LR.
implementVpc / shutdownVpc / updateVpcSourceNatIpcover theVPC lifecycle and active-IP swap.
NetworkACLrules enforced via OVN ACLs scoped to the target tier's LS.present; otherwise the per-network LR.
on the LR.
L3 services
dnat_and_snatrows on the LR with gateway-chassisanchoring; rows are detached from
Logical_Router.natbefore delete toavoid dangling references.
Load_Balancer(not NAT), somultiple PF rules on the same public IP coexist cleanly.
health_check, supported algorithmsroundrobinandsource(declaredvia
Capability.SupportedLBAlgorithms).unsolicited ICMP echo to public IPs is dropped by default (firewall
default-deny semantic for ping).
Logical_Router.natontoLoad_Balancerto remove the row-collision foot-gun on the same publicIP.
Robustness fixes accumulated during bring-up
addStaticRoutescoped per-router (NPE fix when LR alreadycarried routes from a previous attempt).
Load_Balancercleanup tightened with a protocol gate; LB rules on anetwork's SourceNat IP are rejected up front.
default-deny ACL was matching legitimate replies because OVN's
router/localnet LSPs bypass
ct_next— see Javadoc onOvnElement.ensureFirewallDefaultDeny).Architecture notes
Full TCP/UDP unsolicited-inbound enforcement requires moving from public-LS
ACLs to
Logical_Router_Policy(LR's CT zone is populated byct_dnat/ct_snat, the LS's is not for router-attached traffic). Trackedas a follow-up — the current ACL-based path is documented inline.
all OVN-native. The "no-router" offerings flavour exposes this directly.
upstream UI) reads
SupportedLBAlgorithmsfromlistNetworks, so OVN's{roundrobin, source}is reflected automatically without any UI patchhere.
Compatibility
zones, networks and offerings are untouched until an admin enables OVN
on a physical network.
ovn_*tables and the OVN providerrow); no destructive migrations.
addOvnProvider / listOvnProviders / deleteOvnProvidercommands; no changes to existing API contracts.
upgrade-*-to-*.sqlfor the OVN tablesand the seeded offerings.
Types of changes
How Has This Been Tested?
Bring-up environment: KVM + libvirt, OVN 24.x (NB on tcp:6641, SB on
tcp:6642), single management server, two compute hosts acting as
gateway-eligible chassis. Apache CloudStack
mainplus this PR.Provider & offerings
addOvnProvideragainst a healthy and a dead NB endpoint (rejectionpath verified).
create VPC against each.
Isolated network
implement / shutdown / restartlifecycle.22.04/24.04 cloud images via ConfigDrive).
ping replies pass.
same public IP).
VPC
updateVpcSourceNatIpswap with active rules in place.shutdownVpccleanup.Negative paths
addStaticRouteno longer NPEs when the LR already carries routesfrom a previous attempt.
Logical_Router.nat: nowdetached first.
Pending runtime tests (will run before final review):
destroy(tier)cleanup (LB row + SNAT row + LRP).shutdownVpcwith multiple tiers + LBs simultaneously active.Follow-ups (separate PRs)
ui-lb-algorithm-from-capability).(architectural fix for full TCP/UDP unsolicited-inbound enforcement).
encryptUnderlayprovider flag (design done,implementation as next epic).