feat(connectivity): add active/passive CARP support for firewall high…

.gitattributes

@@ -0,0 +1,5 @@
+* text=auto
+
+# These run through bash, including from a local-exec provisioner on Windows checkouts.
+# CRLF makes them fail with a syntax error on the first line.
+*.sh text eol=lf

docs/architecture.md

@@ -111,6 +111,29 @@ Both interfaces are DHCP clients and pick up the fixed addresses STACKIT assigns
 > [!WARNING]
 > After the first apply the web GUI answers on the public IP to the whole internet, and it still carries the `root` password shipped in the image, which is identical on every copy. Port 443 is open, 80 and 22 are not, and there is no security group in front of the WAN interface. Pushing the policy closes it, so do not leave a deployment sitting between the two applies — see [Configure OPNsense firewall](getting-started.md#configure-opnsense-firewall). Binding the GUI to the LAN interface only (**System → Settings → Administration → Listen Interfaces**) is the alternative that does not depend on the ruleset.
 
+#### High availability (optional)
+
+A single appliance is a single point of failure for every corporate landing zone: it is their default route. Setting `connectivity.firewall.ha` turns it into an active/passive CARP pair.
+
+| | Primary | Backup |
+|---|---|---|
+| Availability zone | `firewall.zone` | `firewall.ha.backup_zone` (must differ) |
+| LAN address | `.4` of `lan_network_range` | `.5` |
+| WAN address | `.4` of `wan_network_range` | `.5` |
+| Public IP | own, static | own, static |
+| CARP advskew | 0 (wins the election) | 100 |
+
+A CARP virtual IP on the LAN — `.6` of `lan_network_range` by default — replaces the primary's LAN address in the `firewall_next_hop_ip` output, so the landing zone routes point at the VIP instead of at a node. Failover moves the VIP to the surviving node in about a second and the routes never change.
+
+**CARP runs in unicast mode** (OPNsense >= 24.7). Not a preference: the STACKIT fabric delivers traffic *to* the CARP virtual MAC but drops advertisements sourced *from* it, so multicast CARP ends in a split brain. The module refuses to configure a node whose OPNsense has no unicast `peer` field.
+
+Two things are pushed outside the OPNsense provider, because it only ever talks to the primary:
+
+- `modules/connectivity/scripts/configure-ha.sh` writes the node-local half into each appliance during apply: the CARP VIP, `advskew`, the pfsync peer, and the XMLRPC sync target. These are exactly the settings the config sync does not replicate. It authenticates with the appliance login, so it works on the backup, which has no API key and never needs one.
+- `modules/firewall-config/scripts/sync-ha-peer.sh` replicates the policy to the backup after every change. OPNsense's XMLRPC sync only fires on GUI saves, never on API writes, so without this the backup runs an empty ruleset and black-holes traffic the moment it becomes CARP master.
+
+The `fw_cluster` alias and the `allow-fw-carp` / `allow-fw-pfsync` rules that let the two nodes talk are injected into `firewall_config` automatically when HA is on, sequenced at 90 and 91 — ahead of every rule the example policy ships. They are not in the `.tfvars` because a `block-lz-to-lz` rule placed above them silently kills the election and the state sync.
+
 ### Landing Zone
 
 Instantiated once per workload/environment via `for_each` over the `landing_zones` variable. Each instance creates a fully isolated STACKIT project containing:

docs/getting-started.md

@@ -11,6 +11,7 @@ This guide walks you through deploying the STACKIT Landing Zone from scratch.
 - **Owner permissions** on the STACKIT organization
 - **STACKIT CLI** installed ([Installation guide](https://github.com/stackitcloud/stackit-cli/blob/main/INSTALLATION.md))
 - **OpenTofu** (>= 1.10) or **Terraform** (>= 1.10) installed
+- **`bash`, `curl` and `jq`** on the machine running OpenTofu, for the Hub-Spoke + Firewall flavour only. The API key bootstrap and the HA configuration are shell provisioners; `jq` is used by the HA scripts
 
 > [!NOTE]
 > This guide uses `tofu` commands throughout. If you are using Terraform, replace `tofu` with `terraform` — all commands work identically.
@@ -27,7 +28,7 @@ Three ready-to-use configurations are provided in `src/config/`:
 
 Choose the flavour that matches your requirements and adjust the corresponding `.tfvars` file before deployment (step 7). At a minimum, update `owner_email`, `organization_id`, `company_name`, and `company_code`.
 
-The firewall flavour takes one extra step: the appliance boots unconfigured and its policy is pushed in a second apply, from the `firewall_config` block that ships commented out in the same `.tfvars` file. Until then it filters nothing and its web GUI is reachable from the internet — see [Configure OPNsense firewall](#configure-opnsense-firewall).
+The firewall flavour takes one extra step: the appliance boots unconfigured and its policy is pushed in a second apply, from the `firewall_config` block that ships commented out in the same `.tfvars` file. Until then it filters nothing and its web GUI is reachable from the internet — see [Configure OPNsense firewall](#configure-opnsense-firewall). It deploys a single appliance by default, which is the default route of every corporate landing zone and therefore a single point of failure; the commented `connectivity.firewall.ha` block turns it into an active/passive CARP pair — see [Make the firewall highly available](#make-the-firewall-highly-available).
 
 Both hub-spoke flavours can additionally terminate a site-to-site IPsec VPN in the hub. It is disabled by default — see the commented `connectivity.vpn` block in the `.tfvars` file and [Site-to-Site VPN](architecture.md#site-to-site-vpn-optional). If you deploy the firewall flavour, read [what traffic the firewall actually sees](architecture.md#what-goes-through-the-firewall) before relying on it for VPN inspection.
 
@@ -309,6 +310,28 @@ tofu apply -var firewall_bootstrap=true   # writes it to the Secrets Manager
 rm src/.firewall-api-credentials.json
 ```
 
+### Make the firewall highly available
+
+A single appliance is the default route of every corporate landing zone, so losing it takes the platform offline. Uncomment the `ha` block in `connectivity.firewall` to deploy a second appliance in another availability zone as an active/passive CARP pair:
+
+```hcl
+ha = {
+  backup_zone = "eu01-1" # must differ from firewall.zone
+}
+```
+
+`tofu apply` then does the rest. It deploys the backup, logs into both appliances to write their node-local CARP settings (backup first, primary last, so the primary wins the election), and adds the `fw_cluster` alias plus the `allow-fw-carp` and `allow-fw-pfsync` rules to the policy — those are injected automatically, there is nothing to add to the `.tfvars`.
+
+Two consequences are worth knowing before you enable it:
+
+- **The landing zone routes change.** `firewall_next_hop_ip` becomes the CARP virtual IP (`.6` of `lan_network_range`) instead of the primary's LAN address, so every corporate landing zone gets a rewritten default route in the same apply. Expect a short interruption while those routes are updated.
+- **The public IP does not fail over.** STACKIT binds a public IP 1:1 to a NIC. Egress from the backup is translated to the backup's own address, and inbound port forwards stay on the primary until you repoint DNS at `tofu output connectivity_firewall_backup_public_ip`. Long-lived outbound connections break for the duration of an outage; new ones recover in seconds.
+
+What exactly fails over, and what was measured, is in [High availability](architecture.md#high-availability-optional).
+
+> [!IMPORTANT]
+> The backup appliance boots with the same image default password as the primary and its web GUI answers on its own public IP until the policy is pushed. The lockout warning above applies to both nodes.
+
 ### Kubernetes: DNS automation for Gateway API resources
 
 For Gateway API resources (for example Envoy Gateway with `Gateway` + `HTTPRoute`), use DNS records directly via `stackit_dns_record_set` until native provider support for `extensions.dns.gatewayApi` is available.

src/_firewall-bootstrap.tf

@@ -95,3 +95,65 @@ ephemeral "vault_kv_secret_v2" "firewall_api" {
   mount = module.management.secretsmanager_instance_id
   name  = "firewall_api_${replace(var.company_code, "-", "_")}_pltfm_hub_prod"
 }
+
+#################################
+## FIREWALL  HIGH AVAILABILITY ##
+#################################
+
+# CARP and pfsync between the two appliances have to pass ahead of every block rule, and
+# the alias content is the pair's LAN addresses, which only the connectivity module knows.
+# Neither can come from the .tfvars, so both are injected here whenever HA is on. This is
+# not a convenience: block-lz-to-lz (any protocol, network_area to network_area) silently
+# kills the CARP election and the state sync, and a split cluster black-holes traffic.
+#
+# Injected last, so these three keys win over an entry of the same name in the .tfvars.
+locals {
+    firewall_ha_enabled = try(var.connectivity.firewall.ha, null) != null
+
+    firewall_ha_aliases = local.firewall_ha_enabled ? {
+      fw_cluster = {
+        type        = "host"
+        enabled     = true
+        description = "LAN addresses of the firewall HA pair"
+        content     = try(module.connectivity[0].firewall_cluster_lan_ips, [])
+        update_freq = null
+        stats       = false
+      }
+    } : {}
+
+    firewall_ha_rule_defaults = {
+      sequence           = 100
+      enabled            = true
+      action             = "pass"
+      direction          = "in"
+      interfaces         = ["lan"]
+      protocol           = "any"
+      ip_protocol        = "inet"
+      quick              = true
+      source_net         = "fw_cluster"
+      source_port        = null
+      source_invert      = false
+      destination_net    = "fw_cluster"
+      destination_port   = null
+      destination_invert = false
+      log                = false
+      description        = null
+    }
+
+    # 90/91 puts them ahead of every landing zone rule, which start at 100 — block-lz-to-lz
+    # in particular. They sit behind the two floating GUI rules at 10/20, which match TCP on
+    # port 443 only and can therefore never swallow CARP or pfsync.
+    firewall_ha_rules = local.firewall_ha_enabled ? {
+      allow-fw-carp = merge(local.firewall_ha_rule_defaults, {
+        sequence    = 90
+        protocol    = "CARP"
+        description = "Unicast CARP advertisements between the HA pair"
+      })
+
+      allow-fw-pfsync = merge(local.firewall_ha_rule_defaults, {
+        sequence    = 91
+        protocol    = "PFSYNC"
+        description = "pfsync state replication between the HA pair"
+      })
+    } : {}
+}

src/config/hub-and-spoke-firewall.tfvars

@@ -92,6 +92,19 @@ connectivity = {
     name              = "opnsense-26.1"
     lan_network_range = "10.0.2.0/28"
     wan_network_range = "10.0.2.16/28"
+
+    # Optional: active/passive CARP pair across availability zones. Uncomment to enable
+    #
+    # A second appliance is deployed in backup_zone. Node IPs default to .4/.5 of each
+    # range, the LAN VIP to .6 — the VIP replaces the primary's LAN IP as the next hop the
+    # landing zone routes point at, so a node failure never touches the routes (measured
+    # failover: about one second for new connections). The CARP shared secret is generated
+    # by the module, and the fw_cluster alias plus the CARP and pfsync rules are added to
+    # firewall_config automatically. See docs/architecture.md#high-availability-optional.
+    #
+    # ha = {
+    #   backup_zone = "eu01-1"
+    # }
   }
 
   # Optional: site-to-site IPsec VPN terminating in the hub. Uncomment to enable

src/main.tf

@@ -51,6 +51,12 @@ module "connectivity" {
   firewall            = var.connectivity.firewall
   vpn                 = var.connectivity.vpn
   vpn_pre_shared_keys = var.vpn_pre_shared_keys
+
+  # Only read when connectivity.firewall.ha is set, to log into both appliances and push
+  # their node-local CARP settings. Unset endpoint means the primary's public IP.
+  firewall_admin_endpoint = try(var.firewall_config.endpoint, null)
+  firewall_admin_username = var.firewall_admin_username
+  firewall_admin_password = var.firewall_admin_password
 }
 
 #####################
@@ -61,12 +67,19 @@ module "firewall_config" {
   source = "./modules/firewall-config"
   count  = local.firewall_config_enabled ? 1 : 0
 
-  aliases       = var.firewall_config.aliases
+  aliases       = merge(var.firewall_config.aliases, local.firewall_ha_aliases)
   routes        = var.firewall_config.routes
-  rules         = var.firewall_config.rules
+  rules         = merge(var.firewall_config.rules, local.firewall_ha_rules)
   outbound_nat  = var.firewall_config.outbound_nat
   port_forwards = var.firewall_config.port_forwards
 
+  # OPNsense never replicates API-written config on its own, so the policy is pushed to
+  # the peer explicitly after every change.
+  endpoint       = local.firewall_endpoint
+  ha_sync        = local.firewall_ha_enabled
+  admin_username = var.firewall_admin_username
+  admin_password = var.firewall_admin_password
+
   depends_on = [terraform_data.firewall_api_bootstrap, module.connectivity]
 }
 
@@ -152,8 +165,8 @@ module "landing_zone" {
   labels                = var.labels
   role_assignments      = each.value.role_assignments
   network_prefix_length = each.value.network_prefix_length
-  ipv4_nameservers     = try(module.connectivity[0].network_area_nameservers, null)
-  custom_roles         = each.value.custom_roles
-  observability        = each.value.observability
-  firewall_next_hop_ip = var.connectivity != null && var.connectivity.firewall != null ? module.connectivity[0].firewall_next_hop_ip : null # if firewall is enabled, pass the next hop IP to the landing zones for route configuration
+  ipv4_nameservers      = try(module.connectivity[0].network_area_nameservers, null)
+  custom_roles          = each.value.custom_roles
+  observability         = each.value.observability
+  firewall_next_hop_ip  = var.connectivity != null && var.connectivity.firewall != null ? module.connectivity[0].firewall_next_hop_ip : null # if firewall is enabled, pass the next hop IP to the landing zones for route configuration
 }

src/modules/connectivity/3-external-network.tf

@@ -42,6 +42,7 @@ resource "stackit_network" "wan" {
   project_id       = stackit_resourcemanager_project.this.project_id
   name             = "wan_network"
   ipv4_prefix      = var.firewall.wan_network_range
+  ipv4_nameservers = local.network_area_nameservers
   routing_table_id = stackit_routing_table.wan.routing_table_id
   routed           = true
 }
@@ -52,7 +53,7 @@ resource "stackit_network_interface" "wan" {
   name       = "vtnet0_wan"
   project_id = stackit_resourcemanager_project.this.project_id
   network_id = stackit_network.wan[0].network_id
-  ipv4       = coalesce(var.firewall.wan_ip, cidrhost(var.firewall.wan_network_range, 4))
+  ipv4       = local.firewall_wan_ip
   security   = false
 }
 
@@ -61,4 +62,30 @@ resource "stackit_public_ip" "wan-ip" {
 
   project_id           = stackit_resourcemanager_project.this.project_id
   network_interface_id = stackit_network_interface.wan[0].network_interface_id
+}
+
+#################
+## BACKUP NODE ##
+#################
+
+# The backup's own public IP serves three purposes: the appliance is reachable for the
+# per-node HA configuration before any policy exists, egress NAT keeps working when the
+# backup is MASTER (translated to its own address), and inbound can be failed over
+# manually by repointing DNS. The primary's public IP does NOT move automatically —
+# STACKIT binds a public IP 1:1 to a NIC and has no floating construct.
+resource "stackit_network_interface" "wan_backup" {
+  count = local.firewall_ha_enabled ? 1 : 0
+
+  name       = "vtnet0_wan_backup"
+  project_id = stackit_resourcemanager_project.this.project_id
+  network_id = stackit_network.wan[0].network_id
+  ipv4       = local.firewall_backup_wan_ip
+  security   = false
+}
+
+resource "stackit_public_ip" "wan-ip_backup" {
+  count = local.firewall_ha_enabled ? 1 : 0
+
+  project_id           = stackit_resourcemanager_project.this.project_id
+  network_interface_id = stackit_network_interface.wan_backup[0].network_interface_id
 }

src/modules/connectivity/4-internal-network.tf

@@ -5,10 +5,11 @@
 resource "stackit_network" "lan" {
   count = local.firewall_enabled ? 1 : 0
 
-  project_id  = stackit_resourcemanager_project.this.project_id
-  name        = "lan"
-  ipv4_prefix = var.firewall.lan_network_range
-  routed      = true
+  project_id       = stackit_resourcemanager_project.this.project_id
+  name             = "lan"
+  ipv4_prefix      = var.firewall.lan_network_range
+  ipv4_nameservers = local.network_area_nameservers
+  routed           = true
 }
 
 resource "stackit_network_interface" "lan" {
@@ -17,6 +18,19 @@ resource "stackit_network_interface" "lan" {
   name       = "vtnet1_lan"
   project_id = stackit_resourcemanager_project.this.project_id
   network_id = stackit_network.lan[0].network_id
-  ipv4       = coalesce(var.firewall.lan_ip, cidrhost(var.firewall.lan_network_range, 4))
+  ipv4       = local.firewall_lan_ip
+  security   = false
+}
+
+# security = false is mandatory here, not merely convenient: the CARP VIP answers from
+# the virtual MAC 00:00:5e:00:01:<vhid>, and STACKIT port security cannot express a
+# foreign MAC (allowed_addresses is IP-only)
+resource "stackit_network_interface" "lan_backup" {
+  count = local.firewall_ha_enabled ? 1 : 0
+
+  name       = "vtnet1_lan_backup"
+  project_id = stackit_resourcemanager_project.this.project_id
+  network_id = stackit_network.lan[0].network_id
+  ipv4       = local.firewall_backup_lan_ip
   security   = false
 }