Gateway node cannot create new FQDN proxies (deployment times out ~5min); sibling gateway node works

[mik-tf]

Summary

On mainnet, a public gateway node cannot provision new Gateway FQDN workloads (grid_fqdn_proxy). Creating one hangs for ~5 minutes and then fails, while the same proxy created on a different gateway node succeeds in ~20 seconds. Existing FQDN proxies already deployed on the affected node keep working — only new FQDN workload deployments fail. This looks like a stuck/unhealthy gateway (zos) agent on that specific node rather than a config or DNS issue.

Affected / working nodes

• Affected: node 1 (gent01.grid.tf, 185.69.166.140) — new FQDN proxy creation times out.
• Working: node 8 (gent02.grid.tf, 185.69.166.141) — identical proxy creates fine.
• Network: main. Provider: threefoldtech/grid (OpenTofu/Terraform).
• Observed repeatedly over ~2 days, multiple attempts.

Error

Error: couldn't deploy fqdn gateway with error: error waiting deployment:
waiting for deployment <id> timed out

  with grid_fqdn_proxy.<name>,
  on main.tf ...

The apply sits on Still creating... [Nm elapsed] for ~4m50s, then fails. The contract is rolled back cleanly (no orphan left on the node — verified via tfcmd get contracts).

Reproduce

1. Have a VM deployment reachable from a gateway node, and a DNS A record for the target FQDN pointing at the affected node's public IP.
2. Create a new grid_fqdn_proxy (Gateway FQDN) on node 1 (gent01).
3. Apply hangs ~5 min → waiting for deployment <id> timed out; contract rolled back.
4. Create the identical grid_fqdn_proxy on node 8 (gent02) instead (DNS repointed to gent02's IP; backend reachable, e.g. over mycelium). It completes in ~22s and the Let's Encrypt cert is issued immediately.

Impact & workaround

• Impact: new FQDN gateways cannot be provisioned on the affected node. Already-provisioned proxies (and their previously issued certs) on that node continue to serve normally.
• Workaround: host the FQDN proxy on a different, healthy gateway node.

Ask

Please check / restart the gateway (zos) agent on gent01.grid.tf and investigate why new Gateway FQDN workload deployments time out there while existing proxies and a sibling gateway node are unaffected.