Summary
After a transient network interruption on a brev-managed instance, remote access via brev shell <instance> / ssh <instance> (through the brev gateway) fails with:
kex_exchange_identification: Connection closed by remote host
Connection closed by <gateway-ip> port <port>
…even though the NetBird agent on the instance reports a fully healthy connection (Management: Connected, Signal: Connected, all peers connected, service active + enabled). The gateway's forwarding/registration for the instance appears to go stale after the partition and is not refreshed on reconnect. A manual sudo systemctl restart netbird on the instance re-registers it and restores access immediately.
The key issue: the agent's self-reported health passes while the end-to-end path through the gateway is dead, so nothing triggers a re-registration.
Environment
- brev-managed GPU instance (Ubuntu), NetBird daemon
v0.74.7
- NetBird service
active and enabled (auto-starts on boot)
Steps to reproduce
- Instance is up and reachable via
brev shell / ssh <instance>.
- Cause a transient network interruption — e.g. an upstream switch/power blip or a link flap on the primary NIC (in our case, briefly de-powering a shared network switch).
- Instance OS recovers and is reachable on its LAN / other overlays; NetBird reconnects on its own.
netbird status on the instance reports healthy: Management: Connected, Signal: Connected, Peers count: N/N Connected.
- But
ssh <instance> through the brev gateway still fails with Connection closed by remote host.
sudo systemctl restart netbird → access restored immediately.
Root-cause hypothesis
The NetBird control plane (Management/Signal) reconnects after the partition, so the agent self-reports "Connected," but the gateway's port-forward / peer registration to the instance is left in a stale, half-open state. Because the agent's own health check passes, no re-registration is triggered and the stale mapping persists until the daemon is manually restarted.
Possible solutions
- Refresh the gateway port-forward on peer reconnect (gateway/control-plane side). When a peer's tunnel re-establishes after a partition, refresh the forwarding mapping instead of assuming the prior one is still valid.
- End-to-end liveness detection, not just control-plane status. Have the agent/CLI periodically verify reachability through the gateway (or to a known peer over the overlay) and trigger a re-register/reconnect on failure — rather than relying solely on
Management/Signal: Connected, which can be green while the path is dead.
- Restart/re-register NetBird on network link-up. Ship or document a NetworkManager/networkd dispatcher hook that nudges NetBird when the primary interface transitions down→up, so a link flap forces a fresh registration.
- More aggressive reconnection/backoff after a partition, and confirm WireGuard
persistent-keepalive is set on NAT'd peers so mappings survive / re-punch quickly.
Current workaround
sudo systemctl restart netbird on the affected instance. (Reachable in the meantime via a direct/LAN path or another overlay to run the restart.)
Summary
After a transient network interruption on a brev-managed instance, remote access via
brev shell <instance>/ssh <instance>(through the brev gateway) fails with:…even though the NetBird agent on the instance reports a fully healthy connection (
Management: Connected,Signal: Connected, all peers connected, serviceactive+enabled). The gateway's forwarding/registration for the instance appears to go stale after the partition and is not refreshed on reconnect. A manualsudo systemctl restart netbirdon the instance re-registers it and restores access immediately.The key issue: the agent's self-reported health passes while the end-to-end path through the gateway is dead, so nothing triggers a re-registration.
Environment
v0.74.7activeandenabled(auto-starts on boot)Steps to reproduce
brev shell/ssh <instance>.netbird statuson the instance reports healthy:Management: Connected,Signal: Connected,Peers count: N/N Connected.ssh <instance>through the brev gateway still fails withConnection closed by remote host.sudo systemctl restart netbird→ access restored immediately.Root-cause hypothesis
The NetBird control plane (Management/Signal) reconnects after the partition, so the agent self-reports "Connected," but the gateway's port-forward / peer registration to the instance is left in a stale, half-open state. Because the agent's own health check passes, no re-registration is triggered and the stale mapping persists until the daemon is manually restarted.
Possible solutions
Management/Signal: Connected, which can be green while the path is dead.persistent-keepaliveis set on NAT'd peers so mappings survive / re-punch quickly.Current workaround
sudo systemctl restart netbirdon the affected instance. (Reachable in the meantime via a direct/LAN path or another overlay to run the restart.)