VC: Reconnect head SSE streams and restart on candidate updates

[peter941221]

PR: Fix head monitor resilience + candidate-update restart (sigp/lighthouse#8741)

Summary

• The VC head monitor currently selects over per-BN SSE streams via futures::stream::select_all.
• If any BN SSE stream errors, the whole monitor exits (global failure).
• If a BN SSE stream ends (e.g. BN restart), select_all drops it silently and it never reconnects.
• Separately, update_candidates_list re-enumerates candidate indices; any long-lived stream may keep reporting stale indices unless the head monitor restarts.

What This PR Changes

• Per-BN self-reconnecting head event stream:

  • Each candidate runs a small reconnect loop inside its own Stream built with futures::stream::unfold.
  • Any per-stream Err(_) or None (stream ended) triggers:
    • BeaconHeadCache::remove(candidate_index) to avoid stale cache influence
    • a bounded retry delay (defaults to one slot) before reconnecting
  • Connect attempts are bounded with tokio::time::timeout around BeaconNodeHttpClient::get_events to avoid a hung connect blocking progress.
  • Failure logging is throttled (warn on 1,2,4,8,... consecutive failures; debug otherwise) to avoid per-slot log spam under prolonged BN downtime.
  • Stream ends (None) are logged at debug to avoid false-alarm warnings (warns are reserved for connect timeouts/errors and stream errors).

• Candidate list update restart signal ("generation"):

  • BeaconNodeFallback now maintains head_monitor_generation_tx: watch::Sender<u64>.
  • update_candidates_list(...) bumps the generation (using send_replace) after updating candidates + purging the head cache.
  • The head monitor poll_head_event_from_beacon_nodes(...) subscribes and returns Ok(()) on generation change, causing the outer service loop to restart the monitor with the new candidate indices.
  • Emits an info log on generation change so operators can see deliberate monitor restarts.

Files Touched

• validator_client/beacon_node_fallback/src/lib.rs

  • add head_monitor_generation_tx
  • init watch channel in set_head_send
  • bump generation in update_candidates_list
  • unit test: generation bumps on candidate update

• validator_client/beacon_node_fallback/src/beacon_head_monitor.rs

  • add BeaconHeadCache::remove
  • build reconnecting per-candidate head streams with bounded connect timeout + per-slot retry + warn throttling
  • restart on generation change
  • unit test: remove only removes requested entry

Testing

• cargo test -p beacon_node_fallback

Known Limitations / Follow-ups

• This PR bounds connect time, but it does not add an idle/read-timeout on an established SSE connection. If a stream becomes "stuck silent" without ending/erroring, it may not reconnect until the underlying connection fails. A follow-up could add a read timeout/heartbeat to detect that condition.

[cla-assistant]

All committers have signed the CLA.

[peter941221]

Quick follow-up here because linked issue #8741 is still open and this PR has been quiet since the full green CI run.

The current head is still bf77376, and the change is still intentionally narrow:

• per-candidate self-reconnecting head streams
• bounded connect timeout around get_events
• generation-based restart when the candidate list changes so index/cache state stays coherent

If maintainers would prefer an even smaller first slice, I am happy to split it, but my read was that the reconnect and candidate-update restart paths are the same stale-stream consistency bug family from #8741.

[peter941221]

Quick follow-up here because linked issue #8741 is still open and this PR has stayed green without human review.

The current head is still bf77376, and I still believe the change is one bug-family slice rather than two unrelated features:

• per-candidate SSE streams now reconnect on end / error instead of being silently dropped from select_all
• candidate-list updates now trigger a generation restart so stale index / cache mappings do not survive a topology change
• reconnects stay bounded by the explicit get_events timeout instead of potentially hanging forever

If maintainers would prefer a smaller first merge shape, I am happy to split this further. My current read was just that the reconnect path and the candidate-update restart path are both part of the same stale-stream consistency problem from #8741, so I kept them together.

If there is a preferred smaller slice or a different direction you would rather take here, I am very happy to reshape the branch.