Skip to content

A node dies and nothing is left to report it #624

Description

@bburda

Part of #570.

Summary

A node stops. The process exits, or it is killed, or its container restarts and it never
comes back. Everything that node provided stops with it. Nothing reports this.

The node cannot report it itself: the thing that would send the fault is the thing that
died. The other detectors do not see it either. qos_mismatch needs both sides present to
compare their profiles. orphan looks at a topic with no subscriber, and a dead publisher
takes its topic with it. lifecycle_expectation only watches nodes the operator listed as
required-active, and it hands absence over to the presence class on purpose. This issue is
that presence class.

GRAPH_NODE_DISAPPEARED already exists as a fault code in the package. No detector raises
it today.

Proposed solution

Track every app the reliability gate has armed, and raise GRAPH_NODE_DISAPPEARED when an
armed node stops being alive for longer than a grace period.

These points decide whether the detector is useful or only noisy.

Alive is not the same as present in the snapshot. In runtime discovery a dead node
leaves the entity snapshot, so the two look the same. In manifest and hybrid discovery the
manifest keeps the App in the snapshot and only clears its online flag. A detector that
counted membership would make a manifest node immortal. It has to read the online flag.

A deactivated lifecycle node is not dead. Its process is alive and only its state
changed. That belongs to GRAPH_NODE_INACTIVE, and raising here too would report one
problem twice under two codes.

The key has to be stable. An app id is recomputed on every sweep. It is the bare node
name unless two nodes collide on it right now. On a multi-robot graph the id flips when a
same-named sibling starts or stops, so a live node held under its old id would be reported
as gone. The fully qualified name does not move.

The grace period is wall clock, not ticks. The entity cache is rebuilt on a graph
event, debounced to about one refresh per second. Every tick between two refreshes sees the
same snapshot, so one absent cache generation is counted again on each tick. At a fast
tick, a tolerance of two ticks is shorter than one refresh cycle, and a single stale
generation becomes an immediate ERROR. The tolerance needs a floor in milliseconds, and it
has to say so when it raises the tick count.

Short-lived CLI nodes must not be tracked. ros2 topic echo and the other CLI verbs
each start a real node called _ros2cli_<pid>. Discovery has no hidden-node filter, so a
few seconds of ros2 topic echo is enough to arm one, and Ctrl-C then leaves a permanent
death behind. Every invocation gets a new pid, so they add up.

A new death must not hide behind an old one. A tracked entry never ages out, so one
departure nobody cares about (a closed rviz2, a calibration node that finished) leaves the
fault permanently confirmed. After that the fault manager treats every later report as an
update to a record the operator has already seen and dismissed: no state transition, no
freeze frame, no rosbag. A node joining the dead set has to force a fresh confirmation.

Also in scope: suppression

#570 asks for suppression to be opt-in and explicit, never a side effect. This is the first
detector that needs it, so the shared part lands here.

A suppressor answers one question: should this key be left out of the fault. Two of them
are needed now:

  • an operator allowlist, for nodes whose departure is expected
  • clean lifecycle shutdown, for a node that went through its own shutdown transition
    instead of dying

A suppressor also has to say whether its veto is durable. A durable veto (the node is on
the allowlist, the node shut itself down cleanly) means the bookkeeping for that node can
be reclaimed. A veto that can lift later must never reclaim anything, because then the real
fault is lost for good once the condition ends.

Nothing is suppressed unless the operator names it in the configuration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions