An SSE consumer cannot tell what it lost, or who reported it

[bburda]

Summary

The gateway knows three things about a fault stream that it does not tell the client: how many
events the client lost, whether a replay after reconnect was complete, and how sure it is about
the entity it names on an event.

1. Loss is counted and only logged. #573 made the drop accounting correct. Eviction is now
cursor-aware, entries that every live client already received cost nothing, and only real
client-visible loss is counted. That made the number trustworthy. It still ends in an
RCLCPP_WARN (src/ros2_medkit_gateway/src/http/handlers/sse_fault_handler.cpp:205-214) and
goes nowhere else.

The handler writes exactly two things to a client: an event frame (:285) and ":keepalive"
(:333). The frame payload (format_sse_event, :457-491) contains event_type, fault,
timestamp, optional auto_cleared_codes and optional x-medkit. There is no loss field, no
header, no retry:, and no metrics endpoint. We forced an eviction of 203 events and saw no
signal on the wire at all, only the id jumping from 1597 to 1801.

Today a client can only work out its loss from gaps in id:. That sees only its own connection,
cannot tell an eviction from a coalesce, which the gateway knows and the client does not, and has
to be reimplemented by every consumer.

2. A replay after reconnect can be silently incomplete. collect_pending
(sse_fault_handler.cpp:273-280) selects queued.id > last_event_id and never compares
last_event_id + 1 with event_queue_.front().id. A client that reconnects with a
Last-Event-ID older than the oldest entry still in the buffer receives only what survived, with
nothing marking the gap. The server has both numbers at that point.

3. The event names one entity even when the gateway cannot know which one it is.
resolve_entity_context (sse_fault_handler.cpp:496-503) takes .front() of
reporting_sources, which is a std::set and therefore sorted alphabetically. When a fault code
has several reporters, x-medkit.entity_id is the first one in that order, chosen for no reason
other than its name. The code comment says this is a hint and not authoritative, but the field on
the wire looks exactly like the single-reporter case, so a client cannot tell the two apart.

This is a statement the gateway is not entitled to make. The event carries no triggering source:
FaultEvent.msg has no such field, and publish_fault_event
(src/ros2_medkit_fault_manager/src/fault_manager_node.cpp:1035-1043) drops the source_id from
the request, which is in scope at all five of its call sites.

Reporting the true source needs a new field in FaultEvent, and that decision does not belong
here. ros2_medkit_msgs is a released interface package, and #536 was closed for exactly this
reason: a version mismatch on a generated interface fails silently. That work is tracked with
#467 and waits on the compatibility approach chosen there. What this issue asks for is smaller
and needs no interface change: stop presenting a guess as a fact.

---

Proposed solution (optional)

For (1), send the counters to the client that is losing events: an SSE comment such as
:loss dropped=N coalesced=M, a field on the next event, or a header or first frame on
reconnect. A comment line is the cheapest option and is ignored by clients that do not want it.

For (2), compare the requested Last-Event-ID with the oldest entry still in the buffer and send
the same signal when the replay cannot be complete.

For (3), keep the current shape when there is exactly one reporter, and list the candidates when
there are several:

// one reporter, unchanged
"x-medkit": {"entity_type": "apps", "entity_id": "nav_node"}

// several reporters
"x-medkit": {
  "entity_candidates": [
    {"entity_type": "apps", "entity_id": "nav_a"},
    {"entity_type": "apps", "entity_id": "nav_b"}
  ]
}

This keeps the shortcut that #380 added, since any candidate's rosbag can still be fetched, while
making it visible that the source is not decided. It also stays correct once #467 lands: a real
triggering source would then collapse the list back to one entry.

---

Additional context (optional)

Related to #380, which added entity context to these events in the first place. Part (3) is
bounded by the interface-compatibility decision in #467 and #536.

Worth stating next to this, because it limits what any consumer can conclude: fault_manager
keys records by fault_code alone (fault_storage.hpp:361, sqlite_fault_storage.cpp:152) and
merges every source into one record with a reporting_sources set. For a code that two robots can
both raise, a matching event proves that the code was raised again while source is one of its
reporters. It does not prove that source raised it. The only per-source trace is the
tamper-evident audit log (fault_audit_log.hpp:47-55), which is off by default and has no REST
endpoint.

[bburda]

Corrected part (3) of this issue. It originally asked for a new source field in FaultEvent,
which was the wrong thing to ask for here.

ros2_medkit_msgs is a released interface package. #536 was closed for that reason: changing a
generated interface fails silently on a version mismatch, and picking the compatibility approach
is tracked with #467. Adding a field to FaultEvent runs into the same decision, so it does not
belong in this issue.

The defect that does belong here is narrower and needs no interface change. The gateway takes
.front() of a sorted set and publishes the result as x-medkit.entity_id, which looks the same
on the wire as the case where there really is one reporter. The fix is to stop presenting a guess
as a fact: keep the current shape for a single reporter, and list candidates when there are
several. Reporting the actual triggering source stays with #467 and, once it lands, collapses the
list back to one entry.

Parts (1) and (2) are unchanged.