Skip to content

[BUG] An SSE fault-stream slot is held for up to 30 s after the client disconnects #615

Description

@bburda

Bug report

Steps to reproduce

  1. Start a gateway with the default sse.max_clients (2).
  2. From one client, connect to GET /api/v1/faults/stream, disconnect, and reconnect, about once
    per second.

Expected behavior

A slot is freed when the connection ends, so a single reconnecting client cannot lock itself out.
The default should also be higher than 2, since a browser, a recorder and an operator may all
want this stream at the same time.

Actual behavior

After a few cycles, from that same single client:

HTTP 503
{"error_code": "service-unavailable",
 "message": "Maximum number of SSE clients reached. Please try again later."}

Gateway start-up line:
SSE fault handler initialized, subscribed to /fault_manager/events, max_clients=2.

One client in a reconnect loop blocks the stream for everyone, including the Web UI.

Environment

  • ros2_medkit version: main (a268379), gateway vendor_info.version 0.6.0, SOVD API 1.0.0
  • ROS 2 distro: Jazzy
  • OS: Ubuntu 24.04

Additional information

The release itself is correct. The release_guard deleter
(src/http/handlers/sse_fault_handler.cpp:375-389) is destroyed together with the stack-local
httplib::Response. The problem is when that happens.

cpp-httplib checks strm.is_writable() only at the top of each write_content_chunked
iteration, which means only after the content provider returns true
(src/vendored/cpp_httplib/httplib.h:4206-4217). The fault-stream provider never returns true.
src/http/handlers/sse_fault_handler.cpp:310-349 is an inner while (true) whose only exits are
return false, and the return true at :351 cannot be reached.

So a closed connection is only noticed when the next write happens, either an event flush
(:285) or a keepalive (:333). Between events the loop waits in queue_cv_.wait_for (:319)
for keepalive_interval_, which defaults to kKeepaliveIntervalSec = 30
(include/ros2_medkit_gateway/http/handlers/sse_fault_handler.hpp:290). There is no ROS
parameter for it. A shorter value is only reachable through the test-only constructor.

On a quiet system the window is the full 30 s.

The other two SSE producers do return true on each tick, so the cpp-httplib check runs every
iteration and their slots are freed quickly: src/http/handlers/sse_transport_provider.cpp:184
and src/http/handlers/trigger_handlers.cpp:560. The fault stream is the exception.

There is no reaper and no other user of the tracker. SSEClientTracker
(include/ros2_medkit_gateway/core/http/sse_client_tracker.hpp:30-48) is a plain counter with no
per-connection identity, timestamp or expiry, and no timer or periodic sweep touches it.

Two smaller points in the same area:

  • src/ros2_medkit_gateway/README.md:1239 says the sse.max_clients default is 10. The code
    (src/gateway_node.cpp:112), the shipped YAML (config/gateway_params.yaml:215) and
    docs/config/server.rst:641 all say 2.
  • No test covers disconnect and reconnect, or the release delay. The existing slot tests
    (test/test_sse_fault_handler.cpp:284, :836) call the releaser directly, which skips the
    timing completely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions