You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Run an aggregator with a peer. On the peer, define an entity with an operation that takes
longer than aggregation.timeout_ms (default 2000 ms).
POST /api/v1/components/<id>/operations/<op>/executions on the aggregator, then send the
same request directly to the owning peer.
On a single gateway, execute an operation whose own work takes longer than service_call_timeout_sec (default 10 s).
Request a fanned-out collection such as GET /api/v1/faults while one peer is slow enough to
exceed the fan-out budget.
Expected behavior
A request that ran out of time says so. It returns a timeout status, or at least a message that
names the budget that was exceeded. A timeout can be told apart from a peer that is really down,
and a partial answer can be told apart from a complete one.
Actual behavior
Three paths, three different ways of not reporting a timeout.
1. Peer forward: reported as an unavailable peer.
aggregator -> 502 in 2.002822 s
{"error_code": "vendor-error",
"message": "Peer '<name>' at http://<host>:<port> is unavailable",
"vendor_code": "x-medkit-peer-unavailable"}
owning peer -> 200 in 4.936800 s (the full operation result)
The peer was alive at that moment. Probed from inside the aggregator container at the same time: /api/v1/health returned 200 and /api/v1/components/<id> returned 200. The message reports a
peer as down while that peer answers the same request in 4.9 s.
2. Fan-out: reported as success. A peer that times out during a fanned-out collection gives HTTP 200 with x-medkit.partial = true and x-medkit.failed_peers. No status change, no
error code, no message.
3. Local operation: reported as a generic failure. An operation that exceeds service_call_timeout_sec returns 500 with "message": "Service call failed".
Environment
ros2_medkit version: main (a268379), gateway vendor_info.version 0.6.0, SOVD API 1.0.0
ROS 2 distro: Jazzy
OS: Ubuntu 24.04
Additional information
Why the peer forward cannot tell the two cases apart.httplib::Result::operator bool() is
a null check (src/vendored/cpp_httplib/httplib.h:1053). The forward path branches on !result
at src/core/aggregation/peer_client.cpp:637 and never reads result.error(). So Error::ConnectionTimeout, Error::Read (read timeout), Error::Connection (refused) and Error::Canceled all produce the same 502 with x-medkit-peer-unavailable and "is unavailable".
Why the fan-out path loses the failure completely. The error string is built at src/core/aggregation/peer_client.cpp:741 and then dropped. PeerResult
(src/aggregation/aggregation_manager.cpp:677-681) has no error field, and the per-peer task
only records a boolean (aggregation_manager.cpp:719-723). A timeout, a refused connection, a
peer returning 500, a body over the size limit and unparseable JSON all look the same here.
The local operation path. The transport already builds a useful message, "Service call timed out (10000ms): /path/to/service"
(src/ros2/transports/ros2_service_transport.cpp:116-127). The handler puts it in parameters.details but returns 500 with the fixed top-level message "Service call failed"
(src/http/handlers/operation_handlers.cpp:705-707). A client has to match on the text in details to find out it was a timeout.
The action send-goal path does not build that message at all. "Send goal timed out"
(src/ros2/transports/ros2_action_transport.cpp:214) names neither the action nor the budget,
and maps to 500 with x-medkit-ros2-action-unavailable and "Action execution failed"
(operation_handlers.cpp:675-677).
There is already a precedent in the tree.#576 made this case for the action cancel path and #593 fixed it. ERR_NOT_RESPONDING is reserved for exactly this ("no response from the
underlying entity in time", include/ros2_medkit_gateway/core/http/error_codes.hpp:150-154) and
is returned with 504 in one place only: the cancel path, src/http/handlers/operation_handlers.cpp:282. Nothing in the aggregation layer or the operation
execute path uses it. This issue asks for the same treatment on the three paths that were out of
scope for #576.
Two related facts.
There is no separate budget for operation execute. PeerClient holds one int timeout_ms_,
and both construction sites pass config_.timeout_ms
(src/aggregation/aggregation_manager.cpp:219,275). It is applied as both the connect and the
read timeout (src/core/aggregation/peer_client.cpp:813-814). So a synchronous ROS 2 service
call on a peer gets 2 s end to end, while the peer's own budget for the same call is service_call_timeout_sec, default 10 s. The value itself is [BUG] Aggregator times out fanning out large resource payloads to peers #528; the point here is that one
budget covers both a metadata read and real work.
The write timeout is never set, so it stays at the cpp-httplib default of 5 s
(src/vendored/cpp_httplib/httplib.h:41-46) and does not follow aggregation.timeout_ms.
Bug report
Steps to reproduce
longer than
aggregation.timeout_ms(default 2000 ms).POST /api/v1/components/<id>/operations/<op>/executionson the aggregator, then send thesame request directly to the owning peer.
service_call_timeout_sec(default 10 s).GET /api/v1/faultswhile one peer is slow enough toexceed the fan-out budget.
Expected behavior
A request that ran out of time says so. It returns a timeout status, or at least a message that
names the budget that was exceeded. A timeout can be told apart from a peer that is really down,
and a partial answer can be told apart from a complete one.
Actual behavior
Three paths, three different ways of not reporting a timeout.
1. Peer forward: reported as an unavailable peer.
The peer was alive at that moment. Probed from inside the aggregator container at the same time:
/api/v1/healthreturned 200 and/api/v1/components/<id>returned 200. The message reports apeer as down while that peer answers the same request in 4.9 s.
2. Fan-out: reported as success. A peer that times out during a fanned-out collection gives
HTTP 200withx-medkit.partial = trueandx-medkit.failed_peers. No status change, noerror code, no message.
3. Local operation: reported as a generic failure. An operation that exceeds
service_call_timeout_secreturns500with"message": "Service call failed".Environment
Additional information
Why the peer forward cannot tell the two cases apart.
httplib::Result::operator bool()isa null check (
src/vendored/cpp_httplib/httplib.h:1053). The forward path branches on!resultat
src/core/aggregation/peer_client.cpp:637and never readsresult.error(). SoError::ConnectionTimeout,Error::Read(read timeout),Error::Connection(refused) andError::Canceledall produce the same 502 withx-medkit-peer-unavailableand "is unavailable".Why the fan-out path loses the failure completely. The error string is built at
src/core/aggregation/peer_client.cpp:741and then dropped.PeerResult(
src/aggregation/aggregation_manager.cpp:677-681) has no error field, and the per-peer taskonly records a boolean (
aggregation_manager.cpp:719-723). A timeout, a refused connection, apeer returning 500, a body over the size limit and unparseable JSON all look the same here.
The local operation path. The transport already builds a useful message,
"Service call timed out (10000ms): /path/to/service"(
src/ros2/transports/ros2_service_transport.cpp:116-127). The handler puts it inparameters.detailsbut returns500with the fixed top-level message "Service call failed"(
src/http/handlers/operation_handlers.cpp:705-707). A client has to match on the text indetailsto find out it was a timeout.The action send-goal path does not build that message at all.
"Send goal timed out"(
src/ros2/transports/ros2_action_transport.cpp:214) names neither the action nor the budget,and maps to
500withx-medkit-ros2-action-unavailableand "Action execution failed"(
operation_handlers.cpp:675-677).There is already a precedent in the tree. #576 made this case for the action cancel path and
#593 fixed it.
ERR_NOT_RESPONDINGis reserved for exactly this ("no response from theunderlying entity in time",
include/ros2_medkit_gateway/core/http/error_codes.hpp:150-154) andis returned with 504 in one place only: the cancel path,
src/http/handlers/operation_handlers.cpp:282. Nothing in the aggregation layer or the operationexecute path uses it. This issue asks for the same treatment on the three paths that were out of
scope for #576.
Two related facts.
PeerClientholds oneint timeout_ms_,and both construction sites pass
config_.timeout_ms(
src/aggregation/aggregation_manager.cpp:219,275). It is applied as both the connect and theread timeout (
src/core/aggregation/peer_client.cpp:813-814). So a synchronous ROS 2 servicecall on a peer gets 2 s end to end, while the peer's own budget for the same call is
service_call_timeout_sec, default 10 s. The value itself is [BUG] Aggregator times out fanning out large resource payloads to peers #528; the point here is that onebudget covers both a metadata read and real work.
(
src/vendored/cpp_httplib/httplib.h:41-46) and does not followaggregation.timeout_ms.