Expert bridge self healing via end-2-end heartbeat monitoring by Steve-Mcl · Pull Request #7887 · FlowFuse/flowfuse

Steve-Mcl · 2026-07-21T14:57:33Z

Do not merge before https://github.com/FlowFuse/CloudProject/issues/1519

Description

TLDR

Adds an end-to-end heartbeat check for the Expert Agent bridge. On a schedule, the platform publishes a heartbeat through the MQTT bridge to the Expert Broker and back; if the agent misses enough heartbeats in a row, the platform automatically tears down and re-creates the bridge. New ACL rules gate the (deliberately reversed) heartbeat topics so only the platform or the agent can use them, in the correct direction.

Key Points

New ExpertCommsHandler (forge/comms/expert.js) owns the heartbeat request/response lifecycle: publishes a heartbeat with a fresh correlationData, tracks the outcome in a small (LRU/in memory or Redis for installs with replicas depending on the installation) cache (expert-bridge-heartbeat-cache, configurable via app.config.expert.centralBroker.heartbeat.{bridgeCacheMax,bridgeCacheTTL}), and reports { errorCount } back to the caller via callback once the response arrives or maxResponseTime elapses. Wired into forge/comms/index.js as app.comms.expert.
New housekeeper task (forge/ee/lib/expert/tasks/heartbeat.js), registered in forge/ee/lib/expert/index.js only when the Expert Bridge is enabled (app.config.expert.centralBroker.server set). Validates its own cron schedule at registration time (must be recurring, and must not run more often than 2 * maxResponseTime apart, to prevent overlapping runs) and fails safe (logs and skips registration) if the schedule is invalid. On repeated failure (errorCount hits a multiple of maxSuccessiveFailureCount, default 3), it force re-syncs the bridge via syncBridge(app, { force: true }).
forge/comms/commsClient.js: recognizes the agent's echoed heartbeat message on the bridge and re-emits it as response/platform/expert/bridge/heartbeat, which ExpertCommsHandler listens for.
forge/comms/aclManager.js: adds four hardcoded, exact-topic ACL rules for the heartbeat handshake. The topic naming is intentionally reversed — the platform's heartbeat "request" is published on the topic literally named .../response, and vice versa — because the heartbeat reuses the existing platform command channel rather than adding new topics. Each direction is locked to the correct client type (forge_platform vs expert-agent) and pub/sub direction.

Tests added to `test/unit/forge/comms/expert_spec.js`

New file covering ExpertCommsHandler: cache defaults and config overrides (centralBroker.bridgeCache.max/.ttl), the response-listener storing a payload in cache, the outgoing heartbeat request shape (topic/qos/correlationData), success/failure/reset of errorCount across repeated calls, ignoring a response with a mismatched correlationData, clearing a still-pending timer when called again, tolerating no callback, and forwarding a cache-read error to the callback.

Tests added to `test/unit/forge/comms/authRoutesV2_spec.js`

Added to the existing "Expert Agent" ACL block, plus a new "Platform (forge_platform)" block: allow/deny pairs for all four heartbeat topics, confirming each client type can only use its topic in the correct direction (e.g. the agent can subscribe to the heartbeat response topic but not publish to it, and vice versa for the platform).

Tests added to `test/unit/forge/ee/lib/expert/tasks/heartbeat_spec.js`

New file covering the task factory in isolation (fake app/stubbed syncBridge, no full app boot): schedule/startDelay/maxResponseTime/maxSuccessiveFailureCount validation (including the startDelay: 0 fix and the "too frequent schedule" guard), the startDelay gate in run(), and the re-sync callback logic (no re-sync on success or below the failure threshold, re-sync at each multiple of maxSuccessiveFailureCount, and that a syncBridge failure is logged rather than thrown).

Related Issue(s)

closes #7818

Checklist

Labels

Includes a DB migration? -> add the area:migration label

…FlowFuse/flowfuse into feat-expert-bridge-self-healing

hardillb · 2026-07-21T15:16:50Z

@Steve-Mcl I'm unlikely to get to this before Monday

Steve-Mcl · 2026-07-21T15:18:04Z

Thanks for letting me know @hardillb - will reassign review to @cstns

Steve-Mcl · 2026-07-21T15:30:40Z

@cstns added as reviewer in place of Ben since you have knowledge of the pathways involved.

What you may not be aware of is the outage we had on Expert a couple of Saturdays ago.

Long story short, the bridge became uni-directional. Stopping and starting it recovered the 2 way communications.

The owner issue #7818 was raised to add heartbeat and self healing (have a read and decide if the Pr fulfils that)

NOTE:
The pathway chosen (the command/inflight path way) was a deliberate 1st iteration choice.
I toyed with making this 100% coverage (i.e. sen HB to /chat/, route it back over /platform/request to the platform, bounc it back to Expert on platform/response then bounce it back to /chat/response - however it was quite messy and considerations about source origin, app replicas, and adding extra fiorge app topics to ACLs meant it was a LOT bigger than it already is.

In short, I chose the least path of resistance (the command/inflight channels) and used them in reverse as this mean no new acls and at least covered 2 directions - we can iterate if we find the bridge breaks down for a single channel.

codecov · 2026-07-21T15:37:11Z

Codecov Report

❌ Patch coverage is 86.72566% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.72%. Comparing base (200538d) to head (e9066a0).

Files with missing lines	Patch %	Lines
forge/ee/lib/expert/index.js	22.22%	7 Missing ⚠️
forge/comms/aclManager.js	77.77%	4 Missing ⚠️
forge/comms/commsClient.js	0.00%	3 Missing ⚠️
forge/ee/lib/expert/tasks/heartbeat.js	97.72%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7887      +/-   ##
==========================================
+ Coverage   75.65%   75.72%   +0.07%     
==========================================
  Files         432      434       +2     
  Lines       23086    23198     +112     
  Branches     6126     6161      +35     
==========================================
+ Hits        17465    17567     +102     
- Misses       5621     5631      +10

Flag	Coverage Δ
backend	`75.72% <86.72%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cstns · 2026-07-23T07:00:43Z

+        const payloadData = JSON.stringify(payload)
+
+        // Send the heartbeat request and set a timer to check for a response after the maxResponseTime has elapsed
+        this.client.publish(topic, payloadData, mqttOptions)


The heartbeat request is published with only correlationData (no userProperties) & the heartbeat handler at line ~159 in commsClient is placed after this early return which ignores messages without userProperties.

in the heartbeat reply (generated by the other side, userProperties is added).

cstns · 2026-07-23T07:11:32Z

+            await sleep(randomInt(0, 999))
+
+            // Request a heartbeat from the Expert Agent via the bridge
+            expertCommsHandler.requestBridgeHeartbeat(maxResponseTime, async (err, result) => {


Consider using an object parameter here, e.g. async ({ err, result } = {}) => { ... }. That would make the callback a bit more flexible, since callers that only need the result wouldn’t have to pass null as the first argument just to skip the error

can be easily retroed - lets leave for now

cstns · 2026-07-23T07:27:12Z

+        // This event is triggered (emitted) in the commsClient when a valid heartbeat response is received from the Expert Broker via.
+        // the bridge (Expert Broker → FF App Instance Broker) in response to a heartbeat request made in `this.requestBridgeHeartbeat()`.
+        // Here we simply update the cache with the response. The cache is required since there may be replica instances of the FF App
+        // and the expert platform topic is a shared topic, so the response may be received by a different instance than the one that initiated the request.


Wouldn't it make more sense to keep the heartbeat cache in a shared chase then? this would mitigate cross talk from the initiator replica and responder replica

using app.caches does resolve to an in memory or redis cache. On FFC, we use redis (so it is not a local cache)

cstns · 2026-07-23T07:29:26Z

        const deviceCommsHandler = DeviceCommsHandler(app, client)
        const instanceCommsHandler = InstanceCommsHandler(app, client)
        const platformAutomationHandler = PlatformAutomationHandler(app, client)
+        const expertCommsHandler = new ExpertCommsHandler(app, client)


we should we conditionally create/instantiate the expert commsHandler only if there's a bridge is enabled

fair point. will consider.

Steve-Mcl and others added 7 commits July 21, 2026 15:47

Add ExpertCommsHandler for handling expert heartbeat events

a7cd392

Pass heartbeat replies to expert comms hander

ae2218d

Add heartbeat topic validation for platform and expert agent users

996cd53

Create a new housekeeper task for expert heartbeat

43168ca

Register Expert Agent bridge heartbeat task if enabled

89f0baf

Add tests for Expert Agent bridge heartbeat task and ExpertCommsHandler

15a0d58

Merge branch 'main' into feat-expert-bridge-self-healing

b027680

Steve-Mcl requested a review from hardillb July 21, 2026 14:57

Steve-Mcl changed the title ~~Feat expert bridge self healing~~ Expert bridge self healing via end-2-end heartbeat monitoring Jul 21, 2026

Steve-Mcl temporarily deployed to staging July 21, 2026 15:08 — with GitHub Actions Inactive

Steve-Mcl added 2 commits July 21, 2026 16:14

Remove temp console logs from bridge heartbeat code

98b9aa8

Merge branch 'feat-expert-bridge-self-healing' of https://github.com/…

fe2b8c1

…FlowFuse/flowfuse into feat-expert-bridge-self-healing

Merge branch 'main' into feat-expert-bridge-self-healing

ab7148a

Steve-Mcl temporarily deployed to staging July 21, 2026 15:24 — with GitHub Actions Inactive

Steve-Mcl requested a review from cstns July 21, 2026 15:28

Update test description for centralBroker.heartbeat options

29bb8c5

Steve-Mcl temporarily deployed to staging July 21, 2026 15:50 — with GitHub Actions Inactive

cstns reviewed Jul 23, 2026

View reviewed changes

cstns approved these changes Jul 23, 2026

View reviewed changes

Merge branch 'main' into feat-expert-bridge-self-healing

e9066a0

Steve-Mcl enabled auto-merge (squash) July 23, 2026 10:06

Steve-Mcl temporarily deployed to staging July 23, 2026 10:09 — with GitHub Actions Inactive

Steve-Mcl merged commit d3ebe4a into main Jul 23, 2026
29 checks passed

Steve-Mcl deleted the feat-expert-bridge-self-healing branch July 23, 2026 10:27

Steve-Mcl deployed to staging July 23, 2026 10:27 — with GitHub Actions Active

Uh oh!

Conversation

Steve-Mcl commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Do not merge before https://github.com/FlowFuse/CloudProject/issues/1519

Description

TLDR

Key Points

Tests added to test/unit/forge/comms/expert_spec.js

Tests added to test/unit/forge/comms/authRoutesV2_spec.js

Tests added to test/unit/forge/ee/lib/expert/tasks/heartbeat_spec.js

Related Issue(s)

Checklist

Labels

Uh oh!

hardillb commented Jul 21, 2026

Uh oh!

Steve-Mcl commented Jul 21, 2026

Uh oh!

Steve-Mcl commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cstns Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Steve-Mcl Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

cstns Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Steve-Mcl Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cstns Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Steve-Mcl Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

cstns Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Steve-Mcl Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Steve-Mcl commented Jul 21, 2026 •

edited

Loading

Tests added to `test/unit/forge/comms/expert_spec.js`

Tests added to `test/unit/forge/comms/authRoutesV2_spec.js`

Tests added to `test/unit/forge/ee/lib/expert/tasks/heartbeat_spec.js`

Steve-Mcl commented Jul 21, 2026 •

edited

Loading

codecov Bot commented Jul 21, 2026 •

edited

Loading

Steve-Mcl Jul 23, 2026 •

edited

Loading