Broadcast rate limiting and staged send API

[puddly]

See #43 for a much more in-depth discussion. This PR implements broadcast rate limiting for Ziggurat and changes the send API and updates the wire format, hopefully bringing it into its final state for a long while.

The broadcast budget is effectively a token bucket implementation with a capacity of 15 tokens, draining at a rate of 0.55/sec (just under the steady-state SiLabs router rate). Three token slots are reserved for internal stack operations. Two are reserved just for relaying new frames. The rest are available to the application to consume. I think this minimizes broadcast storms while also avoiding the 5s timer issue SiLabs coordinators end up exposing to the user.

When rate limited, broadcasts are instantly rejected by the stack but the error status now carries information about when you can retry again.

The wire format has been updated and send_aps has been split into three separate commands, one for each send type.

---

This PR also rolls in a rather large send API refactor. Instead of having all-or-nothing sends (and an async API that treats sends as atomic!), we now have a multi-stage API. It allows for:

• Synchronous fire and forget requests, where they are queued up without waiting for any confirmation yet still driven to completion/error by the stack in the background.
• Waiting for a request to be handed off to its next-hop destination.
• Waiting for a request to be "delivered", having separate semantics for APS ACKs, broadcast neighbor quorum, etc.

I'd like to explore something similar for zigpy, since it represents enqueue failures (i.e. rate limiting or concurrency limits), send failures (i.e. CCA), and delivery failures (i.e. no route, no ACK) separately but allows a consumer to opt into the send guarantee that is appropriate.

[puddly]

@burmistrzak You asked about a stable wire API. This branch's wire.rs is I think the final state? Let me know if you spot any issues or run into any missing APIs.

[burmistrzak]

Looking quite good!

From an implementer's perspective, most things can be figured out from wire.rs but stuff like endianness or reserved namespaces/IDs aren't always super obvious.
If not a formal specification, I'd appreciate a few more clarifying comments. 😅

[puddly]

Thanks for the review. I'm writing up some more documentation and doing one final streamlining pass for the framing, to simplify implementations.

[Copilot]

Pull request overview

This PR finalizes the on-the-wire protocol update and send-path refactor to support a staged send API (handoff vs delivery) and adds broadcast admission rate limiting to reduce broadcast storms while preserving burst responsiveness.

Changes:

• Introduces protocol v2 wire format (unified 3-byte header w/ frame type, split request vs notification opcode namespaces, richer status codes including rate-limit retry delay).
• Refactors sending to a staged SendHandle model and adds protocol-layer SendTracker machinery to emit confirm notifications per connection/transport.
• Adds broadcast token-bucket admission control (plus tunables) and updates driver internals to use the new send primitives.

Reviewed changes

Copilot reviewed 35 out of 38 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/ziggurat-zigbee/src/nwk/routing.rs	Minor doc/comment adjustment in routing table entry update logic.
crates/ziggurat-zigbee/src/constants.rs	Adds broadcast budget tunables (token bucket capacity/refill + reserves).
crates/ziggurat-zigbee/Cargo.toml	Dependency bumps (educe, num_enum, thiserror).
crates/ziggurat-spinel/Cargo.toml	Dependency bumps and adds tokio-serial.
crates/ziggurat-server/src/main.rs	Server updated for protocol v2 header parsing and per-connection send confirm sweeping/tracking.
crates/ziggurat-server/Cargo.toml	Dependency bumps (clap, cobs, tokio, tokio-serial, tokio-tungstenite).
crates/ziggurat-protocol/src/wire.rs	Protocol v2 wire format: unified header, new opcodes/statuses, new send payloads, new confirm payloads, rate-limit payload.
crates/ziggurat-protocol/src/send_tracker.rs	New sans-IO tracker that sweeps SendHandle progress into confirm notification frames.
crates/ziggurat-protocol/src/lib.rs	Exposes send_tracker module publicly.
crates/ziggurat-protocol/src/bridge.rs	Bridges new staged send API and maps enqueue/delivery outcomes to wire statuses; updates cancellation path.
crates/ziggurat-protocol/Cargo.toml	Dependency bump and adds tracing (no-default-features).
crates/ziggurat-phy/src/lib.rs	Makes RadioError clonable.
crates/ziggurat-phy/Cargo.toml	thiserror bump.
crates/ziggurat-phy-spinel/Cargo.toml	Dependency bumps (parking_lot, tokio).
crates/ziggurat-ot/src/lib.rs	Initializes firmware-side send tracker cell in App init.
crates/ziggurat-ot/Cargo.lock	Lockfile refresh to match dependency updates.
crates/ziggurat-ncp-api/src/protocol.rs	Embedded-side dispatch updated to protocol v2 header + staged send + tracker-based cancellation.
crates/ziggurat-ncp-api/src/lib.rs	Adds firmware-side tracker cell + sweep reactor; resets tracker per started stack.
crates/ziggurat-ncp-api/Cargo.toml	Adds tracing dependency.
crates/ziggurat-ncp-api/Cargo.lock	Lockfile refresh to match dependency updates.
crates/ziggurat-ieee-802154/Cargo.toml	Dependency bumps (num_enum, thiserror, educe).
crates/ziggurat-driver/src/zigbee_stack/zdp.rs	Converts ZDP command path to staged send and new unicast/broadcast send helpers.
crates/ziggurat-driver/src/zigbee_stack/tasklets.rs	New tasklet runner to multiplex small spawned flows (e.g., join follow-ups).
crates/ziggurat-driver/src/zigbee_stack/send_handle.rs	New staged SendHandle/SendSlot implementation with handoff + delivery checkpoints and cancellation.
crates/ziggurat-driver/src/zigbee_stack/route.rs	Migrates routing-related broadcasts/unicasts to new send APIs and staged handling.
crates/ziggurat-driver/src/zigbee_stack/nwk.rs	Implements broadcast admission budget + migrates enqueue/resolve/cancellation semantics to staged outcomes.
crates/ziggurat-driver/src/zigbee_stack/neighbor.rs	Updates link status broadcasting to staged send + oneshot broadcast helper.
crates/ziggurat-driver/src/zigbee_stack/mac.rs	Updates MAC send error typing to DeliveryError.
crates/ziggurat-driver/src/zigbee_stack/joining.rs	Refactors join flow into tasklets; makes key delivery + join notification dependent on send completion.
crates/ziggurat-driver/src/zigbee_stack/indirect.rs	Aligns indirect path outcome/error typing with staged send model.
crates/ziggurat-driver/src/zigbee_stack/aps.rs	Splits APS send APIs by send type; adds staged send behavior and APS-ack integration with SendSlot.
crates/ziggurat-driver/src/zigbee_stack.rs	Introduces EnqueueError vs DeliveryError, adds broadcast budget + tasklets, updates spawn API, removes legacy confirm/result types.
crates/ziggurat-driver/src/signal.rs	Removes now-obsolete Signal primitive (replaced by staged send slots/notify).
crates/ziggurat-driver/src/runtime.rs	Makes Elapsed clonable.
crates/ziggurat-driver/src/lib.rs	Exposes new broadcast_budget module; removes signal module export.
crates/ziggurat-driver/src/broadcast_budget.rs	New token bucket with per-class reserves for broadcast admission control.
crates/ziggurat-driver/Cargo.toml	Dependency bumps and enables futures alloc feature; bumps runtime deps; bumps spin.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[puddly]

Everything runs fine on my end with this changeset but I'm finding the broadcast rate limiting a little aggressive. I'm going to explore this in a follow-up PR after the beta.

[MattWestb]

@puddly Do you have possibility sniffing 2 or 3 levels router from the coordinator so can see if the Zigbee network or 15.4 is blocking / dropping broadcasts ?
I only have the production network that have that deep and the test networks is only one or two levels so the coordinator is can reaching most or all routers.

By the way we are waiting for something very good i think !!!

[puddly]

@MattWestb I unfortunately have only one two hop router. Everything else is a single hop away from the coordinator 😅.

This is why the broadcast rate limiting was proving counterintuitive: in theory, every SiLabs router on the network should be immediately "full" within 10 broadcasts. But on my network, unlimited broadcasts kept working (albeit with popcorning) for a very long time. The current rate limit is hit within 5s of dragging the color slider in HA.

[MattWestb]

Im most interested if the 15.4 layer is blocking then i think (= not knowing) that you is not getting and error only time out from the network to the host system.
I have multi hope in production system but i rebuilding all motion sensors and there parallel remotes from device binding to group binding (all IKEA gen 3 controllers) and its not all done and i like have it stable before making more crazy things in it.

By the way have you looking what / how the other manufactures Zigbee stacks is doing the broadcast storm limiting like TI for older devices and Telink for most tuya and other devices ?
I think they have implemented some kind of limiting function but very likely not the same as Silabs.

[MattWestb]

Was doing one fast test on the migrated IKEA test network running Ziggurat that shall have 2 routers that have one extra level from the coordinator and remotes.
But im not sure is the Zigbee map is updated OK then i think its only do it some times a day.
One IKEA Signal repeater is rebroadcasting mostly frames 1 time = Zigbee3 and is hearing all neighbors is re broadcasting the frame (Silabs chip 5 neighbors not the coordinator).
One tuya dimmer module (Telink chip 7 neighbors not the coordinator) is normally doing classic broadcast ack = reseeding the frame 3 times.
From ZHA is little tricky but Styrbar remote looks dont have and limits.
If spamming with 3 clicks / second its the signal repeater is pulsing 7 frames #-1 then halting for some seconds and broadcasting 7 frames agen (wireshark is keeping the frame number sequences number so not lost in the air only halted broadcast).
The being extreme the broadcast its the blocking cycle (broadcast + block) around 5 seconds between and sequences number in 15.4, ZBNet, ZBApp and ZCL is OK (halted not lost) and very extreme (around 4 commands / seconds) the cycle is around 4 seconds.

I is like you little confused but i thing i getting blocking in Zigbee network or app layer and not in 15.4 but i cant grantee its one relay between the first sender and the sniffed router then cant trust the zha map.

The remote is having one IKEA GU10 WS as parent so i think its that is limiting or the sequence numbers shall jumping the blocking in the under layers.

The positive thing is its looks some router request (radius 29 x 3 from the device / sniffed router) to 0xfffc is going thru but looks delayed and also some link status with radius 1 x 1 so not complete blocked.

I was filtering wireshark to only to / from the router device MVK in 15.4 layer so all unicast and other broadcast to / from other device not observed.