From 2e571616c3b0f9227dbe68276251b88bc1c90dda Mon Sep 17 00:00:00 2001 From: ewowi Date: Mon, 6 Jul 2026 23:53:21 +0200 Subject: [PATCH 1/3] Add Home Assistant MQTT Discovery (JSON schema) with LWT availability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The device now announces itself to Home Assistant over MQTT: on connect it publishes a retained homeassistant/light//config so HA (or any Discovery-aware hub) auto-creates a wired light entity, no hand-matching topics. It speaks HA's own JSON schema on ha/set + ha/state alongside the existing mqttthing topics, and backs availability with an MQTT Last-Will so HA greys the entity out when the device drops. Discovery is opt-out (haDiscovery, default on) and its buffers are heap-on-use, so a device that never enables it pays zero bytes. KPI: 16384lights | PC:666KB | tick:120/91/119/15/2/274/58/17/21/160/115/17/1/34us(FPS:8333/10989/8403/66666/500000/3649/17241/58823/47619/6250/8695/58823/1000000/29411) | tick:1369us(FPS:730) | heap:155KB | src:176(34978) | test:122(18006) | lizard:126w Note: the ESP32 KPI line (1369us/730) is the stale monitor.log baseline. Live captures on the actual HA-test boards this session: MM-S31 (eth) 2641us/378 FPS, MM-P4 (eth) 7033us/141 FPS — both healthy across the full discovery toggle + command round-trip. Core: - MqttPacket.h: extended buildMqttConnect with optional Last-Will (topic/payload/retain) via the connect-flag bits, same mechanism as username/password; a partial will (topic without message) is dropped per §3.1.3. Pinned by a byte-exact golden vector. - MqttModule: added HA MQTT Discovery — retained JSON-schema light config on homeassistant/light/projectMM_/config (uniq_id is the MAC-stable id, never the editable name), the ha/set inbound command topic (reuses mm::json flat helpers, HA brightness 0-255 with no rescale), the retained ha/state feedback publish (change-gated), and a retained online/offline availability topic backed by the CONNECT Last-Will. Toggling haDiscovery announces/retracts live over the existing socket, no reconnect. Discovery scratch buffers (768 B) allocate on first announce and free in teardown() / on disable / on discovery-off, reported via dynamicBytes — the pay-for-what-you-use rule. - MqttModule: fixed a heap-strand found live on P4/S31 hardware — turning haDiscovery OFF while the socket was mid-reconnect returned before freeing the buffers, leaking 768 B until teardown. The retract path now frees unconditionally (freeing local memory needs no socket); only the entity-clearing empty PUBLISH needs a live link. Docs / CI: - architecture.md: recorded the "Pay for what you use" memory rule under § Memory strategy — a module holds heap only for capabilities it is actually exercising (absent → 0; present-but-off → class instance only; active → +dynamicBytes, freed in teardown). Governs every module; the LED driver and MQTT discovery are the worked examples. - services.md: documented the haDiscovery control on the MQTT card. - home-automation.md: rewrote the HA path — MQTT Discovery (auto-wired, recommended) and the WLED integration (no broker); note not to hand-configure HA's generic MQTT against the mqttthing topics. - backlog-core.md: marked HA MQTT Discovery shipped; kept the generic-topics escape hatch as unbuilt. - Plan-20260706 - Home Assistant MQTT Discovery.md: saved the approved plan. Tests: - unit_MqttPacket: added a byte-exact golden vector for CONNECT with a retained Last-Will (will-flag bits + will topic/payload tail). - unit_MqttModule: added ha/set command tests ({state}, {brightness} no-rescale, key-order-independent, state-only-preserves-brightness), a CONNACK-publishes-retained-discovery-config test via the outbound-capture seam, and a regression test pinning the disconnected-retract fix (connect → allocate → force reset → discovery-off while disconnected → dynamicBytes drops to 0). - scenario_MqttModule_haDiscovery_toggle: added a live-only scenario toggling haDiscovery on/off/on, asserting FPS + heap stay in range (SKIPs on desktop, runs on a board with a broker). Co-Authored-By: Claude Opus 4.8 --- docs/architecture.md | 10 + docs/backlog/backlog-core.md | 9 +- ...0260706 - Home Assistant MQTT Discovery.md | 137 ++++++++++++ docs/moonmodules/core/services.md | 1 + docs/usecases/home-automation.md | 5 +- src/core/MqttModule.cpp | 196 +++++++++++++++++- src/core/MqttModule.h | 59 +++++- src/core/MqttPacket.h | 28 ++- ...cenario_MqttModule_haDiscovery_toggle.json | 69 ++++++ test/unit/core/unit_MqttModule.cpp | 88 ++++++++ test/unit/core/unit_MqttPacket.cpp | 21 ++ 11 files changed, 611 insertions(+), 12 deletions(-) create mode 100644 docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md create mode 100644 test/scenarios/core/scenario_MqttModule_haDiscovery_toggle.json diff --git a/docs/architecture.md b/docs/architecture.md index 5bddabb2..3276aec9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -476,6 +476,16 @@ Network-based drivers (ArtNet, E1.31, DDP) pace their output with a **non-blocki All buffers are allocated as single contiguous blocks outside the hot path, at startup or when configuration changes (LED count, layout size, layer count). They are then reused every frame with zero allocations in steady state. Measured per-module timing and memory for each platform: [performance.md](performance.md). +### Pay for what you use + +A module holds heap **only for capabilities it is actually exercising**, the same zero-overhead principle C++ applies to abstractions ("you don't pay for what you don't use"). Concretely, for every module: + +- **A module not in the tree costs nothing.** Modules are heap-allocated through `MoonModule::operator new` when added (via the factory or boot wiring), so a deviceModel that omits a module pays zero — not even its `classSize()`. This is the base case the rest of the rule extends inward. +- **A feature's buffer allocates on first use, not at `setup()`.** When a module *is* present but a given capability is dormant (a driver with no output attached, an MQTT client with HA discovery toggled off), that capability's buffer is `nullptr` until the code path that needs it runs. Allocating eagerly at `setup()` for a path that may never execute is the anti-pattern this rule forbids — it charges every instance for the worst case. +- **The allocation frees in `teardown()`** (and on the transition that makes the capability dormant again — a disable, a toggle-off), and is reported through `dynamicBytes()` so `/api/system` and the memory scenarios see the real ladder. `MoonModule::teardown()` reverse-recurses into children, so a subtree's memory unwinds bottom-up with no leak. + +The result is a memory ladder that tracks configuration exactly: module-absent → 0; module-present-but-feature-off → just the class instance; feature-active → `+dynamicBytes()`. The LED driver's output buffer (allocated when a physical output exists, per [§ Adaptive allocation](#adaptive-allocation) below) and the MQTT module's discovery-config scratch (allocated when a connected client announces itself to Home Assistant) are the two worked examples; the rule governs every module, not those two. Rationale: on a no-PSRAM ESP32 the internal-heap reserve (`HEAP_RESERVE`) is the tightest constraint in the system, so a module that grabs a buffer it isn't using is spending the reserve that keeps the render loop, WiFi, and HTTP alive. + ### Buffer types - **Layer buffers**: one per active layer, holds the logical light data for one effect chain. Allocated in PSRAM when available. On memory-constrained devices, consumers may read from the layer buffer directly (no mapping, no blending, no physical buffer needed). diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index 6ff009f9..ace0192d 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -300,7 +300,14 @@ Explicitly **out** (no practical MQTT case, and wrong for the transport): **Shape:** ~15 lines routing into the existing `applySetControl` + a state publish; gate behind a `generic API` bool defaulting **off**, so a home user's broker isn't a remote-config backdoor and the curated HomeKit surface stays the clean default. **Don't** auto-generate a semantic topic per control — HomeKit/HA need stable typed topics; a generic `palette/set` whose meaning shifts per firmware breaks their discovery. Keep semantic topics hand-curated; let the generic pair be the escape hatch. -**If the actual goal is deeper HA (not scripting), the better path is HA MQTT Discovery** — the device announces its controls as HA entities via retained `homeassistant/…/config` topics (the industry-standard pattern, *Common patterns first*), giving HA typed entities instead of a raw JSON pipe. Bigger than the generic pair, and the right lift if HA depth — not power-user scripting — is the target. Related integration threads to keep this coherent with: the [DevicesModule command half](#devicesmodule-interop-plugins-the-command-half-discovery-shipped) (Tasmota-MQTT / zigbee2mqtt as *outbound* control of foreign devices — the mirror of this *inbound* surface), and the [LightsControl integration point](backlog-mixed.md) (the eventual single owner of "device ↔ outside world", MQTT/HA included). +**HA MQTT Discovery — SHIPPED.** The device announces a retained JSON-schema light config to +`homeassistant/light//config` (the Tasmota/ESPHome/Zigbee2MQTT pattern), so HA auto-creates a +wired entity — gated on the `haDiscovery` control, with a Last-Will availability topic. This covered +the "HA can't reach the device over MQTT" incidents. The *generic-topics escape hatch* above (the +whole REST API over MQTT, for power-user scripting) is the remaining unbuilt piece; keep it coherent +with the [DevicesModule command half](#devicesmodule-interop-plugins-the-command-half-discovery-shipped) +(Tasmota-MQTT / zigbee2mqtt as *outbound* control — the mirror of this *inbound* surface) and the +[LightsControl integration point](backlog-mixed.md). **Open question — Homebridge example maps both `setBrightness` and `setHSV`.** In `homebridge-mqttthing`'s `lightbulb`, HSV's *value* (V) already carries HomeKit's Brightness characteristic, so mapping both topic pairs may double-drive brightness (mqttthing's docs lean toward using one or the other). The [MQTT § Homebridge example](../moonmodules/core/services.md#mqtt) currently lists both, to show the device's full topic surface. Resolve on hardware: flash a board, run Homebridge + mqttthing with that config, and check whether the Home-app brightness slider misbehaves — if it does, drop the two `brightness` topics from the example; if not, it's a non-issue. (Flagged by CodeRabbit; parked here rather than changing the doc on an untested hunch.) diff --git a/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md b/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md new file mode 100644 index 00000000..ecfee2c2 --- /dev/null +++ b/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md @@ -0,0 +1,137 @@ +# Plan — Home Assistant MQTT Discovery (JSON schema) + +## Context + +A user (Shelly board) tried to control the device from Home Assistant's "Easy MQTT" UI addon and got +nothing: HA's addon publishes to HA's own schema (`homeassistant/light//set` + `{"state":"ON"}` +JSON), while the device subscribes to the mqttthing schema (`projectMM//on/set` + `"true"`). He +unblocked himself by hand-pointing HA at the device's real topics — proving this is an **ergonomics +gap, not a capability gap**. The device is *controllable* from HA today; it just isn't +*auto-discoverable*. + +**What we build:** HA MQTT Discovery — the device announces itself via a retained +`homeassistant/light//config` topic so HA (and any Discovery-aware hub) auto-creates a +correctly-wired light entity. **JSON schema** (PO decision), matching the modern auto-discovery peer +group (Tasmota / ESPHome / Zigbee2MQTT) — *Common patterns first* — and chosen for **extensibility**: +a JSON light carries all state in one atomic message and has native `effect`/`effect_list`, so future +controls (presets, effects, palette-as-colour) add a key, not a new topic + custom HA config. The +existing `projectMM//…` mqttthing topics stay byte-identical (the user's working setup is +untouched); Discovery lands *alongside* them. + +Feature branch `ha-mqtt-discovery` (already created; carries the earlier backlog-priority edit). + +## Decisions locked (PO) + +- **JSON schema** discovery, not default schema. +- **`unique_id` = ``** (the stable id per [decisions.md § MQTT identity](../decisions.md)), `name` = `SystemModule::deviceName()`. Never the editable name as identity. +- Gated on a new **`haDiscovery`** bool control (default on where MQTT ships); toggling re-announces / retracts. +- Existing mqttthing topics unchanged; Discovery is additive. + +## Design (reuse the existing MqttModule seams) + +All in `src/core/MqttModule.{h,cpp}` + `src/core/MqttPacket.h`, on `loop1s()` (off the hot path). The +Explore map confirmed every primitive needed already exists; the work is three small additions. + +**1. Announce (on CONNACK-accept, near the existing `publishName()`/`publishState(true)` at ~cpp:253):** +- New `buildDiscoveryTopic(out, cap)` → `homeassistant/light//config` (independent of + `topicPrefix()`/`buildTopic()`, which hard-code the `projectMM` root). +- New `publishDiscovery(bool announce)` — builds the retained JSON config and publishes it with the + existing `buildMqttPublish(topic, payload, len, buf, len, /*retain=*/true)` (already supports + arbitrary topic + retain). A retracting empty retained payload when `haDiscovery` is toggled off. +- **Config JSON** (JSON-schema light; validated field-by-field against HA's light.mqtt JSON schema — + `schema:"json"` required, `cmd_t`/`stat_t`/`uniq_id`/`avty_t` are the correct abbreviations, + `"brightness":true` enables brightness at the default 0-255 scale, `dev{ids,name,mf,mdl}` shape + correct): + `{"schema":"json","name":"","uniq_id":"projectMM_","cmd_t":"projectMM//ha/set", + "stat_t":"projectMM//ha/state","avty_t":"projectMM//status","brightness":true, + "dev":{"ids":["projectMM_"],"name":"","mf":"MoonModules","mdl":"projectMM"}}`. + (`uniq_id` prefixed `projectMM_` for cross-vendor uniqueness. Dedicated `ha/set` + `ha/state` keep + the JSON-schema traffic separate from the scalar mqttthing `on/set` etc. — no topic carries two + payload formats.) +- **Buffer (P0 — silent-failure risk):** the config PUBLISH is ~285 bytes (payload ~247 + topic 33), + **over `kSendBufLen = 256`** — and `buildMqttPublish` returns 0 on overflow while `sendPacket(buf,0)` + returns true, so the config would **never send with no error**. Use a **dedicated `uint8_t + buf[384]`** in `publishDiscovery()` (not a global `kSendBufLen` bump, which fattens every per-tick + frame), and **guard the `n == 0` return** as a real failure. + +**2. Inbound (new branch in `routePublish`, ~cpp:262):** +- Match suffix `ha/set`, payload `{"state":"ON"|"OFF"[,"brightness":0-255][,"effect":"..."]}`. +- **Reuse `mm::json` (`src/core/JsonUtil.h`) — do NOT hand-roll.** The repo already has flat + `json::hasKey/parseBool/parseInt/parseString`, and `HttpServerModule::applyWledState()` + (HttpServerModule.cpp:1074-1087) is the exact precedent: it parses an inbound `{"on":…,"bri":…}` + body with those helpers (0-255 clamp) and routes via `setControl("Drivers",…)`. `ha/set` is the same + shape (`state` is a string → `json::parseString`). The helpers tolerate HA's whitespace and are + key-order-independent. (NOT the heavy recursive `json::parse`/arena — that's for the nested device + list.) The inbound body needs a **larger local buffer than the existing `value[32]`** (cpp:271). +- Maps to the same `setControlValue("on"/"brightness", …)` calls — HA JSON brightness is 0–255, so + **no rescale** (default `brightness_scale` is 255). +- Extensible: an `effect` key later → `setControlValue("palette"/"effect", …)` with `effect_list` in + the config. +- SUBSCRIBE to `ha/set` when `haDiscovery` is on — at CONNACK (alongside `kSets[]`, cpp:241) AND on a + mid-session toggle-on (subscriptions today happen only once at CONNACK). + +**2b. Availability (LWT — industry standard, cheap; PO: include).** Every serious MQTT device +(Tasmota/ESPHome/Zigbee2MQTT/Shelly) backs an availability topic with an MQTT Last-Will, so HA greys +the entity out the moment the device drops — the exact "dead light shows as on" bug Discovery must not +leave. Cost is small and off the hot path: extend `buildMqttConnect` (MqttPacket.h:124) with optional +**will-topic / will-payload / will-retain** (3 connect-flag bits + two more strings, same mechanism as +username/password already there; ~25 lines + a golden-vector test). Declare the will = retained +`offline` to `projectMM//status`; publish retained `online` to the same topic on CONNACK-accept; +add `"avty_t":"projectMM//status"` to the discovery config. The broker publishes `offline` on an +ungraceful drop — no polling/timers on our side. Zero hot-path / memory cost. + +**3. State (extend `publishState`, ~cpp:341):** +- The existing change-gate (on/bri/palette vs `last*`) already fires "on change." Add, when + `haDiscovery` is on, a retained publish to `ha/state` of `{"state":"ON|OFF","brightness":<0-255>}` + (HA-scale brightness, no rescale) **inside that gated block** (not unconditional per tick — so an + inbound `ha/set`→setControl→publishState emits once, no loop). Retained so a late-joining HA gets + current state. One more `publish(...)` alongside the three `*/get` topics. + +**4. Control + wiring:** +- `addBool("haDiscovery", haDiscovery_)` in `onBuildControls` (~cpp:49), member default + `haDiscovery_ = true` (opt-out; `addBool` has no default-arg, and there's no per-deviceModel hook at + this layer). Retract-on-disable = an empty retained payload to the config topic + (`buildMqttPublish(…, 0, …, retain=true)` handles `payloadLen==0`). +- `onUpdate` (~cpp:62): the `haDiscovery` arm must **NOT `resetConnection`** (that bounces the socket). + Instead call `publishDiscovery(on/off)` directly, and on a mid-session turn-ON also SUBSCRIBE to + `ha/set` — announce/subscribe live, no reconnect. + +## Files + +- **Edit:** `src/core/MqttModule.h` (`haDiscovery_` field, new method decls), `src/core/MqttModule.cpp` + (announce + inbound branch + state + control + subscribe), `src/core/MqttPacket.h` *(only if a + discovery-config helper is cleaner there; likely not — `buildMqttPublish` suffices)*. +- **Tests:** `test/unit/core/unit_MqttPacket.cpp` — golden-vector (a) the CONNECT packet **with the + Last-Will** (byte-exact, the will-flag bits + will topic/payload), and (b) the retained + discovery-config PUBLISH (byte-exact topic + JSON + retain bit) — the pattern the `name` retain test + already uses. `test/unit/core/unit_MqttModule.cpp` — feed `ha/set` `{"state":"OFF"}` / + `{"brightness":128}` via `Rig::publish` and assert `FakeDrivers.on/brightness` (existing + effect-assertion style). For "CONNACK → module emits the retained config", add a **test-only capture + buffer in `sendPacket`** (the cleanest fit for the `feedForTest` pattern; `conn_` is concrete, a fake + socket is heavier than the module's conventions) and assert the captured config bytes. +- **Docs:** `docs/moonmodules/core/MqttModule` `///` (the topic list gains the `ha/set`/`ha/state` + + the discovery announce); `docs/usecases/home-automation.md` (an HA-via-Discovery recipe: "it just + appears" — plus keep the manual-topic + WLED paths); `docs/backlog/backlog-core.md` (the "HA MQTT + Discovery" item ships → delete it, per *Mandatory subtraction*). + +## Extensibility (the PO's question, answered in the design) + +JSON schema grows by adding a key to the config + the state/command JSON — no new topic, no new HA +entity type. Concretely: **presets/effects** → `effect_list:[…]` in the config + `{"effect":"Fire"}` +on the wire (HA renders a dropdown natively); **colour** (when palette→colour matures) → +`{"color":{"h":…,"s":…}}`. The default schema has no `effect` support at all — this is the deciding +reason for JSON. + +## Verification + +1. `cmake --build build` clean; `ctest` (the new golden-vector + inbound tests) + scenarios green; + `check_specs.py` green. +2. Byte-exact: the discovery-config PUBLISH matches the golden vector (topic + `homeassistant/light//config`, retain bit set, the JSON payload). +3. **HW (PO):** flash a WiFi board, set broker + `haDiscovery` on → HA auto-creates the light entity + (no manual YAML) → toggle on/off + brightness from HA, confirm the strip responds and HA reflects + state on device-side change. Toggle `haDiscovery` off → the entity disappears (retained config + retracted). Confirm the existing `projectMM//on/set` mqttthing path still works unchanged. +4. Platform boundary: all socket I/O via the existing `platform::TcpConnection`; no new platform code. + +Save the approved plan to `docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md`. diff --git a/docs/moonmodules/core/services.md b/docs/moonmodules/core/services.md index b62d38cc..521d5c7d 100644 --- a/docs/moonmodules/core/services.md +++ b/docs/moonmodules/core/services.md @@ -74,6 +74,7 @@ Bridges the light to an MQTT broker so a home-automation hub (Homebridge) can co - `broker` — the broker hostname (e.g. `homeassistant.lan`) or IP. A hostname is resolved via DNS. - `port` — broker port (default 1883). - `username` / `password` — broker credentials (optional; the password is stored obfuscated like the WiFi password). +- `haDiscovery` — announce a Home Assistant MQTT-discovery light (default on). HA auto-creates a wired entity; toggling it off removes the entity. See the [home-automation guide](../../usecases/home-automation.md). - read-only — `mqtt_status` (`disabled` / `idle` / `connecting` / `connected` / `disconnected` / an error). Detail: [technical](moxygen/MqttModule.md) diff --git a/docs/usecases/home-automation.md b/docs/usecases/home-automation.md index a3a56bbc..5d1a112b 100644 --- a/docs/usecases/home-automation.md +++ b/docs/usecases/home-automation.md @@ -139,7 +139,10 @@ To set it up: Homebridge above is the worked example; other home-automation platforms adopt the same device through the same primitives. The building blocks are shared — the [MQTT control surface](../moonmodules/core/services.md#mqtt) (broker + the `on`/`brightness`/`hsv` topics) and, for hubs that speak it, the device's WLED compatibility — so a new integration is a new front-end on top, not new device work. -- **Home Assistant** — HA adopts the device **without MQTT**: its built-in WLED integration discovers it over the WLED `/json` API the device already serves (the same interface the WLED apps use), so on/off + brightness work with no broker in the middle. For the full control surface, HA's MQTT Discovery reaches the same [control topics](../moonmodules/core/services.md#mqtt) below. +- **Home Assistant** — two ways, both zero-config: + - **MQTT auto-discovery (recommended when you run a broker).** With the MQTT module's `haDiscovery` control on (the default) and a broker set, the device announces itself and HA **auto-creates a light entity** — no manual topic-matching, no YAML. Just add the MQTT integration in HA pointed at the same broker; the light appears (named after the device) with on/off + brightness, and greys out when the device drops offline. This is the Tasmota/ESPHome-style discovery. + - **WLED integration (no broker needed).** HA's built-in WLED integration discovers the device over the WLED `/json` API it already serves (the same interface the WLED apps use), so on/off + brightness work with no MQTT at all. + - *Note:* don't hand-configure HA's generic "MQTT light" against `homeassistant/light/…` — that's HA's own topic schema, which the device answers only via the auto-discovery above. If you ever configure MQTT by hand, point it at the device's own `projectMM//…` topics (see the [MQTT reference](../moonmodules/core/services.md#mqtt)). - **Others** (Node-RED, openHAB, voice assistants via a hub, …) — anything that speaks MQTT drives the device through the [control topics](../moonmodules/core/services.md#mqtt); the broker setup is the same [install step](#install-a-broker-homebridge) as above, only the consumer differs. Each integration is a self-contained recipe; the shared MQTT reference lives once in [Core › Services › MQTT](../moonmodules/core/services.md#mqtt). diff --git a/src/core/MqttModule.cpp b/src/core/MqttModule.cpp index 3891d2d9..103922aa 100644 --- a/src/core/MqttModule.cpp +++ b/src/core/MqttModule.cpp @@ -1,6 +1,8 @@ #include "core/MqttModule.h" #include "core/Scheduler.h" // setControl — the shared apply-core +#include "core/JsonUtil.h" // json::hasKey/parseBool/parseInt/parseString — the inbound ha/set parse + // (same flat helpers HttpServerModule::applyWledState uses; no arena) #include "light/Palette.h" // Palettes::nearestForHue — a pure hue/sat→index CONVERSION with no // light state or objects, the one narrow reach this core module makes // into the light domain. PO-accepted: routing a HomeKit colour to a @@ -41,16 +43,128 @@ void MqttModule::buildTopic(char* out, size_t cap, const char* suffix) const { std::snprintf(out, cap, "%s/%s", prefix, suffix); } +// The HA MQTT-discovery config topic: homeassistant/light/projectMM_/config. Independent of +// topicPrefix() — the discovery prefix is HA's `homeassistant`, not our `projectMM` root, and the +// object id carries the projectMM_ prefix so the id is unique across vendors on a shared broker. +void MqttModule::buildDiscoveryTopic(char* out, size_t cap) const { + uint8_t mac[6] = {}; + platform::getMacAddress(mac); + std::snprintf(out, cap, "homeassistant/light/%s_%02x%02x%02x/config", + prefixRoot_, mac[3], mac[4], mac[5]); +} + +// The availability (LWT) topic: /status. The broker publishes the retained "offline" Will +// here on an ungraceful drop; the module publishes retained "online" on connect. HA's avty_t points +// at it, so the entity greys out when the device disappears. +void MqttModule::buildStatusTopic(char* out, size_t cap) const { + buildTopic(out, cap, "status"); +} + +// Lazily allocate the two discovery scratch buffers from the heap the first time discovery publishes, +// and report them via setDynamicBytes so the UI's per-module memory line accounts for them. A device +// that never enables HA discovery never calls this → zero bytes. Returns false on OOM (the caller then +// skips the publish rather than deref a null — the module keeps running, discovery just doesn't announce). +bool MqttModule::ensureDiscoveryBuffers() { + if (discoveryBuf_ && discoveryPayload_) return true; + if (!discoveryBuf_) discoveryBuf_ = static_cast(platform::alloc(kDiscoveryBufLen)); + if (!discoveryPayload_) discoveryPayload_ = static_cast(platform::alloc(kDiscoveryPayloadLen)); + if (!discoveryBuf_ || !discoveryPayload_) { freeDiscoveryBuffers(); return false; } + setDynamicBytes(kDiscoveryBufLen + kDiscoveryPayloadLen); + return true; +} + +void MqttModule::freeDiscoveryBuffers() { + if (discoveryBuf_) { platform::free(discoveryBuf_); discoveryBuf_ = nullptr; } + if (discoveryPayload_) { platform::free(discoveryPayload_); discoveryPayload_ = nullptr; } + setDynamicBytes(0); +} + +void MqttModule::publishDiscovery(bool announce) { + // Retract (OFF): free the buffers unconditionally — freeing local memory needs no socket, so this + // runs even while disconnected/reconnecting (else a discovery-off toggle mid-reconnect would strand + // the buffers until teardown, breaking "no memory when discovery is off"). When we happen to be + // connected we ALSO send the empty retained payload so HA removes the entity; when not, the buffers + // still free and the entity clears on the broker's own retained-message expiry / next connect. + if (!announce) { + if (state_ == Conn::Connected && discoveryBuf_) { + char topic[96]; + buildDiscoveryTopic(topic, sizeof(topic)); + const size_t n = buildMqttPublish(topic, nullptr, 0, discoveryBuf_, kDiscoveryBufLen, + /*retain=*/true); + if (n != 0) sendPacket(discoveryBuf_, n); + } + freeDiscoveryBuffers(); + return; + } + + // Announce path needs a live socket (there's nothing to publish to otherwise) and allocates. + if (state_ != Conn::Connected) return; + if (!ensureDiscoveryBuffers()) { setStatusLine("error: discovery alloc failed"); return; } + + char topic[96]; + buildDiscoveryTopic(topic, sizeof(topic)); + + // Identity + display name. uniq_id/object_id derive from the stable MAC; name is the friendly + // deviceName (a rename updates the label, never the id — see the topic-identity decision). + uint8_t mac[6] = {}; + platform::getMacAddress(mac); + char id[24]; + std::snprintf(id, sizeof(id), "%s_%02x%02x%02x", prefixRoot_, mac[3], mac[4], mac[5]); + const char* dn = systemModule_ ? systemModule_->deviceName() : nullptr; + if (!dn || !dn[0]) dn = id; + + char cmd[80], stat[80], avty[80]; + buildTopic(cmd, sizeof(cmd), "ha/set"); + buildTopic(stat, sizeof(stat), "ha/state"); + buildStatusTopic(avty, sizeof(avty)); + + // JSON-schema MQTT light. Abbreviated keys (HA's documented short forms). brightness at the + // default 0-255 scale (no scale key needed). dev{} groups the entity under a device card in HA. + const int pn = std::snprintf(discoveryPayload_, kDiscoveryPayloadLen, + "{\"schema\":\"json\",\"name\":\"%s\",\"uniq_id\":\"%s\",\"cmd_t\":\"%s\"," + "\"stat_t\":\"%s\",\"avty_t\":\"%s\",\"brightness\":true," + "\"dev\":{\"ids\":[\"%s\"],\"name\":\"%s\",\"mf\":\"MoonModules\",\"mdl\":\"projectMM\"}}", + dn, id, cmd, stat, avty, id, dn); + if (pn <= 0 || static_cast(pn) >= kDiscoveryPayloadLen) return; // truncated → don't send a broken config + + const size_t n = buildMqttPublish(topic, reinterpret_cast(discoveryPayload_), + static_cast(pn), discoveryBuf_, kDiscoveryBufLen, + /*retain=*/true); + if (n == 0) { setStatusLine("error: discovery config too large"); return; } + sendPacket(discoveryBuf_, n); +} + +// SUBSCRIBE to /ha/set — the HA-native JSON command topic. Called at CONNACK (with the +// mqttthing set-topics) and on a mid-session haDiscovery turn-on (subscriptions otherwise only happen +// once at CONNACK). +void MqttModule::subscribeHaSet() { + if (state_ != Conn::Connected) return; + char topic[96]; + buildTopic(topic, sizeof(topic), "ha/set"); + uint8_t buf[kSendBufLen]; + const size_t n = buildMqttSubscribe(nextPacketId_++, topic, buf, sizeof(buf)); + if (n != 0) sendPacket(buf, n); +} + void MqttModule::setup() { setStatusLine(enabled() ? "idle" : "disabled"); MoonModule::setup(); } +// Release the lazily-allocated discovery buffers when the module is torn down (deleted from the tree, +// or on device shutdown). The socket is closed via the normal reset path; MoonModule::teardown() +// recurses to children (this module has none). No memory outlives the module. +void MqttModule::teardown() { + freeDiscoveryBuffers(); + MoonModule::teardown(); +} + void MqttModule::onBuildControls() { controls_.addText("broker", broker_, sizeof(broker_)); controls_.addUint16("port", port_, 1, 65535); controls_.addText("username", username_, sizeof(username_)); controls_.addPassword("password", password_, sizeof(password_)); + controls_.addBool("haDiscovery", haDiscovery_); // announce a HA-discovery light (default on) controls_.addReadOnly("mqtt_status", statusStr_, sizeof(statusStr_)); MoonModule::onBuildControls(); } @@ -63,6 +177,12 @@ void MqttModule::onUpdate(const char* controlName) { if (std::strcmp(controlName, "broker") == 0 || std::strcmp(controlName, "port") == 0 || std::strcmp(controlName, "username") == 0 || std::strcmp(controlName, "password") == 0) { resetConnection(enabled() ? "reconnecting" : "disabled"); + } else if (std::strcmp(controlName, "haDiscovery") == 0) { + // Announce or retract live — NO reset (bouncing the socket to change a discovery flag is + // needless). On a turn-ON mid-session also SUBSCRIBE (subscriptions otherwise only fire at + // CONNACK); on turn-OFF the retract clears the HA entity. Publishes only when connected. + publishDiscovery(haDiscovery_); + if (haDiscovery_) { subscribeHaSet(); publishState(true); } } } @@ -75,6 +195,7 @@ void MqttModule::onEnabled(bool enabled) { const size_t n = buildMqttDisconnect(buf, sizeof(buf)); if (n) sendPacket(buf, n); // best-effort courtesy DISCONNECT } + if (!enabled) freeDiscoveryBuffers(); // MQTT turned off → not used → reclaim the discovery heap resetConnection(enabled ? "idle" : "disabled"); } @@ -85,10 +206,23 @@ void MqttModule::onEnabled(bool enabled) { // the blocking write() would do inside loop1s (the hot-path violation this avoids). bool MqttModule::sendPacket(const uint8_t* data, size_t len) { if (len == 0) return true; + // Test seam: mirror the outbound bytes into the capture buffer (null in production) so a unit test + // can assert what the module emits — there's no live socket in ctest, and writeSome returns -1. + if (sendCapture_) { + if (sendCaptureLen_ + len <= sendCaptureCap_) { + std::memcpy(sendCapture_ + sendCaptureLen_, data, len); + sendCaptureLen_ += len; + } + return true; // capture mode always "succeeds" so the connect/publish flow proceeds in tests + } const int sent = conn_.writeSome(data, len); return sent == static_cast(len); // all-or-fail; a partial/0/-1 is a connection problem } +void MqttModule::enableSendCaptureForTest(uint8_t* buf, size_t cap) { + sendCapture_ = buf; sendCaptureCap_ = cap; sendCaptureLen_ = 0; +} + // Close the socket and return to Idle with a status line — the single reset path so every caller // (reconfig, disable, timeout, protocol error, peer close) leaves the same clean state. void MqttModule::resetConnection(const char* status) { @@ -170,7 +304,13 @@ void MqttModule::sendConnectPacket() { platform::getMacAddress(mac); char clientId[32]; std::snprintf(clientId, sizeof(clientId), "projectMM-%02x%02x%02x", mac[3], mac[4], mac[5]); - const size_t n = buildMqttConnect(clientId, user, pass, kKeepaliveSec, buf, sizeof(buf)); + // Last Will: retained "offline" on /status. The broker publishes it if we drop + // ungracefully (power cut, WiFi loss), so HA's avty_t greys the entity out. Declared here at + // CONNECT; we publish the retained "online" ourselves once CONNACK lands (handleInboundByte). + char willTopic[96]; + buildStatusTopic(willTopic, sizeof(willTopic)); + const size_t n = buildMqttConnect(clientId, user, pass, kKeepaliveSec, buf, sizeof(buf), + willTopic, "offline", /*willRetain=*/true); if (n == 0 || !sendPacket(buf, n)) { resetConnection("error: connect send failed"); return; @@ -251,7 +391,18 @@ void MqttModule::handleInboundByte(uint8_t byte) { setStatusLine("connected"); havePublished_ = false; publishName(); // retained friendly name so a hub shows the display name - publishState(true); // publish initial state so mqttthing shows it + // Availability: publish retained "online" to /status (the LWT's counterpart — the + // broker publishes "offline" if we drop). Must precede the discovery announce so HA sees the + // entity available the instant its config lands. + { + char st[96]; buildStatusTopic(st, sizeof(st)); + uint8_t sb[kSendBufLen]; + const size_t sn = buildMqttPublish(st, reinterpret_cast("online"), 6, + sb, sizeof(sb), /*retain=*/true); + if (sn != 0) sendPacket(sb, sn); + } + if (haDiscovery_) { publishDiscovery(true); subscribeHaSet(); } + publishState(true); // publish initial state so mqttthing + HA show it } else if (type == static_cast(MqttPacketType::Publish)) { const char* topic = nullptr; const uint8_t* payload = nullptr; size_t plLen = 0; if (parser_.publish(&topic, &payload, &plLen)) routePublish(topic, payload, plLen); @@ -268,6 +419,33 @@ void MqttModule::routePublish(const char* topic, const uint8_t* payload, size_t if (std::strncmp(topic, prefix, prefixLen) != 0 || topic[prefixLen] != '/') return; const char* suffix = topic + prefixLen + 1; + // HA-native JSON command: {"state":"ON"|"OFF"[,"brightness":0-255]}. Parsed with the same flat + // mm::json helpers HttpServerModule::applyWledState uses (key-order-independent, whitespace-safe); + // needs a bigger NUL-terminated buffer than the scalar `value[32]` below. HA brightness is already + // 0-255, so no rescale (unlike the mqttthing brightness/set 0-100 path). + if (std::strcmp(suffix, "ha/set") == 0) { + char body[128]; + const size_t blen = payloadLen < sizeof(body) - 1 ? payloadLen : sizeof(body) - 1; + std::memcpy(body, payload, blen); + body[blen] = '\0'; + if (json::hasKey(body, "state")) { + char st[8] = ""; + json::parseString(body, "state", st, sizeof(st)); + const bool on = (std::strcmp(st, "ON") == 0); + setControlValue("on", on ? "{\"value\":true}" : "{\"value\":false}"); + } + if (json::hasKey(body, "brightness")) { + int bri = json::parseInt(body, "brightness"); + if (bri < 0) bri = 0; + if (bri > 255) bri = 255; + char json[24]; + std::snprintf(json, sizeof(json), "{\"value\":%d}", bri); + setControlValue("brightness", json); + } + // Future: an "effect" key → setControlValue("palette"/"effect", …) with effect_list in the config. + return; + } + char value[32]; const size_t vlen = payloadLen < sizeof(value) - 1 ? payloadLen : sizeof(value) - 1; std::memcpy(value, payload, vlen); @@ -374,6 +552,20 @@ void MqttModule::publishState(bool force) { return; } + // HA-native state on /ha/state (retained, so a late-joining HA gets current state). One + // JSON message with 0-255 brightness (no rescale, unlike brightness/get's 0-100). Only when + // discovery is on, and inside this change-gated block so it emits once per change, not per tick. + if (haDiscovery_) { + char haState[48]; + std::snprintf(haState, sizeof(haState), "{\"state\":\"%s\",\"brightness\":%u}", + on ? "ON" : "OFF", static_cast(bri)); + char haTopic[128]; + buildTopic(haTopic, sizeof(haTopic), "ha/state"); + const size_t n = buildMqttPublish(haTopic, reinterpret_cast(haState), + std::strlen(haState), buf, sizeof(buf), /*retain=*/true); + if (n == 0 || !sendPacket(buf, n)) { resetConnection("error: state publish failed"); return; } + } + lastOn_ = on; lastBri_ = bri; lastPalette_ = pal; havePublished_ = true; } diff --git a/src/core/MqttModule.h b/src/core/MqttModule.h index bc4e8058..9ae36c0e 100644 --- a/src/core/MqttModule.h +++ b/src/core/MqttModule.h @@ -32,22 +32,38 @@ namespace mm { /// `/hsv/set` ← "h,s,v" → hue+sat → nearest palette → Drivers.palette /// `/hsv/get` → publish the chosen palette's representative "h,s,v" /// +/// **Home Assistant MQTT Discovery** (the `haDiscovery` control, default on). On connect the device +/// publishes a RETAINED JSON-schema light config to `homeassistant/light/projectMM_/config`, so +/// HA (and any Discovery-aware hub — the Tasmota/ESPHome/Zigbee2MQTT convention) **auto-creates a +/// wired light entity** — no hand-matching topics. It then speaks HA's own schema alongside the +/// mqttthing topics above: +/// `/ha/set` ← `{"state":"ON"|"OFF"[,"brightness":0-255]}` → Drivers.on/brightness +/// `/ha/state` → retained `{"state":…,"brightness":0-255}` on change (HA-scale, no rescale) +/// `/status` → retained "online"; the CONNECT **Last-Will** publishes "offline" here +/// on an ungraceful drop, so HA greys the entity out (`avty_t`) +/// Toggling `haDiscovery` re-announces / retracts live (an empty retained config removes the entity), +/// no reconnect. `uniq_id` is the MAC-stable `projectMM_`, never the editable name. JSON schema +/// (not the default schema) so future controls add a key — HA's native `effect`/`effect_list` maps a +/// preset/effect picker with no new topic. +/// /// **Lifecycle** (all on loop1s(), off the render hot path — MQTT is slow control): connect lazily /// once `networkReady() && enabled`, CONNECT → CONNACK → SUBSCRIBE to the `set` topics, PINGREQ every /// keepalive/2, drain inbound bytes through MqttInboundParser and route PUBLISHes to Drivers, and /// publish the `get` topics whenever the local value changes (and on connect, so mqttthing never /// reads "No Response"). A dropped socket reconnects with a backoff. /// -/// **Prior art:** the OASIS MQTT 3.1.1 standard + homebridge-mqttthing's topic conventions (see -/// docs/moonmodules/core/services.md#mqtt for the Homebridge accessory config). The MoonLight sibling -/// bridges the same on/off+brightness surface through a full framework MQTT client + HA discovery; -/// projectMM writes its own lean client over the platform socket primitive instead. +/// **Prior art:** the OASIS MQTT 3.1.1 standard, homebridge-mqttthing's topic conventions, and Home +/// Assistant's MQTT-discovery format (the same retained-`homeassistant/…/config` announce Tasmota / +/// ESPHome / Zigbee2MQTT use). projectMM writes its own lean client over the platform socket +/// primitive rather than a framework MQTT library. See docs/moonmodules/core/services.md#mqtt for the +/// Homebridge accessory config; docs/usecases/home-automation.md for the HA setup. /// @card MqttModule.png class MqttModule : public MoonModule { public: void setSystemModule(SystemModule* s) { systemModule_ = s; } void setup() override; + void teardown() override; // free the lazily-allocated discovery buffers void onBuildControls() override; void onUpdate(const char* controlName) override; // a broker/port/cred change re-homes the socket void onEnabled(bool enabled) override; // enable/disable → connect / clean DISCONNECT @@ -57,6 +73,13 @@ class MqttModule : public MoonModule { /// drive (there's no live broker in ctest). Mirrors IrModule::injectCodeForTest. void feedForTest(const uint8_t* bytes, size_t len); + /// Test seam: capture every outbound packet sendPacket() writes, so a unit test can assert what + /// the module emits (e.g. the retained discovery config on CONNACK) — there's no live socket in + /// ctest. Enable before the exercise; read back the concatenated bytes. Off in production (the + /// capture buffer is null). + void enableSendCaptureForTest(uint8_t* buf, size_t cap); + size_t sentCaptureLenForTest() const { return sendCaptureLen_; } + private: // Connection state machine — advanced by loop1s(). ConnectingTcp = a non-blocking TCP connect is // in flight (polled, never blocks the tick); Connecting = TCP up, CONNECT sent, awaiting CONNACK. @@ -75,6 +98,14 @@ class MqttModule : public MoonModule { void setControlValue(const char* control, const char* valueJson); // → Scheduler::setControl void setStatusLine(const char* msg); + // Home Assistant MQTT Discovery (JSON schema). When haDiscovery_ is on, the device announces a + // retained `homeassistant/light//config` so HA auto-creates a wired light entity; it then + // speaks HA's own JSON schema on /ha/{set,state} alongside the mqttthing on/set etc. + void buildDiscoveryTopic(char* out, size_t cap) const; // homeassistant/light/projectMM_/config + void buildStatusTopic(char* out, size_t cap) const; // /status — the LWT availability topic + void publishDiscovery(bool announce); // announce=false publishes an empty retained config (retract) + void subscribeHaSet(); // SUBSCRIBE to /ha/set (at CONNACK + on live toggle-on) + SystemModule* systemModule_ = nullptr; // The topic prefix is DERIVED from a STABLE hardware id: projectMM/. Not stored (no @@ -91,6 +122,7 @@ class MqttModule : public MoonModule { uint16_t port_ = 1883; char username_[48] = ""; char password_[48] = ""; + bool haDiscovery_ = true; // announce a HA MQTT-discovery light (opt-out); see publishDiscovery char statusStr_[64] = "disabled"; platform::TcpConnection conn_; @@ -111,6 +143,25 @@ class MqttModule : public MoonModule { bool lastConnectFailed_ = false; // widen the backoff after a failure (esp. a slow DNS lookup) + // Discovery-config scratch — HEAP, allocated lazily only when discovery actually publishes + // (connected + haDiscovery on), freed in teardown() and when discovery is turned off, per the + // pay-for-what-you-use rule (architecture.md § Memory strategy): a device that has the MQTT module + // but never enables HA discovery pays ZERO for it. Heap (not a fixed member) also keeps the ~360 B + // config frame off the shared 8 KB main-task stack (the P4 registerType-stack lesson). Two regions + // because buildMqttPublish needs payload + output separate: the JSON builds into discoveryPayload_, + // the framed packet into discoveryBuf_. + static constexpr size_t kDiscoveryPayloadLen = 320; + static constexpr size_t kDiscoveryBufLen = 448; + char* discoveryPayload_ = nullptr; + uint8_t* discoveryBuf_ = nullptr; + bool ensureDiscoveryBuffers(); // lazily alloc both; false on OOM. Sets dynamicBytes. + void freeDiscoveryBuffers(); // free both + dynamicBytes(0). Called on teardown / discovery-off. + + // Test-only outbound capture (null in production). sendPacket appends every emitted packet here. + uint8_t* sendCapture_ = nullptr; + size_t sendCaptureCap_ = 0; + size_t sendCaptureLen_ = 0; + static constexpr uint16_t kKeepaliveSec = 30; static constexpr uint32_t kReconnectBackoffMs = 5000; // after a clean disconnect static constexpr uint32_t kFailedBackoffMs = 30000; // after a failed attempt — a bad diff --git a/src/core/MqttPacket.h b/src/core/MqttPacket.h index 2a0dc1ec..ddc76eda 100644 --- a/src/core/MqttPacket.h +++ b/src/core/MqttPacket.h @@ -41,8 +41,11 @@ inline constexpr uint8_t kMqttProtocolLevel = 0x04; // 4 == MQTT 3.1.1 // CONNECT flag bits (§3.1.2.3). We use clean-session + optionally username/password. inline constexpr uint8_t kMqttConnectCleanSession = 0x02; +inline constexpr uint8_t kMqttConnectWillFlag = 0x04; // §3.1.2.5 — a Will is present in the payload +inline constexpr uint8_t kMqttConnectWillRetain = 0x20; // §3.1.2.7 — the broker publishes the Will retained inline constexpr uint8_t kMqttConnectPasswordFlag = 0x40; inline constexpr uint8_t kMqttConnectUsernameFlag = 0x80; +// Will QoS (§3.1.2.6) is bits 3-4; QoS 0 = 0x00, so no constant needed. // --8<-- [end:mqtt-constants] // --- Remaining-length varint (§2.2.3) --- @@ -119,30 +122,43 @@ inline size_t mqttWriteFixedHeader(uint8_t* out, size_t outLen, return 1 + rl; // total fixed-header size } -// CONNECT (§3.1). clientId is required; username/password optional (nullptr → omitted). keepalive is +// CONNECT (§3.1). clientId is required; username/password optional (nullptr → omitted). An optional +// **Last Will** (willTopic + willPayload) makes the broker publish willPayload to willTopic when this +// client drops ungracefully (power cut, WiFi loss) — the availability seam HA greys the entity out on. +// willRetain keeps the Will retained so a late-joining subscriber still sees "offline". keepalive is // in seconds. Clean session always set (we don't resume state). Returns total bytes, 0 on overflow. inline size_t buildMqttConnect(const char* clientId, const char* username, const char* password, - uint16_t keepaliveSec, uint8_t* out, size_t outLen) { + uint16_t keepaliveSec, uint8_t* out, size_t outLen, + const char* willTopic = nullptr, const char* willPayload = nullptr, + bool willRetain = false) { if (!clientId) return 0; // Treat an empty-string username as "no username" (a UI text control left blank is "", not null), // and — MQTT-3.1.2-22 — the password flag MUST NOT be set without the username flag, so a password // with no username is dropped (a compliant broker would reject the whole CONNECT otherwise). if (username && !username[0]) username = nullptr; if (!username) password = nullptr; + // The Will needs both a topic and a payload; a partial one is dropped (§3.1.2.5 — the Will flag + // gates both length-prefixed fields, so a topic with no message is malformed). + if (willTopic && !willTopic[0]) willTopic = nullptr; + if (!willTopic || !willPayload) { willTopic = nullptr; willPayload = nullptr; } const size_t idLen = std::strlen(clientId); const size_t userLen = username ? std::strlen(username) : 0; const size_t passLen = password ? std::strlen(password) : 0; + const size_t wtLen = willTopic ? std::strlen(willTopic) : 0; + const size_t wpLen = willPayload ? std::strlen(willPayload) : 0; // Variable header: protocol name (6) + level (1) + connect flags (1) + keepalive (2) = 10. - // Payload: clientId, then username, then password — each length-prefixed. + // Payload order (§3.1.3): clientId, Will Topic, Will Message, username, password — each len-prefixed. uint8_t connectFlags = kMqttConnectCleanSession; + if (willTopic) { connectFlags |= kMqttConnectWillFlag; if (willRetain) connectFlags |= kMqttConnectWillRetain; } if (username) connectFlags |= kMqttConnectUsernameFlag; if (password) connectFlags |= kMqttConnectPasswordFlag; const uint32_t bodyLen = static_cast( 2 + 4 + 1 + 1 + 2 + // proto name(len+"MQTT") + level + flags + keepalive 2 + idLen + + (willTopic ? 2 + wtLen + 2 + wpLen : 0) + (username ? 2 + userLen : 0) + (password ? 2 + passLen : 0)); @@ -158,9 +174,13 @@ inline size_t buildMqttConnect(const char* clientId, out[pos++] = static_cast((keepaliveSec >> 8) & 0xFF); out[pos++] = static_cast(keepaliveSec & 0xFF); - // Payload + // Payload — clientId, [Will Topic, Will Message], [username], [password], in §3.1.3 order. pos = mqttAppendString(out, outLen, pos, clientId, idLen); if (pos == 0) return 0; + if (willTopic) { + pos = mqttAppendString(out, outLen, pos, willTopic, wtLen); if (pos == 0) return 0; + pos = mqttAppendString(out, outLen, pos, willPayload, wpLen); if (pos == 0) return 0; + } if (username) { pos = mqttAppendString(out, outLen, pos, username, userLen); if (pos == 0) return 0; } if (password) { pos = mqttAppendString(out, outLen, pos, password, passLen); if (pos == 0) return 0; } return pos; diff --git a/test/scenarios/core/scenario_MqttModule_haDiscovery_toggle.json b/test/scenarios/core/scenario_MqttModule_haDiscovery_toggle.json new file mode 100644 index 00000000..bc629d2a --- /dev/null +++ b/test/scenarios/core/scenario_MqttModule_haDiscovery_toggle.json @@ -0,0 +1,69 @@ +{ + "name": "scenario_MqttModule_haDiscovery_toggle", + "module": "MqttModule", + "mode": "mutate", + "live_only": true, + "description": "Toggle the Home-Assistant MQTT-discovery announce on and off and confirm the render loop survives every change. Pins the robustness + No-reboot-to-apply guarantees for the new haDiscovery control: turning it on publishes a retained homeassistant/light//config (and subscribes ha/set); turning it off retracts the config (empty retained payload) — both live over the already-connected socket on loop1s(), never a reconnect and never on the render task. No toggle, in any order, may crash, wedge the tick, or leak heap. Runs live only — discovery needs a real broker to publish to and MqttModule only exists on a board whose catalog entry includes it (the Shelly model does), so the in-process desktop runner SKIPs it; on a device it flips the control over HTTP. The bound checks FPS + heap stay within range across the on/off/on cycle (a discovery publish is a transient cost, not a permanent degradation), proving the toggle never leaves the network task or render loop degraded. (Background: the announce/retract are on the 1 Hz tick, off the hot path — see the MqttModule /// § Home Assistant MQTT Discovery.)", + "reset": [ + { + "name": "reset-haDiscovery", + "op": "set_control", + "id": "Mqtt", + "key": "haDiscovery", + "value": true + } + ], + "steps": [ + { + "name": "baseline-discovery-on", + "description": "haDiscovery on (default) — captures the baseline FPS/heap for the next two steps.", + "op": "set_control", + "id": "Mqtt", + "key": "haDiscovery", + "value": true, + "measure": true, + "contract": { + "esp32-eth-wifi": { + "tick_us": 100000, + "free_heap": 90000, + "set_by": "2026-07-06", + "reason": "initial contract — pending first hardware capture" + } + } + }, + { + "name": "discovery-off", + "description": "haDiscovery off — the retract publish (empty retained config) goes out; FPS/heap must match or exceed the baseline.", + "op": "set_control", + "id": "Mqtt", + "key": "haDiscovery", + "value": false, + "measure": true, + "contract": { + "esp32-eth-wifi": { + "tick_us": 100000, + "free_heap": 90000, + "set_by": "2026-07-06", + "reason": "initial contract — pending first hardware capture" + } + } + }, + { + "name": "discovery-on-again", + "description": "haDiscovery on again — re-announce + re-subscribe live. Bound: FPS must stay within 20% of the baseline (proves toggling doesn't leave the network task in a degraded state).", + "op": "set_control", + "id": "Mqtt", + "key": "haDiscovery", + "value": true, + "measure": true, + "contract": { + "esp32-eth-wifi": { + "tick_us": 100000, + "free_heap": 90000, + "set_by": "2026-07-06", + "reason": "initial contract — pending first hardware capture" + } + } + } + ] +} diff --git a/test/unit/core/unit_MqttModule.cpp b/test/unit/core/unit_MqttModule.cpp index 65c75856..3cab8b81 100644 --- a/test/unit/core/unit_MqttModule.cpp +++ b/test/unit/core/unit_MqttModule.cpp @@ -17,6 +17,7 @@ #include #include +#include #include using namespace mm; @@ -115,6 +116,93 @@ TEST_CASE("MqttModule: hsv/set maps a hue to the nearest palette + value to brig CHECK(r.drivers->brightness == (40 * 255) / 100); // value → brightness } +// --- Home Assistant MQTT Discovery: the HA-native ha/set command topic --- +// HA drives the JSON-schema light with {"state":"ON"|"OFF"[,"brightness":0-255]} on /ha/set. +// Unlike the mqttthing brightness/set (0-100), HA brightness is already 0-255 — no rescale. + +TEST_CASE("MqttModule: ha/set {state} drives Drivers.on") { + Rig r; + r.drivers->on = true; + r.publish("ha/set", "{\"state\":\"OFF\"}"); + CHECK(r.drivers->on == false); + r.publish("ha/set", "{\"state\":\"ON\"}"); + CHECK(r.drivers->on == true); +} + +TEST_CASE("MqttModule: ha/set {brightness} maps 0-255 with no rescale") { + Rig r; + r.publish("ha/set", "{\"state\":\"ON\",\"brightness\":128}"); + CHECK(r.drivers->on == true); + CHECK(r.drivers->brightness == 128); // HA is already 0-255 — no *255/100 + r.publish("ha/set", "{\"brightness\":255}"); + CHECK(r.drivers->brightness == 255); + // Out-of-range clamps. + r.publish("ha/set", "{\"brightness\":999}"); + CHECK(r.drivers->brightness == 255); +} + +TEST_CASE("MqttModule: ha/set is key-order-independent") { + Rig r; + r.drivers->on = false; + // brightness before state — mm::json's strstr lookup is order-independent (HA emits compact JSON; + // the flat helpers match `"key":` / `"key": `, so we feed the same no-inner-space shape HA sends). + r.publish("ha/set", "{\"brightness\":64,\"state\":\"ON\"}"); + CHECK(r.drivers->on == true); + CHECK(r.drivers->brightness == 64); +} + +TEST_CASE("MqttModule: ha/set with only state leaves brightness untouched") { + Rig r; + r.drivers->brightness = 200; + r.publish("ha/set", "{\"state\":\"OFF\"}"); // no brightness key + CHECK(r.drivers->on == false); + CHECK(r.drivers->brightness == 200); // unchanged (hasKey guard) +} + +// The discovery announce: on CONNACK the module publishes a RETAINED config to +// homeassistant/light/projectMM_/config. Assert via the outbound-capture seam (no live socket). +TEST_CASE("MqttModule: CONNACK publishes a retained HA discovery config") { + Rig r; + uint8_t cap[1024]; + r.mqtt->enableSendCaptureForTest(cap, sizeof(cap)); + // A CONNACK-accept: fixed header 0x20, len 2, session-present 0, return-code 0 (accepted). + const uint8_t connack[] = {0x20, 0x02, 0x00, 0x00}; + r.mqtt->feedForTest(connack, sizeof(connack)); + const size_t len = r.mqtt->sentCaptureLenForTest(); + REQUIRE(len > 0); + // The captured stream must contain the discovery topic, the retain bit, and the key config fields. + std::string sent(reinterpret_cast(cap), len); + CHECK(sent.find("homeassistant/light/projectMM_efcafe/config") != std::string::npos); + CHECK(sent.find("\"schema\":\"json\"") != std::string::npos); + CHECK(sent.find("\"uniq_id\":\"projectMM_efcafe\"") != std::string::npos); + CHECK(sent.find("projectMM/efcafe/ha/set") != std::string::npos); // cmd_t + CHECK(sent.find("projectMM/efcafe/ha/state") != std::string::npos); // stat_t + CHECK(sent.find("projectMM/efcafe/status") != std::string::npos); // avty_t + CHECK(sent.find("online") != std::string::npos); // the retained availability publish +} + +// Regression (found live on P4/S31 hardware): turning haDiscovery OFF must free the discovery buffers +// EVEN when the socket is not currently Connected (mid-reconnect). The original guard bailed on +// `state_ != Connected` before reaching the free, so a discovery-off toggle during a reconnect stranded +// the 768 B until teardown — breaking "no memory when discovery is off". Freeing local memory needs no +// socket, so the retract path frees unconditionally; only the empty-retained PUBLISH needs a live link. +TEST_CASE("MqttModule: retract frees the discovery buffers even while disconnected") { + Rig r; + uint8_t cap[1024]; + r.mqtt->enableSendCaptureForTest(cap, sizeof(cap)); + // CONNACK-accept → Connected → announce allocates the discovery buffers (448 + 320 = 768). + const uint8_t connack[] = {0x20, 0x02, 0x00, 0x00}; + r.mqtt->feedForTest(connack, sizeof(connack)); + CHECK(r.mqtt->dynamicBytes() == 768); + // A broker change re-homes the socket → resetConnection drops state to Idle, buffers still held + // (a reconnect must not churn the heap). Now we're "allocated but not Connected". + Scheduler::instance()->setControl("Mqtt", "broker", "{\"value\":\"10.0.0.9\"}"); + CHECK(r.mqtt->dynamicBytes() == 768); // reset kept the buffers (correct) + // Toggle discovery OFF while disconnected — must free despite no live socket. + Scheduler::instance()->setControl("Mqtt", "haDiscovery", "{\"value\":false}"); + CHECK(r.mqtt->dynamicBytes() == 0); // the fix: retract freed even while not Connected +} + TEST_CASE("MqttModule: a PUBLISH on an unrelated topic is ignored, not a crash") { Rig r; const uint8_t beforeBri = r.drivers->brightness; diff --git a/test/unit/core/unit_MqttPacket.cpp b/test/unit/core/unit_MqttPacket.cpp index 2fdeb14d..7131c595 100644 --- a/test/unit/core/unit_MqttPacket.cpp +++ b/test/unit/core/unit_MqttPacket.cpp @@ -117,6 +117,27 @@ TEST_CASE("MqttPacket: CONNECT drops a password when there is no username") { CHECK(out[9] == kMqttConnectCleanSession); } +// A Last Will (§3.1.2.5-.7 + §3.1.3) makes the broker publish the will payload if the client drops — +// the availability seam HA greys the entity out on. Will Topic + Will Message go in the payload AFTER +// the clientId and BEFORE username/password; the flags byte carries the Will + Will-Retain bits. +TEST_CASE("MqttPacket: CONNECT with a retained Last Will is byte-exact") { + uint8_t out[96] = {}; + const size_t n = buildMqttConnect("c", nullptr, nullptr, 15, out, sizeof(out), + "s", "offline", /*willRetain=*/true); + REQUIRE(n > 0); + // Flags: clean-session | will | will-retain (no username/password). + CHECK(out[9] == (kMqttConnectCleanSession | kMqttConnectWillFlag | kMqttConnectWillRetain)); + // Payload tail: 00 01 'c' (clientId) 00 01 's' (will topic) 00 07 'offline' (will message). + const uint8_t tail[] = {0x00, 0x01, 'c', + 0x00, 0x01, 's', + 0x00, 0x07, 'o', 'f', 'f', 'l', 'i', 'n', 'e'}; + CHECK(std::memcmp(out + n - sizeof(tail), tail, sizeof(tail)) == 0); + // A partial will (topic but no message) is dropped: no will flag, no will fields. + const size_t n2 = buildMqttConnect("c", nullptr, nullptr, 15, out, sizeof(out), "s", nullptr, true); + REQUIRE(n2 > 0); + CHECK(out[9] == kMqttConnectCleanSession); +} + // --- PUBLISH golden vector (§3.3), QoS0 --- TEST_CASE("MqttPacket: PUBLISH is byte-exact (QoS0)") { uint8_t out[64] = {}; From b360706097927e7b394272518330919fb0633cae Mon Sep 17 00:00:00 2001 From: ewowi Date: Tue, 7 Jul 2026 09:42:24 +0200 Subject: [PATCH 2/3] Process CodeRabbit MQTT findings + docs overhaul (ADRs, lessons split, architecture trim) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two areas land together. First, the eight CodeRabbit findings on the HA MQTT Discovery PR are processed and hardware-verified: the discovery config now escapes the editable device name, a discovery-off retract reaches the broker even mid-reconnect, failed HA sends reset the connection like the sibling paths, and unknown HA state values are ignored rather than mapped to OFF. Second, the accumulated decision/lesson knowledge is reorganised into the industry-standard shape: architectural decisions become immutable ADRs, the old decisions.md is split into a pruned lessons log, and architecture.md is trimmed of decision-rationale (now in ADRs) and forward/backward-looking references (it is present-tense only). KPI: 16384lights | PC:666KB | tick:117/90/118/15/2/272/57/17/20/159/116/16/1/35us(FPS:8547/11111/8474/66666/500000/3676/17543/58823/50000/6289/8620/62500/1000000/28571) | tick:1392us(FPS:718) | heap:155KB | src:176(35000) | test:122(18043) | lizard:126w Note: the ESP32 KPI line (1392us/718) is the stale monitor.log baseline. Live captures on the HA-test boards this session: MM-S31 (eth) ~2641us/378 FPS, MM-P4 (eth) ~7033us/141 FPS, both healthy through the full discovery + command round-trip. Core: - MqttModule: escape the user-editable deviceName with the shared jsonEscape before it enters the discovery JSON, so a quote or backslash in the name can't produce an invalid retained config. - MqttModule: the discovery-off retract now sends the empty retained payload from a local buffer and frees unconditionally, and CONNACK retracts when discovery is off, so turning discovery off (even while disconnected or reconnecting) clears the broker's retained config and HA removes the entity. - MqttModule: failed HA subscribe / discovery-publish / availability-publish sends now resetConnection, matching the ping/state paths, so a partial write can't leave a truncated frame on the stream. - MqttModule: ha/set state acts only on exact "ON"/"OFF"; a malformed or truncated value is ignored, not silently treated as OFF. - Two forward-looking comments rephrased to present tense; the discovery buffer size is exposed as kDiscoveryDynamicBytes (static_assert-tied to the real sizes) so the test asserts a named constant. - DevicePlugin / DevicesModule / the LED drivers / platform files: retarget decisions.md references to the new lessons.md or the relevant ADR. Tests: - unit_MqttModule: the CONNACK discovery test now parses the packet stream and asserts the RETAIN bit on the config and availability publishes (a dropped retain fails it); a regression test pins the disconnected-retract fix (dynamicBytes drops to 0); ha/set-unknown-state coverage added. - test_mkdocs_slug: a guard that every catalog-page resolves on disk, closing the --strict blind spot that let broken card-image paths ship. Docs / CI: - docs/adr/: eleven Nygard-format ADRs (immutable, superseded-not-edited) capturing the architectural decisions (persistence, adaptive memory, buffer persistence, composable modifiers, set-control primitive, device discovery, MoonLive, board injection, generated docs, integration identity, pull-not-pub/sub) + an index README. - decisions.md split: renamed to lessons.md (debugging gotchas, pruned as absorbed); the durable rules extracted to coding-standards.md (numeric-width, defaults, a new Debugging-and-verification section) and architecture.md; the decisions promoted to ADRs. 861 -> 236 lines. - architecture.md: decision-rationale moved to ADRs and cross-linked; verbose what-prose tightened (graceful degradation, config provenance, live reconfiguration); "What we leave undesigned" moved to the backlog; all backlog/history references removed so the doc is present-tense only. 12406 -> 11430 words. - Catalog pages (services/effects/modifiers/layouts/drivers/supporting): fixed 34 broken card-image paths (../../../assets -> ../../assets); the wrong depth 404'd on the live site while passing --strict. - CLAUDE.md + history/README.md: doc-map updated for docs/adr/ and lessons.md; the carry-forward gate routes decisions to ADRs, gotchas to lessons.md, rules to CLAUDE.md/coding-standards.md. A plans/README.md added for consistency. Reviews: - 🐇 deviceName not escaped in discovery JSON (Major): fixed via jsonEscape; hardware-verified a quoted name keeps the config valid. - 🐇 discovery-off retract didn't reach the broker when disconnected (Major): fixed (local buffer + CONNACK retract-when-off); hardware-verified the retained config clears. - 🐇 failed HA sends didn't reset the connection (Major): fixed to match the ping/state paths. - 🐇 unknown HA state mapped to OFF (Minor): fixed to act only on exact ON/OFF. - 🐇 two forward-looking comments (Minor): rephrased present-tense. - 🐇 retain bit not actually verified in the test (Minor): the test now parses the packet and asserts the bit. - 🐇 magic 768 in the test (Trivial): exposed as kDiscoveryDynamicBytes with a static_assert. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 11 +- .../0001-persistence-pod-memcpy-not-json.md | 18 + ...002-adaptive-memory-degradation-cascade.md | 23 + ...03-layer-buffer-persists-frame-to-frame.md | 17 + ...omposable-modifiers-physical-to-logical.md | 17 + ...0005-set-control-primitive-on-scheduler.md | 16 + ...evice-discovery-udp-mdns-advertise-only.md | 17 + ...onlive-expressions-host-bound-functions.md | 17 + ...8-board-injection-name-only-http-fanout.md | 16 + ...9-docs-generated-technical-plus-summary.md | 17 + ...integration-identity-stable-hardware-id.md | 16 + ...change-pull-and-prepare-pass-not-pubsub.md | 24 + docs/adr/README.md | 23 + docs/architecture.md | 56 +- docs/backlog/backlog-core.md | 8 +- docs/backlog/backlog-light.md | 8 +- docs/backlog/livescripts-analysis-top-down.md | 2 +- docs/coding-standards.md | 34 +- docs/history/README.md | 2 +- docs/history/decisions.md | 861 ------------------ docs/history/lessons.md | 236 +++++ ...0260706 - Home Assistant MQTT Discovery.md | 2 +- docs/history/plans/README.md | 18 + docs/moonmodules/core/services.md | 20 +- docs/moonmodules/light/drivers.md | 8 +- docs/moonmodules/light/effects.md | 26 +- docs/moonmodules/light/layouts.md | 4 +- docs/moonmodules/light/modifiers.md | 4 +- docs/moonmodules/light/supporting.md | 8 +- docs/usecases/led-signal-integrity.md | 2 +- moondeck/diag/preview_health.py | 2 +- src/core/DevicePlugin.h | 2 +- src/core/DevicesModule.h | 2 +- src/core/MqttModule.cpp | 48 +- src/core/MqttModule.h | 12 +- src/light/drivers/HueDriver.h | 2 +- src/light/drivers/LcdLedDriver.h | 2 +- src/light/drivers/ParlioLedDriver.h | 2 +- src/light/drivers/RmtLedDriver.h | 2 +- src/platform/esp32/platform_esp32.cpp | 2 +- src/platform/esp32/platform_esp32_lcd.cpp | 2 +- test/python/test_mkdocs_slug.py | 38 + test/unit/core/unit_MqttModule.cpp | 43 +- 43 files changed, 717 insertions(+), 973 deletions(-) create mode 100644 docs/adr/0001-persistence-pod-memcpy-not-json.md create mode 100644 docs/adr/0002-adaptive-memory-degradation-cascade.md create mode 100644 docs/adr/0003-layer-buffer-persists-frame-to-frame.md create mode 100644 docs/adr/0004-composable-modifiers-physical-to-logical.md create mode 100644 docs/adr/0005-set-control-primitive-on-scheduler.md create mode 100644 docs/adr/0006-device-discovery-udp-mdns-advertise-only.md create mode 100644 docs/adr/0007-moonlive-expressions-host-bound-functions.md create mode 100644 docs/adr/0008-board-injection-name-only-http-fanout.md create mode 100644 docs/adr/0009-docs-generated-technical-plus-summary.md create mode 100644 docs/adr/0010-integration-identity-stable-hardware-id.md create mode 100644 docs/adr/0011-data-exchange-pull-and-prepare-pass-not-pubsub.md create mode 100644 docs/adr/README.md delete mode 100644 docs/history/decisions.md create mode 100644 docs/history/lessons.md create mode 100644 docs/history/plans/README.md diff --git a/CLAUDE.md b/CLAUDE.md index 9f0c65a8..acf2799a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,7 +22,7 @@ See `docs/architecture.md` for system design. This file contains only rules and - **Robust to any input.** A running device tolerates any sequence of UI actions or API calls: add, delete, replace, or reconfigure any module in any order, at any grid size, and it keeps running. Degraded or idle is acceptable; crashed is not. This robustness is a defining strongpoint of projectMM, and it's guarded by the test framework, not by hope: a discovered crash drives a new test that pins the fix (see the Hard Rule). Out of scope: power loss, malformed OTA, brown-out, and other physical/electrical faults the firmware can't intercept; this principle is about what the software accepts as input. - **No reboot to apply a configuration change.** Every setting takes effect live, on the next render tick — change a pin map, a strand length, an output protocol, a mic pin or rate, anything, on a running device and it just works. There is no init-once-at-boot step, and no *config* change requires a restart, which sets projectMM apart from most LED-controller firmware (where a pin or protocol change means a reboot). Like robustness, this is a defining strongpoint, and it falls out of the architecture for free rather than being hand-built per module: any control whose change reshapes derived state routes through the generic `onBuildState()` rebuild sweep, so drivers, the audio peripheral, effects, layouts, modifiers and network I/O all inherit it. When adding a feature, don't reach for a reboot/restart to apply config; make the change live. Full mechanism + rationale: [architecture.md § Live reconfiguration](docs/architecture.md#live-reconfiguration-every-change-applies-without-a-reboot). The one exception is what you'd expect: a *firmware* OTA flash swaps the binary and needs the usual power cycle — that's not a configuration change, and (like power loss and brown-out) it's the same physical-fault boundary the robustness principle draws. - **Domain-neutral core.** Separate core infrastructure from the light domain as much as practical. When mixing is necessary, use domain-neutral naming so the code stays open to future separation. -- **Present tense only.** Code, comments, and documentation describe the system as it is now. No changelogs, no roadmaps. History lives in git commits. This bans not just future-tense ("will be", "planned") but **absence-narration**: phrases like "no longer", "anymore", "formerly", "used to", "X was removed", or "there's no longer a Y" describe a *change from a past state* a present-tense reader never saw — state what *is*, not what stopped being. (The test: "there is no MCLK pin" is a present-tense *property* — keep it; "there's no SET_DEVICE_MODEL RPC anymore" narrates a removal — cut it, just describe the path that exists.) Exceptions: `docs/backlog/` (forward-looking) and `docs/history/` (backward-looking) — and `decisions.md` lessons, which legitimately contrast before/after because the contrast *is* the lesson. +- **Present tense only.** Code, comments, and documentation describe the system as it is now. No changelogs, no roadmaps. History lives in git commits. This bans not just future-tense ("will be", "planned") but **absence-narration**: phrases like "no longer", "anymore", "formerly", "used to", "X was removed", or "there's no longer a Y" describe a *change from a past state* a present-tense reader never saw — state what *is*, not what stopped being. (The test: "there is no MCLK pin" is a present-tense *property* — keep it; "there's no SET_DEVICE_MODEL RPC anymore" narrates a removal — cut it, just describe the path that exists.) Exceptions: `docs/backlog/` (forward-looking) and `docs/history/` (backward-looking) — and `lessons.md` lessons, which legitimately contrast before/after because the contrast *is* the lesson. ## Hard Rules @@ -56,7 +56,7 @@ The design rationale for each rule below lives in [docs/architecture.md](docs/ar Then check the recommendation against [§ Principles](#principles) (minimalism, data over objects, concrete first) and propose it as a question, not a fait accompli. The product owner picks; the agent implements only what was picked. If the picked option turns out to need a follow-up change (e.g. an updated naming convention to make the new layout consistent), surface that *before* starting the move so it's a single coherent refactor, not three round-trips. -**Plan before implementing.** Use `/plan` mode before every feature. Review plans for: unnecessary files, inheritance where structs suffice, modifications outside the relevant directory. Reject and regenerate bad plans. **Save every approved plan** to `docs/history/plans/` named `Plan-YYYYMMDD - .md` (ISO-8601 date order so the directory sorts chronologically, e.g. `Plan-20260620 - Improv-as-REST.md` for 2026-06-20), as the first implementation step. The plan is the design record that complements `decisions.md` (the lesson record): the plan says what we set out to build and why; decisions.md captures what we learned doing it. **These saved plans are a reference archive for the product owner — agents WRITE a plan when creating one, but do NOT read the existing plan files for context unless the product owner explicitly points to one** (they're under the "Never automatically" rule below alongside the rest of `docs/history/`). Plans are **kept, not pruned** — they are the permanent design-intent record. When a plan's design ships (or doesn't), mark its outcome in the filename with a trailing parenthetical — `… (shipped).md` once it lands, `… (attempted, abandoned).md` if it was tried and dropped — so the directory shows at a glance what's done; an unmarked plan is still in flight. The **one exception** to "kept, not pruned": a multi-phase effort's per-phase plans may be **consolidated into a single `… (shipped).md` record** once the *whole* effort lands, provided that record preserves the design-intent arc (what each phase set out to do + the outcome, including any dead-ends). This isn't losing design intent — it's the same subtraction the rest of the process rewards, applied to a set of plans whose story is now one story. Consolidate only *shipped/settled* phases, never in-flight ones. +**Plan before implementing.** Use `/plan` mode before every feature. Review plans for: unnecessary files, inheritance where structs suffice, modifications outside the relevant directory. Reject and regenerate bad plans. **Save every approved plan** to `docs/history/plans/` named `Plan-YYYYMMDD - <title>.md` (ISO-8601 date order so the directory sorts chronologically, e.g. `Plan-20260620 - Improv-as-REST.md` for 2026-06-20), as the first implementation step. The plan is the design record that complements `lessons.md` (the lesson record): the plan says what we set out to build and why; lessons.md captures what we learned doing it. **These saved plans are a reference archive for the product owner — agents WRITE a plan when creating one, but do NOT read the existing plan files for context unless the product owner explicitly points to one** (they're under the "Never automatically" rule below alongside the rest of `docs/history/`). Plans are **kept, not pruned** — they are the permanent design-intent record. When a plan's design ships (or doesn't), mark its outcome in the filename with a trailing parenthetical — `… (shipped).md` once it lands, `… (attempted, abandoned).md` if it was tried and dropped — so the directory shows at a glance what's done; an unmarked plan is still in flight. The **one exception** to "kept, not pruned": a multi-phase effort's per-phase plans may be **consolidated into a single `… (shipped).md` record** once the *whole* effort lands, provided that record preserves the design-intent arc (what each phase set out to do + the outcome, including any dead-ends). This isn't losing design intent — it's the same subtraction the rest of the process rewards, applied to a set of plans whose story is now one story. Consolidate only *shipped/settled* phases, never in-flight ones. **Use `uv` for every Python invocation.** Never type `python` or `python3` directly; always go through `uv run` (e.g. `uv run moondeck/build/build_desktop.py`, `uv run python -c "…"`). This applies to shell commands, CMake `add_custom_command` / `execute_process`, documentation examples, and anything that shells out. In CMake, resolve `find_program(UV_EXECUTABLE NAMES uv REQUIRED HINTS "$ENV{USERPROFILE}/.local/bin" "$ENV{HOME}/.local/bin")` once and use `${UV_EXECUTABLE} run python …` thereafter. Reason: uv manages the project venv and is the project standard ([moondeck/MoonDeck.md](moondeck/MoonDeck.md)); bare `python3` isn't on PATH on Windows (and macOS Python Launcher pops a Store prompt). If you catch yourself about to type `python`, stop and prefix with `uv run`. @@ -153,7 +153,7 @@ The "this is now trunk" moment. Where the wider hygiene checks live, because onc 1. All commit gates passed on every commit in the PR. 2. PR feedback addressed (CodeRabbit + human review). -3. **Carry forward lessons**: if the branch produced a hard-won lesson, a proven pattern, or a non-obvious decision worth keeping, note it in `docs/history/decisions.md` as part of the branch's commits, so the lesson lands in `main` with the code that proved it. Do this on the branch before the merge commit. +3. **Carry forward lessons**: if the branch produced something worth keeping, record it in the right home as part of the branch's commits, so it lands in `main` with the code that proved it (do this on the branch before the merge commit). A **debugging lesson or gotcha** (a bug, its cause, the fix) goes in `docs/history/lessons.md`. A genuine **architectural decision** (chose approach A over B/C) is a new immutable ADR in `docs/adr/` (`NNNN-<title>.md`, Nygard format; supersede, never edit). A lesson that has hardened into a **general rule** goes to CLAUDE.md or `coding-standards.md`, not a history file. 4. **Documentation sync**: every new module / control / API endpoint has matching docs (`docs/moonmodules/*.md`, `docs/testing.md`, `docs/architecture*.md`). 5. **Reviewer agent**: trigger this **first** so it runs while the other checks (docs sync, carry-forward lessons, conditional gates) proceed in parallel. Opus reviewer over the **whole branch diff** (`git diff main...HEAD`). Scope: domain boundary, **common patterns first** (flag any new convention (naming scheme, file shape, build flag, control mechanism, UI affordance) that isn't recognisable from a widely-used project / framework / canonical resource; bespoke choices must carry a stated reason at the introduction site, see the principle in § Principles), **unnecessary abstractions** (no-op / pass-through wrappers that only rename or re-namespace an existing function, single-call-site indirection that would read clearer inlined, names that obscure where the real code lives), **duplicated patterns** (same logic in multiple places that belongs in a base class or shared function), hot-path violations, spec conformance, bloat, platform boundary. Architectural drift is more visible across N commits than across one: "three commits each added a wrapper" reads as a pattern that one commit hides. Findings either get fixed in additional branch commits before merge, or are accepted with a one-line reason in the PR description. CodeRabbit complements this: CodeRabbit handles line-level bugs in the PR; the Reviewer agent handles architectural drift. 6. **PR title and description**: review and update if the work done differs from what the PR title/description says. The description is the permanent record of what landed and why; it should reflect the actual diff, not the original intent. @@ -201,9 +201,12 @@ docs/ backlog-core.md ← to-build list, core / infrastructure domain (+ UI) backlog-light.md ← to-build list, light domain (drivers, effects, preview, sensors) backlog-mixed.md ← to-build list, items spanning both domains + adr/ ← architecture decision records (Nygard format; immutable, superseded-not-edited) + README.md ← ADR index + NNNN-<title>.md ← one decision each (Status / Context / Decision / Consequences) history/ ← backward-looking: accumulated wisdom README.md ← index: what's here + cross-repo trends + digest prompt - decisions.md ← actions, lessons, proven patterns + lessons.md ← debugging lessons + gotchas (bug → cause → fix; pruned as absorbed) plans/ ← approved feature plans (Plan-YYYYMMDD - <title>.md; PO reference, agents don't auto-read) *-inventory.md ← prior-project surveys (v1, v2, moonlight) <repo>.md ← friend-repo monthly activity digests (FastLED, WLED, …) diff --git a/docs/adr/0001-persistence-pod-memcpy-not-json.md b/docs/adr/0001-persistence-pod-memcpy-not-json.md new file mode 100644 index 00000000..88102e59 --- /dev/null +++ b/docs/adr/0001-persistence-pod-memcpy-not-json.md @@ -0,0 +1,18 @@ +# 1. Persist POD module state with memcpy, not JSON + +Status: Accepted + +## Context + +Plan-09 attempted ~1700 LOC of JSON-based persistence for module state. It was fully abandoned. The premise "persistence is JSON" was assumed without justification: neither human-readability nor manual editability were real requirements. The JSON design spawned ~15 helpers (`rebuildControls`, `LoadAllFn`, `applyNode`, `serializeNode`, `cleanupTmpLeafCb_`, and more), which was the system signalling the design was too elaborate for the job. It forced a Scheduler reorder (3→5 phases) that bred secondary bugs (a duplicate-children bug, a MAC→deviceName guard, multiple "device shows nothing" failures), and needed five defensive null guards that masked an allocate-new-before-free fragmentation invariant. + +## Decision + +Persist POD module state with a single `memcpy(file, this + sizeof(MoonModule), classSize - sizeof(MoonModule))`, loading it back before any module's `setup()` / `onBuildControls()` by memcpy into member memory directly. Plan-10 took this path and shipped. + +## Consequences + +- Save and restore are one line each; no serializer/deserializer helper sprawl. +- Loading before `setup()` means no Scheduler phase reorder and none of its secondary bugs. +- POD-only: state is a flat memory image, so there is no schema-versioned migration across a struct-layout change (a future need would be its own decision, not a reason to pay for JSON now). +- The lesson that generalised: question a format premise before building to it; suspicious helper proliferation is a design smell; fix an invariant, do not paper it with per-call-site guards. diff --git a/docs/adr/0002-adaptive-memory-degradation-cascade.md b/docs/adr/0002-adaptive-memory-degradation-cascade.md new file mode 100644 index 00000000..0d7c8077 --- /dev/null +++ b/docs/adr/0002-adaptive-memory-degradation-cascade.md @@ -0,0 +1,23 @@ +# 2. Adaptive allocation with a degradation cascade + +Status: Accepted + +## Context + +The light pipeline runs on devices from a no-PSRAM ESP32 (~180 KB free internal heap) to a PSRAM-rich P4. A fixed buffer scheme either wastes memory on small installs or fails to fit large ones. The pipeline has intermediate buffers (mapping LUT, driver output buffer) that a 1:1 unshuffled layout does not need at all. + +## Decision + +Allocate intermediate buffers on demand, only when the pipeline actually needs them, and degrade under memory pressure rather than fail: + +- The **mapping LUT** is built only when modifiers exist and the layout is not a plain grid and heap remains after reserving `HEAP_RESERVE` (32 KB) for stack/HTTP/WiFi. +- The **driver output buffer** is built only when a LUT is actually allocated. +- When memory is insufficient, degrade in order: full pipeline → skip the output buffer (map inline) → skip the LUT (forced 1:1) → reduce layer dimensions (halve to a floor of 8×8). + +Each level is observable (`degraded()`, `lutSkipped()`, `outputBufferSkipped()`). Every allocation is predict-then-measure: predict from dimensions + channels + modifiers, compare the heap delta, and flag >5% variance as a leak. + +## Consequences + +- A 1:1 unshuffled layout allocates zero intermediate buffers; ArtNet reads the layer buffer directly. Maximum LED count at minimum memory. +- The device stays running under pressure (degraded is acceptable, crashed is not), instead of failing an allocation outright. +- The mechanism and buffer-type detail live in [architecture.md § Memory strategy](../architecture.md#memory-strategy); this ADR records the decision to make allocation adaptive rather than fixed. diff --git a/docs/adr/0003-layer-buffer-persists-frame-to-frame.md b/docs/adr/0003-layer-buffer-persists-frame-to-frame.md new file mode 100644 index 00000000..b183698c --- /dev/null +++ b/docs/adr/0003-layer-buffer-persists-frame-to-frame.md @@ -0,0 +1,17 @@ +# 3. The layer buffer persists frame-to-frame + +Status: Accepted + +## Context + +An early design cleared the render buffer before every effect frame, on the reasoning "the buffer is the effect's to fill, every time." This silently broke every persistence effect: a scroll reads the prior column and shifts it (reading a wiped buffer, only the fresh pixel survived); a trail calls `fadeToBlackBy` to decay the previous frame (fading zeroes never forms a trail); Game-of-Life reads its prior cell state (gone). The symptom that surfaced it: FreqMatrix lit only one row, and ~13 effects' `fade` controls did nothing. + +## Decision + +The render buffer is **not** cleared each frame. `Layer::loop()` leaves the previous frame's pixels in place, zeroed once on allocation/resize, then persistent, matching the FastLED / WLED / MoonLight convention (their `leds[]` / segment / VirtualLayer buffers all persist). Each effect owns its background inside its own `loop()`: a full-grid effect overwrites every pixel, a trail calls `fadeToBlackBy`, a sparse effect that wants a clean frame calls `draw::fill` itself. There is no per-effect "persist" flag (a flag would be bespoke). `fadeToBlackBy` is a Layer operation collected once per frame: effects register an amount, the Layer keeps the MIN across them and applies one pass at the next frame's start. + +## Consequences + +- Persistence effects (scroll, trail, Game-of-Life) work; multiple effects on one layer deliberately interact through the shared persistent buffer. +- N fading effects cost one buffer pass, not N, and never fade each other's fresh pixels. +- A state-advancing effect (Game-of-Life) separates when the simulation steps (gated on `bpm`) from when it paints (frame rate); its state lives in the persistent buffer. `unit_Layer_persistence` and `unit_GameOfLifeEffect` pin it. diff --git a/docs/adr/0004-composable-modifiers-physical-to-logical.md b/docs/adr/0004-composable-modifiers-physical-to-logical.md new file mode 100644 index 00000000..65d098ff --- /dev/null +++ b/docs/adr/0004-composable-modifiers-physical-to-logical.md @@ -0,0 +1,17 @@ +# 4. Build the map physical→logical so modifiers compose + +Status: Accepted + +## Context + +Modifiers needed to chain (Region, then Multiply, then Rotate). The old interface `mapToPhysical(logicalCoord) → [physical indices]` (a logical→physical fan-out) did not compose: stages emitted flat indices, not coordinates, and chaining would need a product-of-`maxMultiplier` fan-out ceiling, the exact `uint16` overflow that black-screened the high-fan-out MultiplyModifier. + +## Decision + +Invert the map build to **physical→logical**, adopting MoonLight's proven model written in projectMM's own code. Each modifier is an in-place coordinate fold with three hooks: `modifyLogicalSize`, `modifyLogical` (returns false to reject a coordinate), and `modifyLive` (for per-frame transforms like Rotate). The Layer walks the *physical* lights and folds each through the chain. Because a scatter onto arbitrary logical keys does not fit `setMapping`'s in-order contract, the build is a textbook counting-sort CSR construction (count, prefix-sum, scatter, replay) entirely on the cold path. + +## Consequences + +- Fan-out becomes free: N physical lights folding onto one logical cell *is* the fan-out, with no fan-out list, no product ceiling, no overflow. `destinationCount ≤ driverCount` is now a hard invariant; the `maxMultiplier`/scratch-buffer machinery was deleted. +- The per-frame read is byte-identical; the hot path is untouched. +- Static folds (mask/tile/crop) happen at build time; only rotation gathers per frame, so a static-only chain pays nothing per frame. diff --git a/docs/adr/0005-set-control-primitive-on-scheduler.md b/docs/adr/0005-set-control-primitive-on-scheduler.md new file mode 100644 index 00000000..923119a5 --- /dev/null +++ b/docs/adr/0005-set-control-primitive-on-scheduler.md @@ -0,0 +1,16 @@ +# 5. A generic set-control primitive on the Scheduler + +Status: Accepted + +## Context + +Adding IR remote control raised: how does one module drive another module's control? The wrong answer is a bespoke seam per target (`Drivers::adjustBrightness`, then `setPalette`, and N more one-offs). The right primitive already existed half-hidden: the WLED-app bridge set brightness via `applySetControl(module, control, value)`, the generic find → validate → apply → `onUpdate` → persist → conditional-`buildState` path, but it was private on HttpServerModule. + +## Decision + +Lift `setControl` onto the `Scheduler` (which owns the module tree and the persistence hook), and make HttpServer's `applySetControl` a thin result→HTTP-status mapper. Every input (IR, HTTP, Improv, the WLED bridge) composes against one control-agnostic primitive. A factory-created module reaches it through `Scheduler::instance()` (the `FilesystemModule::instance_` pattern), so no per-module inject is needed. + +## Consequences + +- Adding a new input transport never adds control plumbing; HttpServer shrank ~54 lines. +- The rule that generalised: when a second caller wants a capability a transport implemented privately, the capability belongs in core (the tree owner), and the transport keeps only its status-mapping. diff --git a/docs/adr/0006-device-discovery-udp-mdns-advertise-only.md b/docs/adr/0006-device-discovery-udp-mdns-advertise-only.md new file mode 100644 index 00000000..dfd5809a --- /dev/null +++ b/docs/adr/0006-device-discovery-udp-mdns-advertise-only.md @@ -0,0 +1,17 @@ +# 6. UDP presence for discovery, mDNS advertise-only + +Status: Accepted + +## Context + +The DevicesModule refactor reached for a UDP beacon, swung to "mDNS is the standard, use it for discovery too," then, after bench measurement, landed elsewhere. A blocking mDNS PTR query for a service the device *also advertises* exhausts the IDF mDNS pool (`Cannot allocate memory` with megabytes free) and the device's own advertisement vanishes from peers. The "use the standard" instinct was right about announce-to-foreign-apps and wrong about discover-peers; only measuring the wire separated them. + +## Decision + +Use **UDP presence for discovery** (where we control both ends, or a foreign system already broadcasts, e.g. WLED's 44-byte packet on UDP 65506), and **mDNS advertise-only** (only where a foreign app requires it, e.g. the native WLED app finds us solely via mDNS `_wled._tcp`). The `DevicePlugin` seam is transport-agnostic (`discoveryPort()` / `classifyPacket(...)`), not mDNS-shaped. + +## Consequences + +- The mDNS-pool exhaustion is structurally impossible: discovery no longer queries, mDNS shrinks to advertise-only. +- Two directions of one interop can use different transports: we discover WLED over UDP 65506; WLED's app discovers us over mDNS. +- The lesson that generalised: read the prior art's *actual* behaviour rather than assuming it; a self-inflicted bug misattributed to "the network" or "the chip" wastes hours (the WLED app discovering WLED on classic ESP32s proves classic mDNS propagates, so an invisible projectMM device is our bug). diff --git a/docs/adr/0007-moonlive-expressions-host-bound-functions.md b/docs/adr/0007-moonlive-expressions-host-bound-functions.md new file mode 100644 index 00000000..cad42619 --- /dev/null +++ b/docs/adr/0007-moonlive-expressions-host-bound-functions.md @@ -0,0 +1,17 @@ +# 7. MoonLive is expressions + host-bound functions, not statement shapes + +Status: Accepted + +## Context + +The MoonLive compiler was first built around a fixed *statement shape*, `setRGB(idx, r, g, b)`, with per-slot parser rules (the index could be `random16`, colours were literal-only) and an RGB-specific `Store` op baked into the core. Three product-owner remarks exposed one root flaw: `random16` worked only in the index slot; `random16(255)` capped at a byte (validators conflated ranges); and the core compiler was light-domain-specific. + +## Decision + +Adopt the ESPLiveScript / ARTI model: the core knows only **expressions plus a generic call mechanism**; the host registers its functions in a builtin table. Every argument parses as an expression (a literal or a nested call), so `setRGB(random16(256), random16(256), 30, 0)` works and a number is a `uint16`. The LED names and RGB meaning live only in the light-domain registration (`MoonLiveBuiltins_light.h`); the core sees a neutral `BuiltinTable` of `{name → Call(fn ptr) | Inline(opcode tag)}`, where a buffer writer is `Inline` (the hot-path fast path) and a pure helper is `Call`. + +## Consequences + +- A capability the language "can't express in slot Y" is fixed by real expressions, not a per-slot special case. +- Domain-neutrality is testable: a test asserts the core with an empty builtin table knows no functions. +- Two codegen contracts fell out and generalise to any per-ISA backend: a value live across a call must survive it (save/restore the caller-saved set), and register budget is real on the MCU, so a tree-walk register stack with a free-list allocator keeps N calls at a handful of registers, not 2N (this surfaced on the P4 RISC-V backend, the first target where a 4-call statement exhausted the pool). diff --git a/docs/adr/0008-board-injection-name-only-http-fanout.md b/docs/adr/0008-board-injection-name-only-http-fanout.md new file mode 100644 index 00000000..235fb3c4 --- /dev/null +++ b/docs/adr/0008-board-injection-name-only-http-fanout.md @@ -0,0 +1,16 @@ +# 8. Board injection: SET_BOARD name-only, controls over HTTP + +Status: Accepted + +## Context + +The web installer's catalog ships per-board control values the orchestrator pushes to a fresh device during provisioning. Every per-board control shipped today applies post-association (`Network.txPowerSetting`, Ethernet pin maps), so the ~1 s window between WiFi association and HTTP fan-out running at a wrong setting is acceptable. It would not be acceptable for a pre-association control: country code (governs which channels are scanned), antenna selector (wrong RF path at init makes the device deaf), or pre-association TX-power (some chips need the cap before the first probe). + +## Decision + +`SET_BOARD` over Improv-Serial carries only the board name (one vendor RPC, one Text control); every other field ships via HTTP after WiFi association. If a pre-association control is ever added, **do not extend SET_BOARD's wire format** (that couples unrelated controls to the board-name lifecycle and hides the timing constraint). Use one of two explicit escape hatches instead: a second vendor Improv RPC (`SET_<control>`) dispatched before `SEND_WIFI_CREDENTIALS` for user-configurable pre-association controls, or a board-specific sdkconfig fragment baking the value into firmware for truly board-static values. + +## Consequences + +- The board-name lifecycle stays decoupled from control values; the timing contract is explicit rather than buried in a growing wire format. +- The implicit option "just push it earlier in SET_BOARD" is ruled out, it tangles the lifecycle. diff --git a/docs/adr/0009-docs-generated-technical-plus-summary.md b/docs/adr/0009-docs-generated-technical-plus-summary.md new file mode 100644 index 00000000..bf1a526d --- /dev/null +++ b/docs/adr/0009-docs-generated-technical-plus-summary.md @@ -0,0 +1,17 @@ +# 9. Two doc surfaces: generated technical + hand-written summary + +Status: Accepted + +## Context + +Per-module `.md` files were being hand-maintained and hand-shrunk, drifting from the code they described. The technical facts (controls, ranges, members) already lived in the `.h`, so keeping a parallel prose copy was duplication that rotted. + +## Decision + +Every module has exactly two reader surfaces. The `.h` is the single home of technical content (`///` comments); a Doxygen→moxygen pipeline **generates** one technical page per module. A thin hand-written summary page (a table, end-user facing) is the only prose that stays. The prior art is docs.rs / Sphinx-autodoc / Doxygen (a hand-written guide over a generated API reference). The full model lives in [coding-standards.md § Documentation model](../coding-standards.md#documentation-model). + +## Consequences + +- A technical fact is stated once (in the `.h`) and never re-typed in prose. +- The pipeline exposed traps that became build rules, each worth keeping: batch the external tool (one Doxygen pass + one `moxygen`, not 132 per-header invocations, ~150 s → ~7 s); write generated files only on content change (an unconditional write into a watched dir is an infinite rebuild loop); a present-but-failing generator must fail loud, not return empty; and a doc path duplicated in `main.cpp` `registerType` drifts silently, so `check_specs.py` gates that every docPath resolves to a real page + anchor. +- The verification rule that generalised: a generated artifact's ground truth is the rendered output, so verify an anchor against the built HTML (`grep id=` the `.html`), never a re-derived slug. diff --git a/docs/adr/0010-integration-identity-stable-hardware-id.md b/docs/adr/0010-integration-identity-stable-hardware-id.md new file mode 100644 index 00000000..ce7b4d22 --- /dev/null +++ b/docs/adr/0010-integration-identity-stable-hardware-id.md @@ -0,0 +1,16 @@ +# 10. Integration identity is a stable hardware id, never the editable name + +Status: Accepted + +## Context + +The MQTT module first derived its topic prefix from the user-editable `deviceName` (`projectMM/<deviceName>`). On the bench a rename (ShellyOne → ShellyTwo) instantly repointed every topic and orphaned the Homebridge config: the hub kept publishing to the old topics while the device listened on new ones, showing "not responding." WLED, Tasmota, ESPHome, and HA MQTT discovery are unanimous, the machine-facing identity anchors to a stable hardware id (WLED's `wled/<last6-of-MAC>` is rename-stable; HA discovery *requires* a stable `unique_id` and forbids the device name / hostname as identity). + +## Decision + +Any control-plane identity an external system binds to (an MQTT topic prefix, an HA discovery `unique_id`, an API key path) derives from an immutable hardware id, never the editable `deviceName`. MQTT topics derive from `projectMM/<last6-of-MAC>`; the friendly `deviceName` rides a separate retained `<prefix>/name` topic as a published-but-non-identifying field. + +## Consequences + +- A device rename updates only the display label; every external integration stays bound to the stable id. +- The rule applies to any future integration, not just MQTT: derive the identity from something immutable, keep the human name a separate non-identifying field. Restated as an invariant in [architecture.md § Device name](../architecture.md#device-name-one-identity-every-network-name-derives-from-it). diff --git a/docs/adr/0011-data-exchange-pull-and-prepare-pass-not-pubsub.md b/docs/adr/0011-data-exchange-pull-and-prepare-pass-not-pubsub.md new file mode 100644 index 00000000..06af4641 --- /dev/null +++ b/docs/adr/0011-data-exchange-pull-and-prepare-pass-not-pubsub.md @@ -0,0 +1,24 @@ +# 11. Inter-module data and events: pull + prepare-pass, not pub/sub + +Status: Accepted + +## Context + +Modules need two things from each other: one module reads another's data on the hot path (an effect reading a sensor frame, a driver reading a layer buffer), and a change in one module triggers derived-state rebuilds in others (a control edit resizing buffers, a tree mutation re-resolving links). The obvious general-purpose answer is a publish/subscribe event bus with a registry and listener lifecycles. + +## Decision + +Do not use pub/sub. There is one producer per data kind and the consumer explicitly wants that specific data, so the registry overhead and listener-lifecycle complexity buy nothing. Use three narrower mechanisms instead: + +- **Shared-struct pull** for hot-path data: the producer owns a small POD struct overwritten in place each tick; the consumer holds a `const Foo*` (set at wiring time) and reads it per frame. Lock-free for a small POD (a half-updated read self-corrects next tick); a large frame buffer uses the two-core double-buffer swap instead, not this pull. +- **Push to a domain-neutral sink** when a producer hands bytes to a generic core service: the core defines a narrow interface (`BinaryBroadcaster`), the producer pushes, the sink knows nothing about the payload. +- **A framework-driven prepare-pass** for derived-state rebuilds: a three-tier split (`onUpdate` per-control, a `controlChangeTriggersBuildState` gate, `onBuildState` rebuild) where the coordinator walks every module, gated by per-module metadata. This is the recognised layout/prepare-pass pattern (JUCE `prepareToPlay`, UIKit `layoutSubviews`, WPF `AffectsMeasure`), not an event bus: the publisher tells the coordinator to run the pass, the coordinator walks every module. + +Direct method calls cover the one remaining case (a producer notifying one known consumer): the producer holds a pointer set at wiring time and calls it. + +## Consequences + +- No registry, no subscription, no listener lifecycles to leak; each mechanism costs only what its case needs. +- Every change costs proportionally: an in-place tweak is tier-1 only; a shape change ripples through the tree-wide sweep; a structural mutation always rebuilds. +- The mechanism is core and domain-neutral; the light pipeline consumes it (mapping-LUT rebuild, buffer pull) without the core knowing about lights. +- Pub/sub becomes the right pattern only if multiple unknown subscribers per event ever appear; projectMM has none. The mechanisms are described as current behaviour in [architecture.md § Data exchange](../architecture.md#data-exchange-between-modules) and [§ Event triggering](../architecture.md#event-triggering-between-modules). diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 00000000..d2ed139a --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,23 @@ +# Architecture Decision Records + +An [ADR](https://github.com/joelparkerhenderson/architecture-decision-record) captures one significant architectural decision: the context that forced a choice, the option taken, and the consequences that followed. Format is [Michael Nygard's classic](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions.html): **Title, Status, Context, Decision, Consequences**. + +These records are **immutable**. A decision that changes is not edited in place, a new ADR supersedes it and both link, so the reasoning trail stays honest. This is the difference from the [lessons log](../history/lessons.md): lessons are debugging war-stories, pruned as they are absorbed; ADRs are decisions, kept as an append-only record. The forward-looking counterpart, what we set out to build, is the [plan archive](../history/plans/README.md). + +Agents do not read this directory automatically, only when a decision's rationale is in question (the same rule as `history/` and `backlog/`). + +## Index + +| # | Decision | Status | +|---|----------|--------| +| [0001](0001-persistence-pod-memcpy-not-json.md) | Persist POD module state with memcpy, not JSON | Accepted | +| [0002](0002-adaptive-memory-degradation-cascade.md) | Adaptive allocation with a degradation cascade | Accepted | +| [0003](0003-layer-buffer-persists-frame-to-frame.md) | The layer buffer persists frame-to-frame | Accepted | +| [0004](0004-composable-modifiers-physical-to-logical.md) | Build the map physical→logical so modifiers compose | Accepted | +| [0005](0005-set-control-primitive-on-scheduler.md) | A generic set-control primitive on the Scheduler | Accepted | +| [0006](0006-device-discovery-udp-mdns-advertise-only.md) | UDP presence for discovery, mDNS advertise-only | Accepted | +| [0007](0007-moonlive-expressions-host-bound-functions.md) | MoonLive is expressions + host-bound functions | Accepted | +| [0008](0008-board-injection-name-only-http-fanout.md) | Board injection: SET_BOARD name-only, controls over HTTP | Accepted | +| [0009](0009-docs-generated-technical-plus-summary.md) | Two doc surfaces: generated technical + hand-written summary | Accepted | +| [0010](0010-integration-identity-stable-hardware-id.md) | Integration identity is a stable hardware id | Accepted | +| [0011](0011-data-exchange-pull-and-prepare-pass-not-pubsub.md) | Inter-module data/events: pull + prepare-pass, not pub/sub | Accepted | diff --git a/docs/architecture.md b/docs/architecture.md index 3276aec9..854d2c67 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -47,7 +47,6 @@ Coding conventions live in [coding-standards.md](coding-standards.md); how to bu - [Scaling to available memory](#scaling-to-available-memory) - [Multi-device sync](#multi-device-sync) - [Web UI](#web-ui) - - [What we leave undesigned](#what-we-leave-undesigned) ## The problem @@ -133,7 +132,7 @@ Control values and each module's `enabled` flag are persisted to flash so settin - **Conditional controls**: every conditional control is always bound; the module sets a `hidden` flag (`controls_.setHidden(i, …)`) to tell the UI not to render it. The load path can therefore find persisted values regardless of the live conditional state. - **Code-wired children survive a stale file**: some children aren't created by the user; `main.cpp`'s boot wiring attaches them (`ImprovProvisioningModule` under `NetworkModule`; `NetworkSendDriver`, `PreviewDriver` under their parents). Each such child calls `markWiredByCode()` after `addChild()`, a one-bit flag meaning *"I belong here because the code put me here, not because a saved file or a user asked for me."* The problem it solves: persistence reconciles the live tree to match the saved JSON, so a child that exists in code but is absent from an older saved file (written before that child was added) would be trimmed on load. The flag tells the apply step to keep it. Children added through the HTTP API or recreated from JSON stay unmarked; those follow the file's tree shape exactly, so UI deletes still take effect. -How persistence reaches the Scheduler without the Scheduler depending on it: the Scheduler exposes a **function-pointer hook** (`setLoadAllHook`) that the load phase calls if set. FilesystemModule registers its load routine through that hook at startup; the Scheduler never includes or names FilesystemModule (no circular dependency, and persistence is fully optional). With no FilesystemModule registered the hook is null, the load phase is a no-op, and the system runs with member-initialised defaults. +Persistence reaches the Scheduler through a **function-pointer hook** (`setLoadAllHook`) the load phase calls if set: FilesystemModule registers its load routine there at startup, so the Scheduler never names FilesystemModule (no circular dependency, persistence stays optional; a null hook means defaults-only). The choice of a flat POD image over a JSON format, and of load-before-setup, is [ADR-0001](adr/0001-persistence-pod-memcpy-not-json.md). ## Parallelism @@ -160,7 +159,7 @@ No registry, no subscription, no event bus. The consumer reads the latest value **Push through a domain-neutral sink.** When the producer should hand bytes to a generic core service rather than expose a struct, the core defines a narrow interface and the producer pushes to it. The producer owns the data and its wire format; the core sink (the interface's implementer) knows only "take these bytes and do my generic job"; it has zero knowledge of what the bytes mean or which domain produced them. `BinaryBroadcaster` (`HttpServerModule` implements it: "broadcast these bytes to all WebSocket clients") is the example; the producer side lives in the light domain (see [§ The pipeline](#the-pipeline)). -Both shapes extend without ceremony to any future producer/consumer pair (a sensor module owning a state struct, an effect reading it through a `const Foo*` set at wiring time; or a module pushing bytes to a core sink). Neither is pub/sub: there's one producer per data kind and the consumer explicitly wants that specific data: the registry overhead and listener-lifecycle complexity of pub/sub buy nothing. +Both shapes extend to any future producer/consumer pair (a sensor owning a state struct read through a `const Foo*`; a module pushing bytes to a core sink). Neither is pub/sub, and the reasons this project chose pull + a prepare-pass over an event bus are [ADR-0011](adr/0011-data-exchange-pull-and-prepare-pass-not-pubsub.md). ## Event triggering between modules @@ -172,11 +171,11 @@ A control changes, or the module tree is mutated (a child added, deleted, replac `Scheduler::buildState()` fires from two triggers: a tier-2 gate returning true after a control change, **and** any tree mutation (HTTP add/delete/replace/move handlers all call it unconditionally, since a structural change is rare and unambiguously needs a rebuild). Both triggers funnel through the same sweep; each module's `onBuildState` is idempotent (e.g. an effect only reallocs when its grid count actually changed), so over-rebuilding is wasted work, not a correctness hazard. -This is the recognised layout/prepare-pass pattern: JUCE's `prepareToPlay` and UIKit's `layoutSubviews` work the same way: a framework-driven sweep over every object of the primary type, gated by per-object metadata (WPF's `AffectsMeasure`, here `controlChangeTriggersBuildState`). Not pub/sub: the publisher (HttpServerModule, or the mutation site) explicitly tells the coordinator to run the pass; the coordinator explicitly walks every module. The light domain consumes this mechanism for its mapping rebuild (see [§ Mapping and blending](#mapping-and-blending)) but the mechanism itself is core and applies to any module with derived state. +This is the recognised layout/prepare-pass pattern (JUCE `prepareToPlay`, UIKit `layoutSubviews`, gated by per-object metadata like WPF's `AffectsMeasure`, here `controlChangeTriggersBuildState`); the pull-and-prepare-pass-not-pub/sub decision is [ADR-0011](adr/0011-data-exchange-pull-and-prepare-pass-not-pubsub.md). The light domain consumes it for the mapping rebuild ([§ Mapping and blending](#mapping-and-blending)); the mechanism itself is core. ### Live reconfiguration: every change applies without a reboot -A direct consequence of the three tiers above is a property worth naming, because it sets projectMM apart from most LED-controller firmware (where changing a pin map, strand length, or output protocol means editing a config and **rebooting**): **every MoonModule reconfigures itself live the instant a control changes — no *configuration* change requires a restart to take effect.** A pin edit, a leds-per-pin edit, a swap of output protocol, a mic pin or sample-rate change — each flows control-write → `onUpdate` (tier 1) and, when it changes the module's shape, → `Scheduler::buildState()` → `onBuildState()` (tier 3), which tears down and rebuilds exactly the derived state that changed: an LED driver re-targets its RMT channels / DMA bus onto the new GPIOs, an audio module re-inits its I²S channel on the new pins, an effect re-sizes its buffer, the Layer rebuilds its mapping LUT. The hot-path render loop reads the rebuilt state on its very next tick. This holds for **all** module types — drivers, the audio peripheral, effects, layouts, modifiers, network I/O — because the rebuild chain is core, not per-module. It composes with the [robustness rule](#robustness): because any change can be applied at any time in any order, a running device tolerates being reconfigured arbitrarily and keeps running (degraded or idle, never crashed). The one thing that still needs a power cycle is a *firmware* OTA flash — a binary swap, not a configuration change, and the same physical boundary the robustness rule draws (power loss, OTA, brown-out are out of scope there too). +A property that falls out of the three tiers, and sets projectMM apart from most LED-controller firmware (where changing a pin map, strand length, or protocol means a **reboot**): **every MoonModule reconfigures live the instant a control changes — no configuration change needs a restart.** A pin, leds-per-pin, protocol, or mic-rate edit flows control-write → `onUpdate` (tier 1) and, when it changes shape, → `onBuildState()` (tier 3), which rebuilds exactly the derived state that changed (an LED driver re-targets its RMT/DMA onto the new GPIOs, an audio module re-inits I²S, an effect re-sizes, the Layer rebuilds its LUT); the render loop reads it next tick. This holds for every module type because the rebuild chain is core, and it composes with the [robustness rule](#robustness): any change, any order, keeps the device running. Only a *firmware* OTA flash needs a power cycle, the same physical boundary the robustness rule draws. If a module needs to actively notify a specific other module of an event (rather than publish data for polling, or change its own controls), the pattern is a direct method call from the producer to a known consumer: `ImprovProvisioningModule::loop1s` calls `networkModule_->setWifiCredentials(...)` when credentials arrive over UART. No event bus; the producer holds a pointer to the consumer set at wiring time (`main.cpp`). Pub/sub becomes the right pattern only when there are multiple unknown subscribers per event; projectMM has none today. @@ -243,11 +242,9 @@ Firmware-vs-deviceModel is a **two-level** model for **where a pin or setting de So the Ethernet pins live at **both** levels, and that's not a contradiction: the firmware *seeds* a per-chip default, the deviceModel *fixes* the real map. The driver (which Ethernet stack) is firmware-only; the pin map is firmware-seeded but deviceModel-authoritative. -**The deviceModel is one level — there is no separate per-unit "device" provenance level.** Whether a control is PCB-fixed (the Ethernet PHY pins) or user-wired (a mic, LED strands) is not a taxonomy the code tracks; it falls out of **what the catalog entry lists**. A bare dev kit (`Generic ESP32 Dev`, `LOLIN D32`) lists few controls — the user wires the rest, so those stay unset; a finished product (`QuinLED Dig-2-Go`, an LED-controller shield) lists more, because its wiring is fixed. Same kind of catalog entry, different completeness — no `kind:` flag, no separate catalog. +**The deviceModel is one level — there is no separate per-unit provenance level.** Whether a control is PCB-fixed or user-wired is not a taxonomy the code tracks; it falls out of what the catalog entry lists. A bare dev kit lists few controls (the user wires the rest, so those stay unset); a finished product lists more (its wiring is fixed). Same kind of entry, different completeness, no `kind:` flag. -**The governing rule — "default only where the hardware actually fixes it"** — is the [`assign defaults only when they cannot do harm`](history/decisions.md) rule applied to pin provenance. A control whose value the product fixes (Ethernet PHY pins, status LED) *should* default (and *omitting* it harms — a deviceModel with un-defaulted Ethernet pins can never connect); a control the *user* wires (mic pins, LED-driver pins) must not default — any guess can drive a pin the user wired elsewhere — so it stays unset until set. This is enforced purely by **what each catalog entry lists**: an entry defaults a control by *including* it, and leaves a user-wired control unset by *omitting* it. No level-tagging needed — the data carries the rule. - -The rule covers **settings, not just pins**. `txPowerSetting` is the worked example: whether the assembled rig can *sustain* the current spike of full-power WiFi TX depends on the product's power regulation and how the user powers it (USB port, PSU, cable) — a brownout property of the physical product, not the chip. In-tree, the catalog sets `Network.txPowerSetting: 8` for the ESP32-S3 N16R8 Dev, which browns out at full power on typical USB; the entry *suggests* the safe floor, and a unit can lower it further for a weak supply. The general lesson: any setting whose safe value depends on the physical assembly or its power, not the chip, is set by the deviceModel's catalog entry. (The catalog is [`web-installer/deviceModels.json`](../web-installer/deviceModels.json); see the [installer README](../web-installer/README.md) for its schema.) +**The governing rule — "default only where the hardware actually fixes it"** — is the [Defaults rule](coding-standards.md#defaults) applied to pin provenance: an entry defaults a control by *including* it and leaves a user-wired control unset by *omitting* it, so the data carries the rule with no level-tagging. It covers **settings, not just pins**: `txPowerSetting` is set per-entry because whether a rig sustains full-power WiFi TX is a brownout property of the assembly and its power supply, not the chip (the catalog pins `Network.txPowerSetting: 8` for the ESP32-S3 N16R8 Dev, which browns out at full power on typical USB). The catalog is [`web-installer/deviceModels.json`](../web-installer/deviceModels.json) (schema in the [installer README](../web-installer/README.md)). ## Peripherals @@ -278,6 +275,8 @@ A device has **one** network name, `deviceName`, and every name the device prese - **Always a valid hostname.** Because all three uses are DNS/SSID names, `deviceName` must satisfy the RFC-1123 label rules (`[A-Za-z0-9-]`, no spaces, no leading/trailing hyphen). `SystemModule` enforces this at the source: it runs `mm::sanitizeHostname()` (in `core/Control.h`) on the value in `setup()` and every `loop1s()`, coercing whatever the user typed or persistence restored (`"My Living Room!"` → `"My-Living-Room"`) and falling back to the MAC-derived `MM-XXXX` if the result is empty. Sanitising *at the owner* means every consumer is correct for free — no per-consumer validation, no chance a raw name reaches mDNS. (`unit_sanitizeHostname` pins the rule.) - **Follows a live rename.** Renaming the device re-advertises immediately, no reboot — the [live-reconfiguration](#live-reconfiguration-every-change-applies-without-a-reboot) rule applied to identity. `NetworkModule::syncMdns()` (called each `loop1s()`) compares the current name to the last-registered one and re-registers mDNS when it changed, so `<new-name>.local` resolves within a tick. +**A machine-facing identity that an external system binds to is never the editable name.** An MQTT topic prefix, a Home Assistant discovery `unique_id`, an API key path: anything a foreign system keys off must derive from an immutable hardware id (MAC / chip-id, e.g. `projectMM/<last6-of-MAC>`), because a live `deviceName` rename would silently repoint every topic and orphan the peer's config. The human-readable name rides a *separate*, published-but-non-identifying field (WLED, Tasmota, ESPHome, and HA discovery all anchor identity this way). `deviceName` above is the network-presentation identity (mDNS / AP / DHCP, where the name *is* the address); an external-integration identity is the opposite case and stays decoupled from it. + # Light domain The light domain is everything specific to driving lights. **Light** here means any controllable light source: an addressable LED pixel (WS2812, APA102), a DMX fixture (RGB par, moving head, dimmer), or any other output that takes colour/intensity data. The term is used instead of "pixel" because the system controls both LEDs and conventional lighting fixtures. @@ -313,7 +312,7 @@ Modules in the light pipeline can be added, replaced, or removed dynamically at - *Shared-struct (pull):* `Drivers` hands every child driver a `Buffer*` (source) plus a `Correction*` (shared brightness/reorder/white), and `Layer` exposes its pixel buffer to `Drivers` directly on the identity-mapping fast path: each consumer holds a `const`-pointer and reads it per frame. The pointers are **(re)bound on every rebuild**, not just at boot: `Drivers::onBuildState()` re-resolves the active `Layer` (`Layers::activeLayer()`) and calls `passBufferToDrivers()`, which re-runs `setSourceBuffer()`/`setLayer()` on each child (clearing them to `nullptr` when there is no active Layer). So a held pointer is valid only until the next rebuild — which is exactly why the consumers re-read it each frame and tolerate a null (the [robustness rule](#robustness)): a Layer add/delete/replace re-binds or clears it live, no dangling reference. - *Push to a core sink:* `PreviewDriver` owns the preview wire format (a one-time coordinate table + per-frame RGB point list) and pushes the bytes to a `BinaryBroadcaster` (the core HTTP server). The server broadcasts them over WebSocket without knowing they're a preview: the format and the light types stay entirely in the driver. See [PreviewDriver](moonmodules/light/moxygen/PreviewDriver.md). -**Graceful degradation under transport backpressure.** The preview is the project's worked example of a property worth naming generically, because it is the transport-side sibling of the memory-side [§ Degradation cascade](#degradation-cascade): when a consumer can't keep up, **shed quality first** — degrade the stream rather than reach for a stall. The link to a browser is the slow consumer; a full-resolution frame (128² = 16384 lights = ~49 KB) may not drain in the budget one tick allows. Rather than block the loop until it drains, the producer **degrades**, shedding in the order video streaming does — frame rate first, then resolution: (1) the full-resolution frame streams from the driver buffer with no intermediate copy, drained a memory-adaptive chunk per transport tick (a **resumable** send), and the next frame starts only when the previous one finished — so the **effective frame rate self-limits** to what the link sustains, with no loop stall and no connection drop; (2) only when even one frame can't drain promptly does it shed **resolution** via a spatial-lattice downsample, the same congestion-responsive, adaptive-bitrate idea behind HLS/DASH/WebRTC applied to a binary WebSocket. The point budget is itself memory-derived (per [§ Scaling to available memory](#scaling-to-available-memory)), so a tighter board downsamples sooner. The render loop is charged only a bounded slice per tick; each delivered frame is a faithful **complete** frame at a lower rate or coarser sample — a WebSocket message is atomic to the browser, so a frame is whole or absent, never partial/torn. The coordinate table and the downsampled frames take a bounded synchronous send (begin/push/end): a client whose socket stays blocked past the spin budget is closed and reconnects (the browser re-handshakes and the next coordinate table re-primes it) — the bound caps tick occupancy, and a reconnect is a brief blip rather than a frozen preview. This is *graceful degradation*: a fast link sees every light at full rate, a slow link sees a faithful coarser sample at a few fps, and a wedged client is dropped and recovers rather than stalling the device. The mechanism (resumable cross-tick send + drop-new backpressure — a frame offered while one is in flight is dropped, the in-flight one is kept — plus adaptive frame rate + adaptive lattice) lives in `PreviewDriver` + `HttpServerModule`; it is payload-agnostic, so other bulky streams can ride the same transport. +**Graceful degradation under transport backpressure.** The preview is the transport-side sibling of the memory-side [§ Degradation cascade](#degradation-cascade): when the browser can't keep up with a full-resolution frame (128² = ~49 KB), the producer sheds quality rather than stall the loop, in video-streaming order, frame rate then resolution. The frame streams from the driver buffer with no intermediate copy, a resumable memory-adaptive chunk per tick, and the next frame starts only once the previous drained, so the effective frame rate self-limits to what the link sustains. Only when a single frame can't drain promptly does it downsample via a spatial lattice (the adaptive-bitrate idea behind HLS/DASH, on a binary WebSocket). Each delivered frame is whole (a WebSocket message is atomic), the render loop is charged a bounded slice per tick, and a client blocked past the spin budget is closed and reconnects (a blip, not a freeze). The mechanism is payload-agnostic and lives in [PreviewDriver](moonmodules/light/moxygen/PreviewDriver.md) + `HttpServerModule`, so other bulky streams can ride the same transport. **Naming convention.** Capital `Layouts`, `Layers`, `Drivers` are class names (always capitalised when referring to the class). Lowercase "layouts", "layers", "drivers" is the English plural, used freely when context makes it clear. Singular "layout", "layer", "driver" is an individual instance. @@ -338,7 +337,7 @@ Layouts cover both addressable LEDs and DMX fixtures. An LED-strip layout yields Positions are computed algorithmically, not stored. Grid is the most commonly used layout, but any geometry works: spheres, rings, cones, spirals, arbitrary point clouds. Grid is full-density (every position maps to a light); a wheel is sparse (only spoke positions are mapped, gaps are unmapped). -Multiple layouts can live in one Layouts container. Each layout describes one light type; mixing light types in a single Layouts (e.g. LED strips + par lights) is listed in [§ What we leave undesigned](#what-we-leave-undesigned). +Multiple layouts can live in one Layouts container. Each layout describes one light type: the model is one light type per layout (LED strips, or par lights), not mixed in a single Layouts. ## Layers and Layer @@ -371,16 +370,14 @@ Effects know nothing about hardware, protocols, physical LED layout, or mapping. ### Buffer persistence — the layer does not clear each frame -The Layer's buffer **persists** frame to frame: `Layer::loop()` does not clear it before running effects. The buffer holds the previous frame until an effect overwrites or fades it, and is zeroed **once** on allocation/resize. (This is the standard LED-animation model — FastLED, WLED, and MoonLight all persist their frame buffer rather than auto-clear.) Persistence holds **between frames, not across rebuilds**: `Layer::onBuildState()` clears the buffer once after `rebuildLUT()`, so adding, replacing, or reconfiguring an effect (or a resize) starts from black, then persistence takes over frame to frame again. Each effect owns its own background: - -- A **full-grid** effect (Plasma, Rainbow, Fire, Noise) writes every pixel each frame — the previous frame is simply overwritten. -- A **trail** effect calls `layer()->fadeToBlackBy(amt)` to decay the previous frame before painting its new pixels — a comet leaves a fading tail because the old pixels are still there to fade. -- A **read-prior** effect (a scroll like FreqMatrix, Game-of-Life, a blur) reads last frame's pixels via `draw::get` / `draw::blur` and acts on them — the persistence *is* its state. -- A **sparse** effect that wants a clean frame calls `draw::fill(buf, {0,0,0})` itself (e.g. RubiksCube, whose draw touches only surface voxels). +The Layer's buffer **persists** frame to frame: `Layer::loop()` does not clear it before running effects (the decision, and why not clear-each-frame, is [ADR-0003](adr/0003-layer-buffer-persists-frame-to-frame.md)). It is zeroed once on allocation/resize, and once more in `Layer::onBuildState()` after `rebuildLUT()`, so a rebuild starts from black and persistence then holds between frames. Each effect owns its background: -There is deliberately **no per-effect "persistence" flag**: persistence is universal, so a flag would change no framework behaviour (unlike `dimensions()`, which drives extrude). Multiple effects on one layer *interact* through the shared persistent buffer — that is a feature. +- A **full-grid** effect (Plasma, Rainbow, Fire, Noise) writes every pixel each frame. +- A **trail** effect calls `layer()->fadeToBlackBy(amt)` to decay the previous frame, so a comet leaves a fading tail. +- A **read-prior** effect (FreqMatrix scroll, Game-of-Life, a blur) reads last frame's pixels via `draw::get` / `draw::blur`; the persistence *is* its state. +- A **sparse** effect that wants a clean frame calls `draw::fill(buf, {0,0,0})` itself (e.g. RubiksCube). -**Collected fade (`Layer::fadeToBlackBy`).** Fade is a Layer operation, not a per-effect buffer pass (MoonLight's `VirtualLayer::fadeToBlackBy`). Effects register a fade amount; the Layer keeps the **MIN** across all requesting effects (the gentlest fade wins, so the longest requested trail is honoured) and applies **one** whole-buffer pass at the start of the next frame, then resets the collected amount. So N fading effects on one layer cost a single pass, not N, and one effect's fade never darkens another's just-painted pixels mid-frame. An auto-clear would make trails and read-prior effects impossible without a shadow buffer; this model gets them for free. +Fade is a Layer operation: effects register an amount, the Layer keeps the MIN across them and applies one whole-buffer pass at the next frame's start, so N fading effects cost one pass, not N, and never darken each other's fresh pixels. ### Dimensionality @@ -392,7 +389,7 @@ Every effect declares its native dimensionality through `EffectBase::dimensions( D1/D2 are **opt-in promises**: declaring them tells the framework it can fill the missing axes, saving the per-effect work of iterating z (or x and z). Effects that don't make that promise stay at the D3 default and iterate the whole buffer. -**Why 1D runs along Y, and the unified expand rule.** A lower-D effect occupies the *low* axes and the framework expands it across the *next* axis: **1D → 2D adds columns across X** (the 1D output is the first column, duplicated rightward); **2D → 3D adds slices across Z** (the 2D front face, duplicated in depth). 1D-along-Y is the deliberate choice (shared with MoonLight): it makes a 1D effect the natural **first column** of its 2D form — write the effect once down Y, and expanding to a panel is just "repeat the column," same math, no special-casing. (The alternative, 1D-along-X, would make 1D a *row* that expands *downward* — a less natural fit, since a strip is a column and a 2D effect's columns are what you tile.) Concretely: a 1D effect draws its shape down Y, so it renders correctly on a layer whose single populated axis is Y (a `1 × N` grid, width 1, height N); on an `N × 1` grid the extrude would run the wrong way and collapse the shape to a flat line. How a physical output (a strip, a row of [Hue lights](moonmodules/light/moxygen/HueDriver.md)) maps to a `1 × N` grid is a layout concern — see the layout docs. +**Why 1D runs along Y, and the unified expand rule.** A lower-D effect occupies the low axes and the framework expands across the next: **1D → 2D adds columns across X**, **2D → 3D adds slices across Z**. 1D-along-Y (shared with MoonLight) makes a 1D effect the natural first column of its 2D form, so expanding to a panel is just "repeat the column," same math (1D-along-X would make it a row expanding downward, a worse fit since a strip is a column). A 1D effect therefore renders correctly on a `1 × N` grid (width 1, height N), but on `N × 1` the extrude runs the wrong way and flattens it. How a physical output (a strip, a row of [Hue lights](moonmodules/light/moxygen/HueDriver.md)) maps to `1 × N` is a layout concern. Hot-path cost: extrude pays one comparison and returns for the D3 case. For D1/D2 on a layer whose unused axes are size 1 (a D2 effect on a 2D layer, a D1 effect on a 1D `1 × N` layer) the inner loops are guarded by `depth_ > 1` / `width_ > 1` and never run. Real `memcpy` work happens only for a D1 or D2 effect on a layer with more dimensions than the effect writes: exactly the case where you wanted the framework to do the duplication. @@ -422,10 +419,10 @@ MoonLive lets you author an effect (later: a layout, modifier, driver, or core r The engine is a **domain-neutral core** with one narrow seam, structured as three tiers so adding a CPU is additive, never a rewrite: - **Front-end** (`src/core/moonlive/`, platform-independent): a recursive-descent lexer + parser over an expression grammar (every function argument is a literal or a nested call) that lowers each statement to a typed **IR** — a flat list of three-address ops over virtual registers. The IR is the seam: it knows *operations*, never an ISA and never a domain. It is compile-time only — consumed during lowering and discarded, so it costs nothing at run time; the CPU executes only the final native instructions. -- **Host builtin table** (the domain seam): the core owns no function names. A *host* registers `{name → descriptor}` — `setRGB`/`fill`/`random16` for LEDs (`src/light/moonlive/`), something else for a display or sensor. A descriptor is either a `Call` (a generic call to a host C function pointer — a pure helper like `random16`) or an `Inline` op (a neutral opcode tag the backend emits inline — a buffer writer, no per-pixel call). This is the [ESPLiveScript / ARTI bound-function model](backlog/livescripts-analysis-top-down.md); it is what keeps the core LED-free while the hot path stays inline. The LED *names* and the "an element is 3 RGB bytes" meaning live only in the light-domain registration and the per-ISA lowering, never in core. +- **Host builtin table** (the domain seam): the core owns no function names. A *host* registers `{name → descriptor}` — `setRGB`/`fill`/`random16` for LEDs (`src/light/moonlive/`), something else for a display or sensor. A descriptor is either a `Call` (a generic call to a host C function pointer — a pure helper like `random16`) or an `Inline` op (a neutral opcode tag the backend emits inline — a buffer writer, no per-pixel call). This is the ESPLiveScript / ARTI bound-function model; it is what keeps the core LED-free while the hot path stays inline. The LED *names* and the "an element is 3 RGB bytes" meaning live only in the light-domain registration and the per-ISA lowering, never in core. - **Per-ISA backend** (`src/platform/`, behind the boundary): a tiny named-instruction MacroAssembler (the textbook V8 / LLVM / asmjit shape — append one instruction, back-patch label offsets) plus the IR→bytes lowering that drives it. Xtensa (classic ESP32 / S3), RISC-V (P4), and the host ISA (desktop arm64/x86-64) each are *a new backend file behind the unchanged IR* — the front-end and IR never branch on ISA. Emitted code goes into an `allocExec` block (see [§ Platform abstraction](#platform-abstraction)) and is called each tick. -A recompile is the normal cold-path rebuild: editing the `source` control routes through the same `onBuildState()` sweep every control change uses, so a new script swaps in live (no reboot), and a parse error surfaces in the module status while the layer renders dark — robust to any input. The full design (the staged language ladder, the safety model, the performance budget, the memory-arena plan as the language grows) lives in [docs/backlog/livescripts-analysis-top-down.md](backlog/livescripts-analysis-top-down.md); the module contract is [docs/moonmodules/light/MoonLiveEffect.md](moonmodules/light/MoonLiveEffect.md). +A recompile is the normal cold-path rebuild: editing the `source` control routes through the same `onBuildState()` sweep every control change uses, so a new script swaps in live (no reboot), and a parse error surfaces in the module status while the layer renders dark — robust to any input. The module contract is [MoonLiveEffect](moonmodules/light/MoonLiveEffect.md). ## Modifiers @@ -457,7 +454,7 @@ Because mapping and blending happen in a single pass over each layer, there is n **Drivers** (a MoonModule) is the top-level container for one or more drivers. It is the consumer side of the pipeline. The Drivers container owns a shared output buffer and performs blend+map from every layer's buffer into it each frame. Individual drivers then read from this buffer to push to hardware / network. -The shared output buffer is necessary when blend+map writes to arbitrary physical positions via the LUT: the output is not filled sequentially, so a driver cannot read chunk-by-chunk until the full buffer is populated. It is *not* needed for the single-layer, no-blend case (identity or serpentine-shuffle mapping): there a driver can fuse map + output correction + protocol encode into one pass straight into its own output (DMA buffer / packet), skipping the shared buffer. Full detail in [the LED-driver design doc](backlog/leddriver-analysis-top-down.md). +The shared output buffer is necessary when blend+map writes to arbitrary physical positions via the LUT: the output is not filled sequentially, so a driver cannot read chunk-by-chunk until the full buffer is populated. It is *not* needed for the single-layer, no-blend case (identity or serpentine-shuffle mapping): there a driver can fuse map + output correction + protocol encode into one pass straight into its own output (DMA buffer / packet), skipping the shared buffer. Each driver (a MoonModule) speaks one protocol: @@ -468,7 +465,7 @@ Each driver (a MoonModule) speaks one protocol: Each driver child reads from the Drivers container's output buffer. Everything before the Drivers container is platform-independent. -**Output correction** turns logical RGB into the physical signal every physical driver needs: **brightness** scaling, channel **reorder** (RGB→GRB etc. via a *light preset*), and **white** derivation for RGBW fixtures. The Drivers container owns the shared correction state, a brightness lookup table plus the light-preset, exposed as `brightness` and `lightPreset` controls. Each *physical* driver applies the correction per-light as it reads its source buffer, into its own output buffer/packet. Preview is exempt: it shows the raw logical buffer (the effect's true output, not the dimmed/reordered wire signal). Every physical driver consumes the correction via a `const Correction*` set by `Drivers`, the RMT / LCD_CAM / Parlio LED drivers and `NetworkSendDriver` alike; `Drivers` hands the same pointer to each child it wires. The brightness LUT rebuilds on the cheap `onUpdate` tier (see [§ Event triggering between modules](#event-triggering-between-modules)), so the slider stays fluent. +**Output correction** turns logical RGB into the physical signal: **brightness** scaling, channel **reorder** (RGB→GRB via a *light preset*), and **white** derivation for RGBW. The Drivers container owns the shared correction state (a brightness LUT + the light-preset, exposed as `brightness` / `lightPreset` controls) and hands each physical driver a `const Correction*`; the driver applies it per-light into its own buffer/packet. Preview is exempt (it shows the raw logical buffer). The brightness LUT rebuilds on the cheap `onUpdate` tier ([§ Event triggering](#event-triggering-between-modules)), so the slider stays fluent. Network-based drivers (ArtNet, E1.31, DDP) pace their output with a **non-blocking elapsed-time gate**, never a blocking wait (no `delay`/`vTaskDelay` — that would stall the single-threaded tick, the hot-path rule). The gate is the `lastSendTime`/`millis()` pattern: `if (now − lastSendTime < interval) return;` early-exits the tick so every other module's loop keeps running, exactly how FPS limiting works (`NetworkSendDriver`, `fps` control). **Frame-rate pacing is required** and implemented this way. **Inter-packet pacing** (spacing the universes within one frame) uses the same non-blocking gate *if* a receiver drops packets under a burst — it is not needed by default (the bench ArtNet matrix test runs clean bursting the universes), so it is added only when a target requires it, never as a busy-wait between packets. @@ -484,7 +481,7 @@ A module holds heap **only for capabilities it is actually exercising**, the sam - **A feature's buffer allocates on first use, not at `setup()`.** When a module *is* present but a given capability is dormant (a driver with no output attached, an MQTT client with HA discovery toggled off), that capability's buffer is `nullptr` until the code path that needs it runs. Allocating eagerly at `setup()` for a path that may never execute is the anti-pattern this rule forbids — it charges every instance for the worst case. - **The allocation frees in `teardown()`** (and on the transition that makes the capability dormant again — a disable, a toggle-off), and is reported through `dynamicBytes()` so `/api/system` and the memory scenarios see the real ladder. `MoonModule::teardown()` reverse-recurses into children, so a subtree's memory unwinds bottom-up with no leak. -The result is a memory ladder that tracks configuration exactly: module-absent → 0; module-present-but-feature-off → just the class instance; feature-active → `+dynamicBytes()`. The LED driver's output buffer (allocated when a physical output exists, per [§ Adaptive allocation](#adaptive-allocation) below) and the MQTT module's discovery-config scratch (allocated when a connected client announces itself to Home Assistant) are the two worked examples; the rule governs every module, not those two. Rationale: on a no-PSRAM ESP32 the internal-heap reserve (`HEAP_RESERVE`) is the tightest constraint in the system, so a module that grabs a buffer it isn't using is spending the reserve that keeps the render loop, WiFi, and HTTP alive. +The result is a memory ladder that tracks configuration exactly: module-absent → 0; module-present-but-feature-off → just the class instance; feature-active → `+dynamicBytes()`. The LED driver's output buffer and the MQTT module's discovery-config scratch are the worked examples; the rule governs every module. It matters most on a no-PSRAM ESP32, where the internal-heap reserve (`HEAP_RESERVE`) is the tightest constraint, so a buffer held but unused spends the reserve the render loop, WiFi, and HTTP depend on. ### Buffer types @@ -498,7 +495,7 @@ Network input (ArtNet receive, WebSocket) is processed synchronously at a define ### Adaptive allocation -The system checks available heap before each allocation and degrades gracefully when memory is insufficient. A minimum reserve (`HEAP_RESERVE = 32 KB`) is kept for stack, HTTP, WiFi, and overhead. +The system checks available heap before each allocation and degrades gracefully when memory is insufficient (the allocate-on-demand-with-a-cascade decision, over fixed buffers, is [ADR-0002](adr/0002-adaptive-memory-degradation-cascade.md)). A minimum reserve (`HEAP_RESERVE = 32 KB`) is kept for stack, HTTP, WiFi, and overhead. - **Mapping LUT** is created only if all of: modifiers exist on the layer; layout is not a simple non-serpentine grid (where physical == logical); enough heap available after the reserve. - **Driver output buffer** (see [§ Drivers](#drivers) for what it's for) is created only when the pipeline must write into physical space rather than hand a driver a layer's logical buffer directly — that is, when **two or more layers are enabled** (they must be composited into one buffer) **or** a layer has a **mapping LUT** actually allocated (logical≠physical) — and enough heap is available. A single enabled layer with no LUT needs no output buffer: drivers read its buffer directly (the zero-copy fast path). @@ -572,10 +569,3 @@ A module's chips come from three sources, rendered identically on the card and t | **Audio** (`tags()`) | 🔊 audio-reactive | reads `AudioModule::latestFrame()` | `tags()` carries **only** origin + creator + audio (+ any genuinely module-specific marker); a module can carry several (e.g. `💫🦅` = MoonLight origin, a named creator). Role and dim are added by the UI, so a module never duplicates them in its string. When migrating, set each module's `tags()` from this legend so the chip set is consistent across the library. - -## What we leave undesigned - -Genuinely open questions, *not* the same as a 🚧 marker. A 🚧 item has a settled, committed design (two-core handover, clock sync, device-to-device light distribution) — code is written toward it; the items here are ones where the *design itself* is still open, deferred until a concrete need forces the decision: - -- **WiFi runtime disable**: today the eth-only build profile compiles WiFi out. Whether runtime gating should key off detected hardware presence, an explicit control, or a deviceModel-catalog field isn't decided; the eth-only build covers the need until one is. -- **Mixing light types in one Layouts**: each layout child describes one light type (all LED strips, or all par lights). Whether a single Layouts container should hold mixed types (LED strips + par lights together), and how the channel layout would reconcile across them, isn't designed; one Layouts per light type is the current model. diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index ace0192d..0705877b 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -147,7 +147,7 @@ On `esp32-eth-wifi`, default 128×128 grid, free heap at boot is ~28 KB — not Fix options in increasing scope: - **Cap the default grid** — drop to 64×64 on `esp32-eth-wifi` (Layer ~32 KB + LUT ~16 KB = 48 KB, comfortably under). Simplest. -- **PSRAM for Layer buffer + LUT** — ESP32-Gateway has 4 MB PSRAM unused on non-S3 builds. Moving the 49 KB pixel buffer + 64 KB LUT out of DRAM frees ~110 KB for radios. Cost: ~25% FPS hit (PSRAM bandwidth ~12 MB/s vs DRAM ~80 MB/s); needs measurement. See [decisions.md](../history/decisions.md) "Adaptive memory allocation design" for the allocation rules. +- **PSRAM for Layer buffer + LUT** — ESP32-Gateway has 4 MB PSRAM unused on non-S3 builds. Moving the 49 KB pixel buffer + 64 KB LUT out of DRAM frees ~110 KB for radios. Cost: ~25% FPS hit (PSRAM bandwidth ~12 MB/s vs DRAM ~80 MB/s); needs measurement. See [lessons.md](../history/lessons.md) "Adaptive memory allocation design" for the allocation rules. - **Lazy WiFi init** — skip `esp_wifi_init` when `ssid_` is empty and no AP-fallback is pending. Helps only when credentials exist but the network is unreachable — niche. ### Boot-time buffer degradation on non-PSRAM at 128×128 (investigation) @@ -170,6 +170,10 @@ No FreeRTOS tasks are pinned today. At 16K LEDs the render task takes ~52 ms/tic ## Architecture +### WiFi runtime disable — open design question (undesigned) + +Today the eth-only build profile compiles WiFi out (`MM_NO_WIFI`). Turning WiFi off *at runtime* instead is undesigned: whether the gate should key off detected hardware presence, an explicit control, or a deviceModel-catalog field isn't decided. The eth-only build covers the need until a concrete case forces the choice. (Moved from architecture.md § What we leave undesigned; it's a deferred design decision, not a settled 🚧 one.) + ### Consolidate the two module-by-name tree-walkers (backlog) `HttpServerModule::findModuleByName` (`findInTree` recursion) and `Scheduler::firstByName` (`firstInTree`) are two implementations of the same "find the first module in tree-walk order with this name" operation. The duplication predates the `setControl`-to-Scheduler extraction, but that extraction made `Scheduler::firstByName` public *and* added `Scheduler::instance()`, so HttpServer no longer needs its own copy: its ~9 remaining call sites (identify, addModule, removeModule, clearChildren, the WLED/system shims) can call `scheduler_->firstByName()` and the private `findModuleByName`/`findInTree` pair deletes. Purely a *No duplication* cleanup — no behaviour change — worth doing so the walk order/semantics live in exactly one place. (Reviewer note, IrModule/setControl branch.) @@ -239,7 +243,7 @@ When picked up: add `offsetX/Y/Z` (lengthType) controls to `LayoutBase`; `Layout ### Improv as a child of NetworkModule (deferred — needs scheduler work first) -Architecturally the right shape; attempted in plan-21, reverted. Blocker: `Scheduler::tick()` only walks top-level modules for `loop20ms`/`loop1s` — children silently miss those callbacks. See [decisions.md](../history/decisions.md) "Trying to add a child module to NetworkModule". +Architecturally the right shape; attempted in plan-21, reverted. Blocker: `Scheduler::tick()` only walks top-level modules for `loop20ms`/`loop1s` — children silently miss those callbacks. See [lessons.md](../history/lessons.md) "Trying to add a child module to NetworkModule". Minimum-scope fix before the move: 1. `MoonModule::loop20ms`/`loop1s` propagate to children (or Scheduler walks them) — pick whichever costs less at runtime. diff --git a/docs/backlog/backlog-light.md b/docs/backlog/backlog-light.md index 64b6d32d..532c8074 100644 --- a/docs/backlog/backlog-light.md +++ b/docs/backlog/backlog-light.md @@ -20,7 +20,7 @@ The **classic ESP32 has 8 RMT TX channels** (`platform_config.h`: "8 on classic ### Audio-reactive follow-ups -The manual level + 16-band FFT spectrum has shipped (AudioModule; what landed and why is in [decisions.md](../history/decisions.md)). These are the deferred follow-ups, each its own increment: +The manual level + 16-band FFT spectrum has shipped (AudioModule; what landed and why is in [lessons.md](../history/lessons.md)). These are the deferred follow-ups, each its own increment: - **Per-band noise-floor (kill a steady single-frequency hum)** — the bench mic picks up a constant ~258 Hz tone (a mains harmonic via the mic/supply) that lights one band even in silence. A high-pass can't remove it (it's well above the ~40 Hz DC-blocker cutoff) without also killing real bass; the clean fix is a per-band adaptive floor that learns each band's idle baseline and subtracts it, so a constant tone in one band gates to dark while the others stay sensitive. Minimal version ≈ 16 floats of state + ~16 ops/frame. This is the next concrete audio step. - **Adaptive conditioning** — auto noise-floor / auto-gain / smoothing so the display self-calibrates to a room ("sound off → dark, sound on → vivid") instead of being tuned by hand. A self-calibrating version was prototyped and removed; the manual `floor`/`gain` is the shipped baseline. Reinvent from scratch when wanted, and **tune it in a quiet room** — a noisy environment (a strong, varying low-frequency ambient) is the adversarial case that made the prototype hard to settle. (The per-band floor above is the first piece of this.) @@ -110,6 +110,10 @@ The light `Buffer` already holds `channelsPerLight = 4` (RGBW), and the device o Today a "light" is a point at a static coordinate with a colour. A **moving head** is a fixture that emits a *beam* in a direction it controls live (pan + tilt), plus colour, beam-width, etc. — per-light **vector** state, not just colour, and a different draw (a cone/ray, not a disc). The static-positions-`0x03` + colour-`0x02` split can't express "this fixture's beam now points here." The industry-standard model is **DMX/GDTF fixtures**: a fixture has a position *and* a set of typed attributes (color, pan, tilt, beam). The preview becomes a fixture renderer (disc for a pixel, cone for a beam); this is also the "make Preview a general-purpose module, not light-specific" goal. A domain-model change (the fixture/attribute model), not just transport. Plan when moving heads are actually on the bench. +### Mixing light types in one Layouts — open design question (undesigned) + +Today each layout child describes one light type (all LED strips, or all par lights), and the current model is one Layouts container per light type. Whether a single Layouts should hold mixed types (LED strips + par lights together), and how the per-channel layout would reconcile across them, isn't designed. Deferred until a concrete need forces it; it's adjacent to the fixture model above (a real fixture/attribute model may reframe how mixed types are expressed). (Moved from architecture.md § What we leave undesigned; a deferred design decision, not a settled 🚧 one.) + ### Extract the resumable backpressure transport as a domain-neutral channel (long term) The preview's transport — resumable cross-tick send from a stable buffer + newest-wins backpressure drop + adaptive graceful degradation (see [architecture.md § graceful degradation under transport backpressure](../architecture.md)) — is **payload-agnostic**: any bulky throttled stream (a future MJPEG/video preview, fixture-state streams, fleet telemetry) could ride it. The *payload* model (count/stride/RGB) is light-specific; the *byte-pump* is not. When a second consumer for this transport appears, promote the pump into a domain-neutral core primitive (a `ThrottledChannel`-style sink) that PreviewDriver becomes *a* producer on, rather than owning the protocol. Concrete-first: extract on the second use, not before — until then the seam stays inside HttpServerModule/PreviewDriver. @@ -131,7 +135,7 @@ For driving **lots of LEDs**, internal SRAM is the scarce resource and the paral ## LED drivers — deferred -The LED-driver increments **shipped**: increment 1 (RMT/WS2812B single-strand on classic ESP32 — [`RmtLedDriver.h`](../../src/light/drivers/RmtLedDriver.h), `RmtSymbol.h`, `platform_esp32_rmt.cpp`) and increment 2 (2a multi-pin RMT, 2b parallel LCD_CAM on the S3 — [`LcdLedDriver.h`](../../src/light/drivers/LcdLedDriver.h) via [`ParallelLedDriver.h`](../../src/light/drivers/ParallelLedDriver.h), `platform_esp32_lcd.cpp`), all with host + on-board-loopback tests, hardware-proven. The locked decisions, file-by-file phases, the WiFi-flicker test-rig analysis, and the bench deviations (8-GPIO i80 bus, 2.67 MHz slot clock, SOC-macro gate, real-frame loopback) are in [decisions.md](../history/decisions.md), the [driver docs](../moonmodules/light/moxygen/RmtLedDriver.md), and the [analysis docs](leddriver-analysis-top-down.md). What remains here is only the work that has **not** shipped and is tracked nowhere else. +The LED-driver increments **shipped**: increment 1 (RMT/WS2812B single-strand on classic ESP32 — [`RmtLedDriver.h`](../../src/light/drivers/RmtLedDriver.h), `RmtSymbol.h`, `platform_esp32_rmt.cpp`) and increment 2 (2a multi-pin RMT, 2b parallel LCD_CAM on the S3 — [`LcdLedDriver.h`](../../src/light/drivers/LcdLedDriver.h) via [`ParallelLedDriver.h`](../../src/light/drivers/ParallelLedDriver.h), `platform_esp32_lcd.cpp`), all with host + on-board-loopback tests, hardware-proven. The locked decisions, file-by-file phases, the WiFi-flicker test-rig analysis, and the bench deviations (8-GPIO i80 bus, 2.67 MHz slot clock, SOC-macro gate, real-frame loopback) are in [lessons.md](../history/lessons.md), the [driver docs](../moonmodules/light/moxygen/RmtLedDriver.md), and the [analysis docs](leddriver-analysis-top-down.md). What remains here is only the work that has **not** shipped and is tracked nowhere else. - **sigrok/fx2lafw cross-check + MoonDeck "LED driver test" Python script** — the independent-clock proof and the run-from-MoonDeck flow ([analysis §5.3](leddriver-analysis-top-down.md)). The on-board RMT-RX loopback (shipped) is the cheap CI correctness gate but a *compromised witness* for WiFi-induced flicker — the RX capture runs on the same ESP32 whose WiFi causes the glitch. The real flicker test is a **sustained capture (seconds) with WiFi associated + a packet flood**, decoding every frame for a byte-slip or reset-gap deviation; it belongs with the core-1 driver-task work below, since that task pinning is the *fix* it validates. A DSLogic Plus (100 MS/s) upgrade is reactive — only if a flicker reproduces that 24 MS/s can't resolve. - **Dedicated core-1 driver task + per-module core-affinity control** ([analysis §7.2](leddriver-analysis-top-down.md)) — the WiFi-glitch mitigation, shared across all the LED drivers. (See also [backlog-core § Task core-pinning](backlog-core.md#task-core-pinning-backlog) for the general task-pinning question.) diff --git a/docs/backlog/livescripts-analysis-top-down.md b/docs/backlog/livescripts-analysis-top-down.md index 2aa2dfdd..76dae106 100644 --- a/docs/backlog/livescripts-analysis-top-down.md +++ b/docs/backlog/livescripts-analysis-top-down.md @@ -378,7 +378,7 @@ The first sequencing question is **depth-first** (build the whole engine on Xten The [MoonLight effects tutorial](https://moonmodules.org/MoonLight/moonlight/effects-tutorial/) is a ready-made *start-small-grow* curriculum (random pixel → control → trails → oscillators → 2D → 3D → audio → Cosmic Noise). Each rung is **one engine-capability spike** with a concrete acceptance bar. **RipplesEffect.h is the graduation test** (the hard real effect, after the 3D rung). Each step is a normal small commit; the multi-target part is sequenced per §9.1 — the *seam* is proven on a second ISA at Stage 0.5, but the *full* second backend (all stages) comes later, opportunistically. -**Next up: Stage 2 (buffer read-modify-write / trails).** Stages 0, 0.5 and 1 (Controls) are done — what landed and why is recorded in the git history and [decisions.md](../history/decisions.md); this ladder tracks the remaining rungs. +**Next up: Stage 2 (buffer read-modify-write / trails).** Stages 0, 0.5 and 1 (Controls) are done — what landed and why is recorded in the git history and [lessons.md](../history/lessons.md); this ladder tracks the remaining rungs. | Stage | Capability proven | Acceptance bar (the spike) | |---|---|---| diff --git a/docs/coding-standards.md b/docs/coding-standards.md index a86df7ac..721217a5 100644 --- a/docs/coding-standards.md +++ b/docs/coding-standards.md @@ -34,6 +34,12 @@ Guidelines: Counter-example to avoid: storing `char rssiStr_[12]` and re-`snprintf`'ing `"-58 dBm"` into it every tick. The right shape is `int8_t rssi_` (1 byte) plus a control type that knows the unit. Saves 11 bytes per metric, scales linearly across the codebase. +**Width the intermediate, not just the result.** Any `a * b` where both operands are `nrOfLightsType` (or a count times a multiplier) can overflow `uint16_t` even when each operand is individually small (`256 * 256 = 65536` wraps to 0 on a no-PSRAM device); do the arithmetic in a wider type (`uint64_t`), clamp to the ceiling, then narrow. Likewise a counter derived from a cell count (a delta, a stagnation check) must be the domain typedef (`nrOfLightsType`), not a fixed `uint16_t`. A width-sensitive path is invisible on the uint32 desktop build, so pin it with a `uint16`-typed unit test or hardware confirmation. + +**When a validation field's storage is narrower than what it claims to validate, the validation is wrong, not the field.** A `uint8_t` min/max slot can't bound an `Int16` control (it clamps to `[0..0]`); the fix is a wider bound (or per-type bound slots), and until then the constraint is documented at the field's declaration. + +**When one control type does two jobs with different UX, that's the smell for a new type, not a range hack.** An `int16` control the UI renders as a slider can't also mean "GPIO pin number"; a dedicated `Pin` type (smallest storage that fits the domain, `int8_t` for a GPIO) is the fix, not overloading the range. + ## Per-type behaviour lives with the type When a struct or enum is the semantic owner of some data — a control descriptor, a packet, a module role — the functions that interpret, serialise, validate, or otherwise operate on it should live next to the type, not at the call sites that use it. Free functions in the same `.cpp` count; member methods on the owning class are stronger; virtual methods on a base class are strongest. The wrong shape is the same `switch (type)` repeated in every consumer — adding a variant means hunting across N files for switches to extend, and the compiler can't tell you when one gets missed. @@ -48,6 +54,10 @@ Counter-example to avoid: a `switch (c.type)` on `ControlType` duplicated in Htt When a `switch (type)` outside the type's home file is legitimate: the caller has a genuinely different concern (HttpServerModule mapping `ApplyResult` to HTTP status codes is a transport policy, not per-type behaviour; scenario_runner's `switch (JsonVal::type)` dispatches on *its own* discriminator, not `ControlType`). The rule is "per-type dispatch lives with the type", not "switches are banned". +**A flat parser that returns a zero sentinel for a missing key must be paired with a presence check before the value is applied as authoritative.** `json::parseInt(json, key)` returns 0 for an absent key, indistinguishable from a real 0; on a persistence-overlay load path that clobbers a non-zero default when an older/partial file omits the key. Guard with `json::hasKey()` first: an absent key leaves the control untouched (its default stands); a present key (even value 0) applies. Any "control resets to its default/0 after reboot" symptom is this overlay smell, not a control-init bug. + +**A reader that decodes only a subset of the escapes its writer emits is a latent asymmetry bug.** If the writer emits `\n` / `\t` escapes, the reader must decode them (not only `\"` / `\\`); make the escape set symmetric so multi-line text round-trips. + ## File shape: header-only vs `.h` + `.cpp` - **Light-domain modules and the `MoonModule` base: header-only.** Every effect, modifier, driver, layout, the light-domain containers (`Layouts`, `Layers`, `Drivers`, `Layer`), and the `MoonModule` base class live in a single `.h` with implementation inline. The benefit is concrete: a contributor copies `RainbowEffect.h`, edits, saves as `MyEffect.h`, registers one line in `main.cpp` — no "where does the `.cpp` go, what does CMake need" friction. The chain `RainbowEffect.h → EffectBase.h → MoonModule.h` is uniform; readers don't pivot to a different file shape at the base. When a light-domain file outgrows one concern, extract a helper into its own header (`BlendMap`, `MappingLUT`) rather than splitting to `.h` + `.cpp`. Header-only is a feature of the light domain. @@ -96,7 +106,7 @@ All targets build warnings-as-errors: `-Wall -Wextra -Werror` on Clang/GCC (macO ## Static checks - **Platform boundary** (`moondeck/check/check_platform_boundary.py`) — scans all files outside `src/platform/` for `#ifdef` / `#if defined` with platform macros and `#include` of platform-specific headers (`esp_*`, `freertos/*`, `driver/*`, `SDL.h`, `wiringPi.h`, …). Fails if any are found. The platform boundary rule itself: [architecture.md § Platform abstraction](architecture.md#platform-abstraction). -- **Hot path lint** — flags allocation calls (`new`, `malloc`, `make_unique`, `make_shared`, `push_back`, `std::string` constructors) inside functions identified as hot path (render loop and callees). A code-review convention, enforced by hand. The hot path rule itself: [architecture.md § Hot path discipline](architecture.md#hot-path-discipline). +- **Hot path lint** — flags allocation calls (`new`, `malloc`, `make_unique`, `make_shared`, `push_back`, `std::string` constructors) inside functions identified as hot path (render loop and callees). A code-review convention, enforced by hand. The hot path rule itself: [architecture.md § Hot path discipline](architecture.md#hot-path-discipline). A "no blocking in the hot path" audit must sweep *every* syscall the path can reach (connect, DNS, read, *and* write), not just the loudest one: a single-threaded loop that services I/O must make no blocking call at all (a socket timeout is not a fix, it is the size of the freeze; non-blocking + poll is the only safe shape). Fixing one blocker while an equally-blocking sibling survives is a partial fix that reads as complete. - **Code formatting** — `clang-format` with a project `.clang-format` file. Applied in CI; code that doesn't match fails the check. Run locally via editor integration or `clang-format -i`. ## When checks run @@ -148,3 +158,25 @@ A module's story therefore lives in exactly two places: its `.h` (technical, gen ### Test inventories: their own generator, not moxygen `docs/tests/*.md` is generated by [`moondeck/docs/generate_test_docs.py`](../moondeck/docs/generate_test_docs.py), not moxygen. Unit tests are `TEST_CASE("…")` macros tagged with `// @module` and per-case `//` descriptions — a convention Doxygen documents nothing of (it parses C++ *entities*, not macro string literals) — and scenario tests are JSON, which moxygen cannot read. moxygen is for the `.h` module pages; the test generator owns the test pages. + +## Defaults + +**Assign a default only where the hardware, not the user's soldering iron, fixes the value.** The test is *who fixes the pin/setting*: + +- **Chip-/board-fixed → default it, and you must.** The RMII Ethernet pin map, the on-board status LED, a country code per region: silicon-/PCB-wired, so a default cannot do harm, and *omitting* it does (a no-WiFi board with un-defaulted Ethernet pins can never connect to be configured, a chicken-and-egg lockout). +- **User-soldered → leave it unset.** A MEMS mic, an LED strand, an LED-driver pin goes wherever the user ran the wire, so any default is a guess that can drive a pin the user committed elsewhere. Empty until set; idle with a "set pins" status meanwhile (degraded is fine, crashed is not). + +A "default" that is really one specific board's values is bespoke masquerading as standard (a § Principles violation): make the capability opt-in and require each consumer to state its own values, so a missing declaration fails loudly instead of inheriting a stranger's wiring. Never auto-run a peripheral whose init can block on absent hardware. The design rationale (the MCU → deviceModel provenance model) lives in [architecture.md § Config provenance](architecture.md#config-provenance-mcu-devicemodel). + +## Debugging and verification + +Hard-won discipline for diagnosing hardware and infrastructure failures, distilled from the war stories in [lessons.md](history/lessons.md). + +- **Prove the failure is *about* the change before editing code.** When something fails right after a change, re-run it isolated, probe the actual end state (ping/curl the device, read the real CI error line, check machine load), and confirm the artifact under test is the one you built (process uptime, `build` timestamp, what is bound to the port). A stale process, a loaded machine, or an async-confirmation timeout reads as a regression it isn't. +- **A status/dimension assertion does not prove the pipeline renders.** A correctness test for a mapping or effect asserts the buffer or LUT is non-empty with the expected coverage (e.g. LUT destinations == physical light count), not just that the declared dimensions look right. +- **For a hardware bring-up, "it compiles" tells you almost nothing.** The truth is in the boot log on the actual board; a min-revision trap, a wrong PHY pin, a filename-keyed capability gate, or a Kconfig *choice* that an incremental build silently keeps all build clean and fail only on hardware (`rm -rf` the build dir when a sdkconfig choice changes). +- **When test and reality disagree, enumerate what the test abstracts away and make the test transmit the genuine article.** Each closed gap either finds the bug or eliminates a theory with proof (a whole-frame loopback that sends the driver's real frame, not a synthetic pattern). Prove the firmware is *not* the cause with a measurement at each layer before editing code or buying parts. +- **A measurement tool must be faithful to the real client, or it invents and hides bugs.** A one-shot WebSocket probe that gives up on close reports stalls a reconnecting browser never sees and misses blips it does; match the client's real behaviour (keepalive ping, auto-reconnect). +- **Stop at the first failed fix on a working path.** Revert to the known-good state at attempt two rather than re-engineering a seam that already worked (§ *Anti-stalling* in CLAUDE.md). +- **A generated artifact's ground truth is the rendered output, not a re-derivation.** Verify a doc anchor against the built HTML (`grep id=` the `.html`), not a reimplementation of the slug algorithm; verify an emitted machine instruction against the real toolchain's disassembler before flashing. +- **A cross-boundary fact duplicated in code drifts silently; gate it.** A `docPath` in `main.cpp` that points at a docs page, a firmware projection that mirrors a build dict: the moment a check can resolve it against ground truth, that check is cheaper than the drift. diff --git a/docs/history/README.md b/docs/history/README.md index 8700da2b..cdadf334 100644 --- a/docs/history/README.md +++ b/docs/history/README.md @@ -31,7 +31,7 @@ One-time surveys of earlier projects, used to decide what to harvest into projec ### Our own lessons -- [decisions.md](decisions.md) — hard-won lessons, proven patterns, and non-obvious decisions, recorded with the code that proved them (the PR-merge carry-forward gate writes here). +- [lessons.md](lessons.md) — hard-won debugging lessons and gotchas (a bug, its cause, the fix), recorded with the code that proved them and pruned as they are absorbed (the PR-merge carry-forward gate writes here). Genuine architectural *decisions* live in [`../adr/`](../adr/README.md) instead; a lesson that hardened into a *rule* lives in CLAUDE.md / coding-standards.md. ## Cross-repo trends diff --git a/docs/history/decisions.md b/docs/history/decisions.md deleted file mode 100644 index b5d2710e..00000000 --- a/docs/history/decisions.md +++ /dev/null @@ -1,861 +0,0 @@ -# Decisions and Lessons Learned - -## Actions for the Future - -Distilled from four project iterations (MoonLight, v1, v2, v3). These -are concrete rules, not aspirations. - -### Before writing any code - -1. **CLAUDE.md is the constitution.** Write it first. Include: platform - boundary rule, memory rules, build commands, code style, agent - workflow restrictions. Vague rules produce vague code. -2. **architecture.md must be stable.** Don't start coding while the - architecture is still being debated. Every mid-implementation - architecture change costs more than the original design time. -3. **Write module specs before module code.** Each docs/modules/*.md - defines: purpose, controls, behavior, edge cases, interactions. - These are specifications, not reverse-engineered documentation. -4. **Write a UI specification.** Design the UI before implementing it. - Define layout, controls, API, interaction rules. Don't discover - the UI through bug reports. -5. **Define integration tests upfront.** Unit tests per component are - necessary but not sufficient. Define full-pipeline tests that - exercise the path from effect → buffer → LUT → blend+map → driver - output. - -### During implementation - -6. **Concrete first, abstract later.** Build one working feature - end-to-end before extracting patterns into abstractions. Don't - design the MoonModule base class before the first effect works. -7. **Use /plan for every feature.** Review plans for: unnecessary - files, inheritance where structs suffice, modifications outside - the relevant directory. Reject and regenerate bad plans. -8. **Commit after every small success.** Git is the undo button. If - the agent fails a build twice, rollback and re-prompt. Don't let - it spiral into a fix-up loop. -9. **The agent never stages or commits.** The product owner stages, - reviews, and commits manually. -10. **Refactor first, create second.** Prefer extending existing code - over adding new files. Deletion is preferred over duplication. - Every addition must pay for itself. - -### Technical rules (embedded-specific) - -11. **Zero allocations in the render loop.** No malloc, new, push_back, - std::string in any code path called per-frame. Enforce with a - test that intercepts malloc/free. -12. **PSRAM for bulk, IRAM for speed.** Large contiguous buffers - (light buffers, LUTs) in PSRAM at boot. Never scattered small allocations - in PSRAM during loops. -13. **Pace all network output.** Never blast UDP packets in a tight - loop. Add inter-packet delay and FPS limiting. Missing pacing - looks like rendering bugs but is network flooding. -14. **Use virtual interfaces, not dynamic_cast.** Modules interact - through virtual methods. Layer should not know about MultiplyModifier - or CheckerboardModifier by name. -15. **Rebuild propagation must be in the framework.** Don't check dirty - flags per-module in main.cpp. Use an event/observer system or a - centralized pipeline-changed signal. -16. **Separate build entry points per platform.** Root CMakeLists.txt - for desktop, esp32/ wrapper for IDF. Don't pollute CMake files - with if(ESP_PLATFORM) conditionals. - -### Documentation and process - -17. **Present tense only.** Docs describe the system as it is. Git - commits are the history. No changelogs, no sprint files, no - roadmaps in the repo. -18. **Single source of truth.** CLAUDE.md (rules), architecture.md - (design), docs/modules/*.md (specs). Nothing else. If it's - duplicated, delete one copy. -19. **Mandatory subtraction.** Periodically review and remove code - and docs that no longer earn their place. -20. **Test suite is the safety net.** For both human and AI - development. If the agent breaks a test, it broke the system. - No exceptions, no skipping. - ---- - -## Lessons from MoonLight (WLED fork) - -### Memory and resource constraints -- On classic ESP32 (no PSRAM, 4MB flash), the framework's footprint - left little room for effects. OTA updates and expandability were - limited by tight resource budgets. - -### Upstream dependency burden -- Diverging from the upstream framework (ESP32-SvelteKit) required - tracking 1,271+ local changes to survive rebases. Ongoing - maintenance friction. -- Feature implementation was delayed waiting for upstream adoption - (Ethernet, async webserver). - -### Testing was retrofitted, not foundational -- Automated tests only verified trivial cases. Complex interactions - required code restructuring that broke encapsulation. -- Testing should be designed into the architecture from day one, not - bolted on later. - -### Sustainability -- Single maintainer carrying ~95% of workload while issue backlog - grew with user base. -- Actively-developed system frequently broke previously working - features — no regression safety net. - -### What to carry forward -- Platform abstraction (zero `#ifdef` in module code) is essential. -- Self-describing UI (JSON-driven, no frontend code per new module). -- Test suite as safety guardrail for both human and AI development. - -### Proven patterns from MoonLight (harvest for v3) - -- **PhysMap — 2-byte mapping entries on no-PSRAM.** Union packing: map type (2 bits) + physical index or RGB cache (14 bits) in a single uint16_t. On PSRAM: 4 bytes with 24-bit indices. The map type is stored IN each entry, not in a separate array. Saves memory vs v3's CSR approach which uses a separate offsets array. -- **nrOfLights_t typedef.** `uint16_t` on no-PSRAM, `uint32_t` on PSRAM. Compile-time selection. Proven in production. Same concept as v3's `nrOfLightsType`. -- **addControl binds to class variable by reference.** Control stores a `uintptr_t` pointer to the variable. Hot-path code reads the variable directly — zero overhead. UI updates write through the pointer. Supports uint8_t, int8_t, uint16_t, uint32_t, int, float, bool, Coord3D. -- **Node — minimal memory.** Base class ~29 bytes + vtable. Effects add only their control variables (uint8_t each). A typical effect adds just 2 bytes on top. No std::string members. -- **LightsHeader — one struct for LEDs AND DMX fixtures.** Configurable `channelsPerLight` (3=RGB, 4=RGBW, up to 32 for moving heads) with offset fields for red/green/blue/white/pan/tilt/zoom/rotate/gobo. This IS the "light = LED pixel or DMX fixture" concept. -- **Region start/end percentages.** MoonLight's `startPct`/`endPct` as Coord3D percentages (0-100) of the total fixture. Shipped in v3 as a modifier, not as Layer controls. -- **oneToOneMapping / allOneLight fast paths.** Boolean flags that skip the mapping table entirely when mapping is 1:1 identity. Essential for the no-PSRAM 12K LED case. -- **Transition brightness.** Per-layer animated brightness overlay (current, target, step). Enables smooth fade-in/out when switching effects. Should be added to v3. -- **SharedData — zero-allocation inter-node communication.** Single struct shared by all nodes: 16-band FFT audio, volume, beat detection, gravity (IMU), status. Lightweight alternative to pub/sub. -- **Coord3D with rich operators.** Full arithmetic (+, -, *, /, %), comparison, distanceSquared, isOutofBounds. Uses `int` (not int16_t) to avoid intermediate overflow. -- **10+ layout types proven.** Panel, Ring, Rings241, HexaPanel, Cloud, Human, Cone, Globe, SpiralGlobe — all algorithmic, no stored positions. -- **60+ effects across 5 libraries.** MoonLight originals, MoonModules community, WLED ports, FastLED demos, SoulmateLights. -- **11 driver types.** FastLED, Parallel LED, Hub75, ArtNet/E1.31/DDP in/out, DMX in/out, audio (WLED + FastLED), IMU, infrared. - ---- - -## Lessons from projectMM v1 - -### Module system collapse -- `StatefulModule` conflated five distinct jobs: lifecycle, JSON - descriptors, persistence, child management, and timing windows. - Should have been separate concerns. - -### PAL boundary erosion -- Only ~50 of 1,397 PAL lines handled actual platform abstraction. - The rest was networking, OTA, NTP, and system queries that should - have been modules. The PAL grew into a kitchen-sink. - -### Manager overloading -- `ModuleManager` became REST adapter, memory accountant, state - persister, and dirty-flag debouncer simultaneously. Too many - responsibilities in one class. - -### Documentation bloat -- 90 markdown files across ~15,400 lines. A contributor cannot grasp - the core concept in five minutes. -- Architecture documentation read as historical narrative rather than - prescriptive design. -- Per-module documentation duplicated JSON schemas already rendered - by the UI. - -### Testing pyramid never materialized -- Four parallel test surfaces (unit, live, scenario, code-analysis) - were created independently and never unified despite each solving - real problems. - -### What worked -- Line-count tagging per file enabled fast assessment of bloat. -- Frugality principle: every addition must pay for itself. -- Growth budgets with CI enforcement prevented drift. -- Architecture as constraint document (not narrative). - -### Key insights for v2+ -- Guardrails precede code. Framework enforced before first module - written prevents drift accumulation. -- Mandatory subtraction: every release must remove something. -- Recurring evaluation sprints to catch bloat early. - ---- - -## Lessons from projectMM v2 - -Extracted from design review session with Gemini (2026-05-18). - -### The "Architecture Astronaut" trap -- v2's "maximize minimalism" philosophy led to over-engineering minimal - abstractions. 19 sprints for release 1 is a red flag — the agent was - caught in a loop of refactoring abstractions instead of shipping - concrete features. -- When agents are told to be "minimal," they write highly abstract, - overly clever code (template metaprogramming, deep trait trees) to - keep things "elegant." On embedded systems, minimalism means flat, - simple, predictable code — not abstract code. - -### The MoonModule base class dilemma -- The generic MoonModule lifecycle (setup, loop, loop20ms, loop1s, - teardown) tried to be an OS-level framework before the domain logic - (LED pipeline) was working. -- Giving an agent a highly abstract concept like "a generic lifecycle - module that handles UI, WiFi, and LED drivers simultaneously" - generates massive boilerplate, interfaces, and registries. -- Solution: build concrete features first, extract common patterns - later. "Concrete first, abstract later." - -### Uncontrolled creation -- If you let an agent write code, tests, and build scripts without - strict guardrails, it will solve every new problem by creating more - code. It doesn't care about technical debt because it doesn't have - to maintain it. -- The "Refactor First" rule: when adding a feature, prefer extending - an existing module over creating a new file. Deletion is preferred - over duplication. - -### Sprint file accumulation -- Using release/sprint files as "AI short-term memory" backfired. - When the agent reads its own past sprint text, it treats abandoned - or over-engineered ideas as strict instructions. Outdated - documentation wastes context window. -- Solution: single source of truth (CLAUDE.md + architecture.md). - Git history is the log. Active workspace stays clean. - -### Context drift and compilation loops -- When the agent hits a compilation error, it tries to rewrite - multiple files to fix it, bloating the context window. If it fails - twice, it's stuck in a context loop. -- Solution: rollback with Git immediately. Don't let the agent keep - trying. Approach the bug with a different, tighter prompt. - -### CLAUDE.md as "Constitution" -- CLAUDE.md must be written BEFORE any code. It's the project's - constitution. If it's weak or vague, the agent exploits the - loopholes and writes messy code. -- Must include: exact compiler flags, memory rules, error handling - limits, style guide, platform boundaries. -- Anti-stalling clause: "If a build error takes more than 2 attempts - to fix, STOP and ask the human." - -### /plan mode is essential -- Must use /plan before every feature. It's the design review phase. -- Red flags to watch for in plans: adding dependencies when a small - function would do, introducing inheritance when structs suffice, - modifying files outside the relevant directory. -- Reject bad plan steps explicitly. Only approve when the plan is - minimal and precise. - -### Git as the undo button -- Commit after every small successful feature. Be ruthless with - rollbacks. -- The agent has direct file system access. If it panics on a build - error, it writes 500 lines of messy fix-up code. `git reset --hard` - is faster than debugging the mess. - -### Build system -- PlatformIO vs CMake: PlatformIO handles toolchains well for ESP32 - but its `native` desktop support is limited. CMake is ESP-IDF's - native build system. Dual-purpose CMakeLists.txt that works for - both desktop and idf.py is the cleanest approach, but risks - conditional pollution. Separate entry points (root CMake for - desktop, esp32/ wrapper for IDF) is pragmatic. - -### Memory and PSRAM -- Banning PSRAM entirely is wrong for 10K+ LEDs. A single RGB frame - is 30KB. With double-buffering and LUTs, you exceed 320KB internal - SRAM easily. -- OPI PSRAM (80-120MHz, 16-bit bus) has sufficient bandwidth for - sequential light data streaming. -- The rule: PSRAM allowed for large contiguous bulk allocations at - boot. Never small scattered allocations in loops. - -### ArtNet packet pacing -- Blasting 97 UDP packets per frame with zero delay causes receiver - packet drops. Inter-packet delays (50us) and FPS limiting are - required. -- The absence of pacing produced symptoms that looked like rendering - bugs (missing lights, random output) when the actual issue was - network flooding. - -### Proven patterns from v2 (harvest for v3) - -- **DataBuffer — lock-free single-slot SPSC.** Atomic revision counter with acquire/release semantics. Zero branches on hot path. Producer calls `acquire_write()` + `publish()` (one atomic store). Consumer calls `try_acquire_read()` (two atomic loads, nullptr if no new frame). Multiple consumers each track their own read position. Teardown-safe via invalidate() sentinel. -- **DataRegistry — type-erased buffer directory.** Producers declare buffers by id, consumers resolve by id. Hot-path cost: zero (consumers cache the pointer). Domain-neutral: stores void* + count + elem_size + dimensions. -- **onBuildControls() / onAllocateMemory() separation.** Controls registered in onBuildControls() (supports rebuild via clearControls()), memory allocated in onAllocateMemory() (sets moduleAllocBytes_ — single source of truth). Better than v1's "do everything in setup()". -- **PAL split into one file per concern.** PalHeap, PalRtos, PalUdp, PalWifi, PalFs, PalHttp, PalWs, PalSystemInfo. Each has ESP32 and PC implementations. Clean, focused, testable. -- **PixelEffectBase — shared effect spine.** Eliminates ~70 lines of boilerplate per effect. Concrete effect implements only `build_effect_controls()` + `render_(px, w, h, d)`. Base handles layout resolution, buffer management, teardown safety. -- **Multi-core scheduler with per-module core affinity.** Effects on core 0, drivers on core 1. Module declares `coreAffinity()` in constructor. Scheduler pins tasks via PalRtos. -- **Canvas view (UI).** v2 introduced a node-graph canvas view alongside the traditional tree view. Modules shown as draggable nodes with SVG connection lines. Powerful for understanding pipeline topology. Note: introduces significant UI complexity — adopt carefully in v3. -- **AutoWireSpec — declarative input wiring.** Modules declare dependencies as data (inputKey, searchType, allMatches, backKey). ModuleManager auto-wires at startup. Eliminates manual strcmp chains. -- **Footprint reporting — zero boilerplate.** classSize set once at registration via `register_type<T>()`. No CRTP, no macro. dynamicMemorySize() = moduleAllocBytes_ + framework overhead. -- **Field order optimized for padding.** MoonModule fields ordered 8B → 4B → 2B → 1B, saving 24 bytes vs naive order. Matters when many modules are loaded. -- **Separate frontend files.** index.html + app.js + style.css (not one monolithic HTML). Easier to maintain. - -### Map-on-the-fly vs separate Map stage -- A separate Map stage that copies between logical and physical - buffers doubles memory usage. -- Map-on-the-fly (applying the LUT during generation or blend) avoids - intermediate buffers and halves memory. -- But: blend+map writes to arbitrary physical positions (not - sequential), so the output buffer must be fully populated before - any driver reads from it. - ---- - -## Lessons from projectMM v3 - -### Product owner as critical success factor - -The single biggest improvement in v3's approach: the human is an active, hands-on product owner — not a passive requester. In v1 and v2, the agent had significant autonomy: it designed, implemented, tested, and committed with light oversight. The result was bloat, architectural drift, compounding bugs, and code the human couldn't fully understand. - -In v3, the product owner: -- Reviews every line of generated code before committing -- Specifies requirements in detail — the agent asks, it doesn't guess -- Controls all git operations (staging, committing, pushing) -- Tests on real hardware before approving -- Questions design choices ("why static_cast here?", "is this future-proof?", "do we need this?") -- Catches overengineering early ("that's too much code for something we might change later") -- Rejects suggestions that add complexity without clear value - -This is the fundamental lesson: in agentic coding, the agent writes code but the human must think. The agent is a tool, not a decision-maker. Tight human control produces cleaner, simpler, more predictable systems than giving the agent autonomy. - -### Specs-before-code works - -Writing module specs before implementation prevented the architectural drift that plagued v1 and v2. Each spec documents: purpose, controls, behavior, edge cases, prior art. When the code deviates from the spec, one of them is wrong — the spec serves as the reference. One source of truth per module: the spec lives in `docs/moonmodules/`, deleted nowhere else. - -### Zero-copy preview driver (memory lesson) - -The initial PreviewDriver allocated a 49KB frame buffer to copy pixel data before sending via WebSocket. Following MoonLight's pattern (drivers read directly from the physical buffer), we eliminated the copy — saving 49KB on ESP32 without PSRAM. The lesson: always check if an existing buffer can be reused before allocating a new one. - -### WebSocket GUID typo cost hours - -A single wrong character in the RFC 6455 magic GUID (`5AB5FDF632E5` instead of `C5AB0DC85B11`) caused the WebSocket handshake to fail silently — the SHA-1 was correct, the response format was correct, but the browser rejected it. The accept key matched our computation but not the browser's because the GUID was wrong. Lesson: when implementing protocols, verify against the RFC test vectors, not just internal consistency. - -### KPI tracking in every commit - -Adding standardized KPIs (binary size, FPS, heap usage, test count, lizard warnings) to every commit message makes performance regressions visible in git history. The `collect_kpi.py --commit` script automates this. - -## Lessons from projectMM v3 (this project) - -### Single-file MoonModules are good -- Keeping each MoonModule in a single .h file (no .cpp) reduces - file count and makes authoring easy. A developer creates one file - and implements the interface. This pattern should be kept. - -### Controls work well -- The fixed-capacity control array with typed values (uint16, bool, - text) and auto-rendering in the UI is a good pattern. Adding a - control to a MoonModule automatically makes it visible and - editable. No UI code changes needed. - -### The plan.md pattern works -- Having a plan.md that shrinks as steps are completed gives clear - progress visibility. Removing completed steps keeps the document - focused on what's next. - -### Generic children eliminate boilerplate -- Moving children array + addChild/removeChild to MoonModule base eliminated ~120 lines of duplicated code across Layer, DriverGroup, LayoutGroup. The typed arrays (effects_, modifiers_, drivers_, layouts_) and typed add methods were unnecessary — role() distinguishes child types at the call site. - -### Dynamic arrays over fixed-size -- Replacing fixed arrays (MAX_CHILDREN=8, ControlList<8>) with grow-on-demand eliminated arbitrary limits. Leaf modules pay zero cost (nullptr). No hot-path impact — allocation happens during setup only. - -### classSize via template, not per-class override -- ModuleFactory::registerType<T>() captures sizeof(T) automatically. Eliminated 10 boilerplate `classSize() const override` lines. Same pattern as v2's register_type. - -### Naming matters -- `isOneToOne()` was misleading — "1:1" includes both sequential and shuffled. Renamed to `hasLUT()` (asks the real question: is there a table?). `setIdentity()` for the no-table case. Four mapping types defined: 1:1 identical, 1:1 shuffled, 1:0 unmapped, 1:N multimap. -- Semantic variable names (`availableHeap` not `available`, `internalHeap` not `internal`) — a reader should understand without looking at the assignment. - -### Per-module timing reveals bottlenecks -- tickTimeUs as primary metric (FPS derived) exposed that ArtNet takes 51% of frame time on ESP32 128x128. Without per-module timing this would be invisible. -- Memory reporting (classSize + dynamicBytes per module) revealed the LUT over-allocation: maxMultiplier() on MirrorModifier saved 64KB by using 4x instead of hardcoded 8x. - -### Live scenarios must be non-destructive -- Initial implementation created modules on the running device without cleanup — the device ended up slower with extra effects. Fix: track created modules, delete them after each scenario. Show `=` for existing vs `+` for new. - -### freeHeap vs freeInternalHeap -- On ESP32 with PSRAM, freeHeap() returns combined (internal + PSRAM). The HEAP_RESERVE check must use freeInternalHeap() because stack/HTTP/WiFi need internal RAM, not PSRAM. On desktop both return 0 (unlimited). - -### Rule of Five catches real bugs -- MoonModule and ControlList owned raw pointers but had implicit copy/move — CodeRabbit caught the double-free risk. Delete copy/move on any class that owns raw memory. - -### setName must copy, not store pointer -- HTTP module creation stored a pointer to a stack-local buffer. After the function returned, the name was garbage. Fixed by making name_ a char[24] buffer with memcpy in setName(). - -## Lessons from the next-iteration branch (plans 08-12) - -The branch covering SystemModule/NetworkModule, persistence, the UI rewrite, the eth-only build, and the side-nav. Brief, actionable takeaways: - -- **Plan-09 was right to abandon part of itself.** The first persistence attempt (~1700 LOC) didn't pay for itself; ~700 LOC of genuine foundations (partition scheme, platform fs API, MoonModule additions) were kept and the rest dropped. Plan-10 then succeeded with a far smaller control-list-driven design. Lesson: a plan that produces "keep the foundations, drop the feature, re-plan smaller" is a success, not a failure. -- **ESP-IDF v6.x removes WiFi via `EXCLUDE_COMPONENTS`, not a Kconfig flag.** `CONFIG_ESP_WIFI_ENABLED` is non-settable in v6.x — setting it `n` is silently ignored. The eth-only profile excludes `esp_wifi`/`wpa_supplicant`/`esp_coex` components (NOT `esp_phy` — the EMAC needs it) plus an `MM_NO_WIFI` define gating `if constexpr` branches in core. Verify a build profile actually changed something (image size, `nm` for symbols), don't trust the config. -- **The render tick collapses on any blocking network write on the hot path.** The FPS-swing was a blocking 49 KB preview WebSocket write spinning `vTaskDelay` until lwIP drained. Fix: non-blocking scatter-gather write + downsample the payload to fit the send buffer. Any per-tick socket write must be non-blocking and abandon-on-backpressure. -- **WiFi UDP is ~4× the Ethernet per-packet cost.** ArtNet at 16K lights is ~7 FPS on WiFi vs ~19 on Ethernet — WiFi physics (CSMA/CA, retries, rate adaptation), not a code regression. Recommend Ethernet (or the eth-only profile) for large installations. -- **No-op wrappers are an "unnecessary abstraction" the Reviewer must catch.** Two were found and removed (`HttpServerModule::parseJsonString` re-namespacing `mm::json::*`; `NetworkModule::rebuildLocalControlsAndPipeline` whose name contradicted its body). A pure pass-through that only renames is the opposite of duplication — a single thing that shouldn't exist. CLAUDE.md's Reviewer definition now names this pattern explicitly. -- **Always escape strings going into hand-built JSON.** A control value containing `"` or `\` produced malformed JSON in `/api/state` and the persisted config. Hand-rolled JSON serialization must escape on write and un-escape on read — caught by a round-trip test, not the reviewer. -- **Doc-vs-code drift hides in design changes.** When the password approach changed mid-implementation (length-only → XOR+base64), the docs and one header were updated but a second header's comment was missed — it then documented a security property the code didn't provide. When a design changes, grep for every mention of the old behaviour. - -## Lessons from this branch (plans 13-16) - -Plan-13 (nest child cards inside parent box), plan-14 (replace-type button), plan-15 (stream `/api/state`), plan-16 (Layouts/Layers/Drivers top-level reshape) — plus the unplanned mid-branch work the user surfaced (effect-animation freeze, Int16 zero-corruption, Layouts container disable crash, MoonModule status slot, src/light/layers/ reorganisation, FilesystemModule + Scheduler split). - -- **A single `c.min`/`c.max` field can't bound multiple numeric widths.** `ControlDescriptor.min/max` are `uint8_t`; applying them to `Int16` / `Uint16` controls clamps every value into `[0..0]`. Default `addInt16` and `addUint16` leave bounds at `0,0` precisely because the slot can't represent the wider range — and on the load path that mistake silently zeros every Int16 control on every reboot. The bug shipped, was caught next session by "Layouts cannot be activated after reboot," and was reverted with explicit comments in `Control.h` documenting why bounds stay 0,0. Lesson: when a validation field's storage is narrower than what it claims to validate, the validation is wrong, not the field. A type-correct alternative (wider bounds, or per-type bound slots) is the real fix; until then, document the constraint at the field's declaration. - -- **Per-tick integer division can round the animation rate to zero.** `phase += dt * bpm * 256 / 60000` truncates to 0 when `dt < 234 / bpm` ms. ESP32 at 16K LEDs (`dt ≈ 62ms`) animates fine; desktop at `dt ≈ 0..1ms` and small grids on ESP32 freeze. Four effects shipped frozen on desktop for an entire iteration before a user noticed. Fix is to keep the raw `dt * bpm` numerator in the accumulator and divide only at the read site (the pattern NoiseEffect already used). Rule, now in CLAUDE.md § Hard Rules: "effects must run at every grid size and tick rate." - -- **Early-return on degenerate inputs leaks state, not safety.** `Layer::onAllocateMemory` early-returned on empty layouts (zero lights) without resetting its LUT or buffer; Drivers then reallocated its output buffer to 0 bytes while the stale LUT still pointed at 16K destinations, and the next `blendMap` dereferenced null. "Nothing to do" branches must still bring the module into a consistent zero state (dims = 0, LUT freed, buffer freed), not just skip work. Same lens as the zero-grid effect rule. - -- **HTML5 `dragstart`'s `e.target` is always the draggable element, not the deepest descendant under the mouse.** Excluding regions of a draggable element via `e.target.closest(".child-class")` in `dragstart` never matches because `e.target === card`. The reliable signal is the *mousedown* target; toggle `draggable` on mousedown based on where the grab actually landed, with `touchstart` mirror for mobile. The naive `dragstart` filter pattern shipped silently because the existing exclusion list happened to include `<input>` (covered most controls); only when the controls region used a `<details>`/`<summary>` did the bug surface. - -- **Severity is a real axis on the "module has something to say" channel.** A single `warning` slot conflates "this is fine, here's your IP" with "this degraded silently, look closer" with "this failed." Three levels (`Status` / `Warning` / `Error`) earn their keep when more than one module produces non-degradation messages (NetworkModule routes its old ReadOnly `status` control through the same slot). Default = `Status` (neutral info) so `setStatus("connected")` looks right; severity is only set explicitly when something is bad. Wire format mirrors the C++ enum names lowercased (`"status"` / `"warning"` / `"error"`) — collision with the field name `status` is documented at the introduction site so the bespoke choice carries its reason. - -- **Lifecycle events fall into "commit" and "merge"; "push" had no work of its own.** A first cut of the lifecycle gates carved out a separate Push event for the Reviewer agent — but that produced the "address-reviewer" follow-up-commit anti-pattern (reviewer flags an issue → noise commit → trail full of "fix reviewer"). Moving the reviewer to Commit was worse: every substantial commit paid 5-7 min + tokens. Final shape: reviewer at PR-merge over the whole branch diff (where architectural drift is visible across commits, not within one); on-demand pre-commit as the safety valve when the product owner asks. Push has no gate. - -- **The reviewer agent flags real findings AND wrong ones; triage matters.** Three different reviewer passes over this branch produced ~20 findings. ~30% were valid, the rest either misread line numbers (cited the wrong region), or repeated a finding already accepted in an earlier commit, or proposed a "fix" that would re-introduce a bug we just fixed (the `c.min`/`c.max` Int16 finding twice). When a reviewer finding is wrong, "skip with one-line reason in the commit body" is more honest than either fixing it to satisfy the reviewer or rejecting it silently. The reason text becomes the audit trail. - -- **The docs hierarchy now has a name shape: top-level system docs are flat (`architecture.md`, `coding-standards.md`, `building.md`, `testing.md`), per-module specs live under `docs/moonmodules/<role>/<Name>.md`.** The earlier `architecture.md` + `architecture-light.md` pair was asymmetric ("the general doc + the light-domain deepening") and pulled toward growing more suffixes. Merging back into one `architecture.md` with `# Core` / `# Light domain` top-level sections matched the convention every well-known C++/web project uses (Linux, Django, Rust). Subdirectories `core/` and `light/` only kick in under `moonmodules/` because there's a plural of each kind there. Bespoke `*-light.md` / `*-coding.md` suffix patterns are now ruled out by the "common patterns first" principle. - -- **`util/` is a category that classifies by "what is this thing" rather than "what does it do" — and it's the kind a new contributor learns nothing from.** When `src/core/` grew to 17 files I considered a `util/` bucket for the small headers (`Base64.h`, `Sha1.h`, `JsonSink.h`, etc.) and rejected it: each file already names what it does (it's not "util", it's WebSocket-handshake base64 / hash); grouping by file-shape rather than concern is the Frankenstein pattern. Same reason `modules/` got rejected: the four system services (Filesystem / Network / System / HttpServer) are singletons of their kind, not "modules of role X" — folder-grouping requires a plural of each kind to earn its keep. The right cleanup turned out to be header-only → `.h+.cpp` splits (done lazily as files were touched), not folders. - -- **`.claude/scheduled_tasks.lock` is harness runtime state.** Per-session lock file the Claude Code harness uses to coordinate scheduled wakeups; not project content. Add `.claude/*.lock` to `.gitignore` (a broader pattern than per-file, since other harness lock files will follow the same shape). The single-file ignore on `.claude/settings.local.json` was too narrow. - -## Lessons from this branch (plans 17-23) - -The plan-18 branch landed plans 17 + 18 + six unplanned follow-ups (19, 19.1, 20, 21, 22, 23). What earned its keep, what surprised us, what to remember. - -- **CORS-on-static-files isn't a thing GitHub Pages can fix from your side.** Plan-17's web installer assumed cross-origin fetches of GitHub release-asset `.bin` files would work; they don't (the GitHub Releases CDN returns no `Access-Control-Allow-Origin`). Plan-18 step 0 falsified this empirically and pivoted to "self-host the last N releases' binaries on Pages content" — same-origin with the install page, no CORS. The lesson sits both ways: don't assume CORS shape; AND don't fix CORS via a third-party proxy (WLED's `proxy.corsfix.com` dependency) when self-hosting works. - -- **HTTPS-page → HTTP-LAN fetches are blocked by Chrome's mixed-content policy, no override available cleanly.** A Pages-hosted install page can't `fetch("http://192.168.1.X/api/state")` even if the device has `Access-Control-Allow-Origin: *` set correctly. The block happens before the request leaves the browser. Plan-20's Diagnose feature moved to the device UI (same-origin, mixed-content moot) instead. Lesson: pick the side of the security boundary that actually works; trying to "fix" mixed-content from the device side is wasted code. - -- **ESP Web Tools' rich panel is in-browser-session-only.** "Visit Device" + "Configure Wi-Fi" rows appear right after provisioning; close the tab and the panel collapses to "Install + Logs" because the device URL is browser-side memory, not asked-back from the device. Plan-18 fix-pack added a `GET_CURRENT_STATE` → URL follow-up on the device (ESPHome pattern, mirrors what Improv allows) — verifiable via `improv_probe.py`, but it does NOT change the ESP Web Tools UI. Lesson: when a feature lives in a third-party tool's state machine, surfacing data from the device is necessary but not sufficient; the third-party tool's own state model also has to use it. - -- **`improv_provision` rejects new credentials when WiFi STA is already connected — that's by design.** The Improv listener returns `ERROR_UNABLE_TO_CONNECT` if `wifiStaConnected()` at the time of `WIFI_SETTINGS` — protects large installs from a scan-induced ArtNet drop. The browser dialog says "Unknown error (255)" though, because Improv's error code mapping doesn't carry a human reason. Lesson: protocol-level "this is fine, expected" and browser-level "something is wrong" don't line up automatically. Document the rejection at every layer the user encounters it. - -- **Per-board build directories should land as `build/<board>/`, not `<chip>/build/<board>/`.** Plan-19.1's choice of `build/esp32-<board>/` (e.g. `build/esp32-esp32-eth-wifi/`) keeps every target under one root (`build/`) and shares the namespace with desktop targets (`build/macos/`, `build/linux/`, `build/windows/`). The doubled prefix on the ESP32 side (`esp32-` for the chip family + `<board>` for the variant) is intentional — earlier draft put ESP32 builds under `esp32/build/<board>/`, which made `clean --all` more fragile because it had to walk both `esp32/build/` and `build/`. One root, one cleaner. - -- **`idf.py -B <dir>` needs `-DSDKCONFIG=<dir>/sdkconfig` to keep per-build-dir sdkconfigs isolated.** Without the SDKCONFIG override, idf.py writes `<project>/sdkconfig` at the project root, which all per-board build dirs then share — switching boards trips the "project sdkconfig was generated for target X, but CMakeCache contains Y" abort. The fix is a one-liner per script that invokes idf.py, but it's not obvious until the failure happens on the second board build. Caught by the per-board build sanity-test step of plan-19.1, retrofitted to build_esp32 / flash_esp32 / collect_kpi. - -- **Trying to add a child module to NetworkModule wasn't the small refactor it looked like.** Plan-21 (Improv as Network child) reverted the same session. The blocker looked like `Scheduler::tick()` only walking **top-level** modules for `loop20ms` / `loop1s`, so children never got tick callbacks. Lesson: "obviously the right shape" can hide infrastructure that isn't ready for that shape. **Resolved by the Peripheral-role branch:** the real gap was narrower than "the scheduler doesn't tick children" — `MoonModule`'s base lifecycle *does* propagate every callback (`loop`/`loop20ms`/`loop1s`/`setup`/`teardown`/`onBuild*`) to children; a child only misses a callback when the parent **overrides that method and forgets to chain to base**. The fix is therefore per-parent, not a scheduler refactor: SystemModule (which accepts user-added Peripheral children) overrides `setup()` and `loop1s()` and now chains both to `MoonModule::`, so peripheral children init and poll. `unit_SystemModule` pins this. The general rule — override-and-chain, with the parent-before/child-before convention per callback — lives in [coding-standards.md § Override-and-chain convention](../coding-standards.md#override-and-chain-convention). - -- **Splitting `platform_esp32.cpp` is safe at public-API boundaries, not at section banners.** Plan-23 cut Improv + OTA + LittleFS into siblings because each owns its private state and talks back to the rest of the platform layer only through `platform.h`. Network stayed in the core file because Eth + WiFi + sockets + mDNS share eight file-scope variables (event handlers, netif pointers, init-done flags). The criterion: a split happens at a public-API boundary, not at a "this section reads coherent in the diff" boundary. - -- **Desktop's platform.cpp is correctly asymmetric with ESP32's.** Plan-23 deliberately didn't split `platform_desktop.cpp` even though it has matching OTA / Improv / FS sections, because each is a 6-line stub. Per-subsystem stub files would be all overhead, no payoff. Lesson: symmetry across platforms is a heuristic, not a rule; the right axis is "does each file pay for itself." - -- **Nightly builds belong in their own workflow, not the main release workflow.** Plan-22 added `.github/workflows/nightly.yml` that tags `nightly-YYYY-MM-DD` and lets the existing `release.yml` do the actual build via tag push. Zero duplication of the build matrix or Pages staging logic. The skip-on-no-change check (`should-tag=false` when `main` HEAD hasn't moved since the last nightly tag) means quiet days cost ~2s of API calls, not a full build. Lesson: a scheduled trigger that tags-and-reuses an existing trigger is cleaner than a parallel workflow that duplicates build steps. - -- **`workflow_dispatch` reads the workflow YAML from the default branch, not the dispatched branch.** Cost us a CI cycle on RC2: dispatched against `plan-18` with a tag that wasn't valid against `main`'s older `release.yml`. The fix landed in the same branch (`3e2acb5` — branch allowlist + verify-version `--tag` arg), but the lesson is to read the GitHub Actions semantics for dispatch carefully: `inputs.tag` arrives correctly, but the workflow logic that consumes it is whatever main has at dispatch time. - -- **The reviewer agent's job at PR-merge is architectural drift across N commits, not line-level bugs.** CodeRabbit catches line-level bugs in each PR commit; the reviewer agent at Event-2 reads the full branch diff and asks "did three commits each add a wrapper that one commit would hide?". Plan-18 ran into this productively — the in-branch CodeRabbit findings got triaged and either fixed or skipped-with-reason; the reviewer-agent run at merge is the final architectural sanity check. Two agents, two scopes. - -- **A 13-commit branch is the upper end of what a single merge should carry.** Plans 18 + 19 + 19.1 + 20 + 20.1 + 21 (reverted) + 22 + 23 in one branch is a lot — the merge commit train is heavy and the reviewer-agent's job gets harder as more commits stack up. Future branches should aim for "ship the first 3-4 plans, merge, start the next branch" rather than "let the branch grow until everything's tidy." This branch worked because the plans were mostly independent (no two of them touched the same files in conflicting ways), but that won't always hold. - -- **Asymmetric lifecycle propagation in MoonModule was historical, not principled.** Before this change, `setup` / `teardown` / `onBuildControls` / `onAllocateMemory` propagated to children via base defaults, but `loop` / `loop20ms` / `loop1s` defaulted to empty no-ops — every container that wanted to tick its children wrote the same 5-line per-child block (Layers, Drivers, would-be-NetworkModule-with-Improv). The fix was a one-line base-default change per callback into a shared `tickChildren` helper that gates by `!respectsEnabled() || enabled()` (children that opted out of the enabled gate keep ticking; the rest tick only when enabled) and accumulates per-child timing the same way Scheduler does for top-level modules. Leaf modules with `childCount_ == 0` pay one predicted-not-taken branch per call — sub-nanosecond. PC tick stayed in the same 55-160 µs band across scenarios; no measurable regression. Lesson: when one half of a lifecycle propagates and the other half doesn't, the asymmetry is usually historical (no one wrote the helper yet), not principled. With the helper in place, the parked Plan-21 (Improv-as-Network-child) move was a four-line follow-up: `addChild` instead of `addModule`, plus chain `NetworkModule::setup` / `loop1s` / `teardown` / `onBuildControls` to the base. - -- **Override-and-chain convention: option A for loop, option B for setup.** When a container needs custom work alongside child dispatch, the convention is "parent prepares, children consume" for the loop callbacks — parent runs its work first, then chains to `MoonModule::loop()` so children read the freshly-prepared state (`Drivers::loop` runs `blendMap` before driver children read `outputBuffer_`). For `setup()` the convention is the opposite — chain to base first so children are initialised before the parent depends on them. `teardown` chains late (parent shuts down its own state, then the base reverse-iterates children). The conventions are explicit in `architecture.md § Lifecycle propagation to children`; deviations need a one-line comment at the override. - -- **Control-change reactions are a three-tier split; the coarse-grained rebuild debt is closed.** The long-standing debt ("the build pass ran on all top-level modules for every control change") is resolved. `handleSetControl` now: (1) always calls the cheap `MoonModule::onUpdate(controlName)`; (2) calls `scheduler_->buildState()` only when `controlChangeTriggersBuildState(controlName)` returns true; (3) the pass reaches each module's `onBuildState()`. `controlChangeTriggersBuildState` defaults false and is overridden to true on `LayoutBase` and `ModifierBase` (every control they expose changes physical dims / LUT shape). Effect/driver value controls — including the new Drivers `brightness` — take tier 1 only, so slider drag no longer triggers a tree-wide realloc sweep. The model mirrors MoonLight's `onUpdate` / `requestMappings` (`hasOnLayout`/`hasModifier`) / `onSizeChanged` split — confirmed against MoonLight source before implementing. The verb is "build" not "rebuild" (idempotent, history-agnostic) and `onAllocateMemory` was renamed `onBuildState` for the same reason — boot and a later change are the same call. Lesson: when the spec already describes selective behaviour (architecture.md § Rebuild propagation said this for months) but the code does the coarse thing, the gap is usually a missing cheap hook, not a missing design. - -- **Output correction (brightness/reorder/white) is a per-driver stage, shared via the Drivers container, not a WS2812-specific or effect-side concern.** Every physical driver needs to turn logical RGB into a physical signal; ArtNet was sending raw bytes (no brightness/order/white) — a gap, now fixed. The Drivers container owns a `Correction` (256-entry brightness LUT + channel-order table + derive-white flag) and hands each child a `const Correction*`; each physical driver applies it per-light into its own output buffer/packet. Preview is exempt (raw logical buffer). Brightness applied *before* white derivation (white = min of scaled channels). One LUT now; gamma/white-balance fold in later as a per-channel R/G/B split (the field is named `briLut` not `gammaLut` so that's a fill change, not a rename). Following MoonLight, which applies brightness/gamma/color-order at the driver edge via per-channel `redMap/greenMap/blueMap/whiteMap` LUTs + a lightPreset→offset table. - -- **Per-`ControlType` behaviour belongs with the type, not with the caller.** Three call sites (`HttpServerModule::writeControls`, `FilesystemModule::writeValue`, `scenario_runner.cpp::writeJsonValAsValue` and twin apply paths) were each carrying their own `switch (c.type)` over the same enum. Each was 50-60 lines, hand-maintained in parallel, and silently drifted (the scenario runner had stopped recognising new types added for the FS path). The fix was to extract the switch into a small set of free functions in `src/core/Control.cpp` (`writeControlValue`, `writeControlMetadata`, `applyControlValue`, `isPersistable`, `hasDefault`, `controlTypeName`) and route the three call sites through them. Two cross-cutting requirements made this work: (1) `JsonSink` gained a third "fixed-buffer" mode (alongside socket and heap-grow) so the FS path could share the serializer without growing a per-value allocation, and (2) the apply path got an `ApplyPolicy` parameter (`Strict` for HTTP, `Clamp` for FS load) so the tolerant-load semantics survived the consolidation. Codified in `docs/coding-standards.md § Per-type behaviour lives with the type` — applies whenever the same `switch` appears in 2+ places. Counter-example also in that section: if only one caller needs the behaviour, keep it at the call site (a one-shot switch is cheaper than a function with one user). - -- **Local Improv testing closes the dev-loop gap that made every Improv change a high-friction commit.** Before this branch, the only way to verify the Improv flow end-to-end was to tag a release, wait for CI, deploy to GitHub Pages, then flash from the live web installer — each iteration of the docs/install page or release-picker.js took minutes and burned a release tag. `moondeck/run/preview_installer.py` now has a "flash-ready" mode that stages every local `build/esp32-*/projectMM.bin` under `releases/latest/` in the preview server's tree, generates matching Pages-relative manifests via the same `generate_manifest.py` production uses, and serves the install page at `localhost:8000`. The picker resolves the GitHub `latest` tag to the staged local bins via `toLocalUrl`, and Web Serial works on `localhost` without the secure-origin requirement that gates the public site — so an end-to-end flash + Improv WiFi provision is verifiable without touching any release. Paired with `moondeck/build/improv_smoke_test.py` (probe + WiFi provision + LAN reachability, all three steps as a single CLI), which gives a deterministic pass/fail for the device-side Improv state machine and is wired into MoonDeck. Lesson: when a feature can only be tested in production, the iteration cost compounds into "don't touch it" — building the local test loop, even if it takes a day, pays for itself within a few commits. - -## Lessons from this branch (Board injection follow-ups) - -- **`src/ui/release-picker.js` is now `src/ui/install-picker.js` (exported symbol `installPicker`, embedded C array `installPickerJs`).** Renamed once the picker grew from "pick a release" to "pick release + board + firmware + install button" — the old name only described one of the three axes. The rename was a wide sweep (~20 files: source + CMake + release.yml + preview script + spec doc + every comment that mentioned the picker by name) but mechanically simple. Recorded here so a future search for `release-picker.js` lands on this entry and finds the new name; the file itself doesn't carry a "renamed from" comment because that would be the kind of past-tense history `CLAUDE.md § Principles` says belongs only in this folder. - -- **Why the web installer dropped ESP Web Tools for a custom orchestrator.** ESP Web Tools 10.x's `<esp-web-install-button>` held the SerialPort exclusively across its flash + provision lifecycle and fired its `state-changed` event inside its dialog's shadow DOM (verified by reading `esp-web-tools/src/install-dialog.ts`). Two consequences: post-PROVISIONED board injection from the installer page was structurally impossible (we couldn't hook a side-call between provision and reboot), and `devices.js`'s "Your devices" auto-add silently broke because the URL was emitted behind a shadow-root event we couldn't subscribe to from outside. Owning the SerialPort end-to-end in `install-orchestrator.js` lets both fixes land in one place. The same dispatcher seeds future per-control injectables (device name override, MQTT broker URL, DMX universe) — each new field adds one vendor command ID + one dispatcher case. Lesson: a third-party install widget that owns the transport and emits lifecycle events behind a shadow root is a dead end for any flow that needs to do work after provision; if you need composability, own the transport. - -## Adaptive memory allocation design (plan-07) - -The core rules for how the light pipeline allocates and degrades under memory pressure. These are the invariants the code was designed around; the implementation lives in `Layer.h` and `DriverGroup.h`. - -**Allocation rules:** -- **MappingLUT** is created only when ALL are true: modifiers exist on the layer; layout is not a simple non-serpentine grid (where physical == logical); enough heap available after reserving `HEAP_RESERVE` (32 KB) for stack/HTTP/WiFi. -- **Driver output buffer** is created only when: at least one layer has a LUT actually allocated (not just "has modifiers") and enough heap is available. -- Result for 1:1 unshuffled (no modifiers, or grid without serpentine): zero intermediate buffers — ArtNet reads directly from the layer buffer. Maximum LED count at minimum memory. - -**Degradation cascade** — when memory is insufficient, degrade in this order: -1. Full pipeline — LUT + driver output buffer (modifier applied, clean separation) -2. Skip driver output buffer — LUT exists, DriverGroup does mapping inline (slower, sequential) -3. Skip LUT — modifier not applied, forced 1:1 mapping -4. Reduce layer dimensions — halve until buffer fits, minimum 8×8 - -Each degradation level is observable via flags on the module (`degraded()`, `lutSkipped()`, `outputBufferSkipped()`). - -**Predict-measure-compare:** before each allocation, predict memory impact from grid dimensions + channelsPerLight + modifier presence; after allocation, compare heap delta. Variance > 5% signals a leak or accounting error. Buffer sizes: layer = W×H×D×cpl; LUT ≈ `MappingLUT::estimateBytes(logicalCount, maxDest)`; driver buffer = physicalCount×cpl. - -## Plan-09 persistence failure — why JSON didn't pay for itself - -Plan-09 attempted ~1700 LOC of JSON-based persistence that was fully abandoned. The five root causes are worth keeping because they recur in any persistence design: - -1. **Question the format premise.** "Persistence is JSON" was assumed without justification. Neither human-readability nor manual editability were real requirements. For POD-only module state, `memcpy(file, this + sizeof(MoonModule), classSize - sizeof(MoonModule))` is one line and a complete save. Plan-10 took this path and succeeded. - -2. **Suspicious helper proliferation signals over-elaborate design.** The plan spawned: `rebuildControls`, `clearControlsRecursive`, `LoadAllFn`, `setLoadAllHook`, `noteDirty`, `loadAll`, `loadTopLevel`, `applyNode`, `applyControls`, `serializeNode`, `serializeControls`, `buildTopLevelPath`, `cleanupTmpFiles_`, `cleanupTmpCb_`, `cleanupTmpLeafCb_`. That list is the system telling you the design is too elaborate for the job. - -3. **Persistence forced a Scheduler reorder that bred secondary bugs.** Overlaying persisted values onto bound control variables grew the Scheduler from 3 phases to 5. This required `onBuildControls` to be idempotent, bred a duplicate-children bug, required SystemModule to guard MAC→deviceName derivation with a `deviceName_[0] == 0` check, and caused multiple "device shows nothing" hardware failures. The right approach: load BEFORE any module's setup or onBuildControls, by memcpy'ing into member memory directly. - -4. **Defensive guards under memory pressure mask design bugs.** The plan added 5 null guards across BlendMap, DriverGroup, and Layer to handle failure modes that fragmentation produced. Each guard was locally correct; collectively they obscured the design problem (allocate-new-before-free fragmentation). Fix the invariant, not the call site. - -5. **Test isolation reveals persistent-state contamination.** Live scenarios that mutated state (mirror toggles, grid size) contaminated each other across runs — failures appeared random until previous runs leaving state in `.config/` was identified. Any persistence layer's tests must reset state explicitly. - -## Board-injection pipeline timing constraint - -The web installer's `boards.json` catalog ships per-board control values that the orchestrator pushes to the device. SET_BOARD over Improv-Serial carries only the board name (one vendor RPC, one Text control on BoardModule); every other field in `controls.*` ships via HTTP after WiFi association. - -This split works because every per-board control we ship today applies *post-association*: `Network.txPowerSetting` (the weak-power brown-out cap), and the future Ethernet pin maps, default-config overrides, etc. The radio briefly runs at the wrong setting for the ~1 s between association and HTTP fan-out completion — acceptable for power capping, would be unacceptable for: - -- **Country code.** Governs which channels the radio scans; a wrong code at scan time picks wrong channels. -- **Antenna selector.** Wrong RF path at radio init makes the device deaf. -- **Pre-association TX-power.** Some chips need the power cap applied before the first probe request, not after association. - -If we ever add such a control, **don't extend SET_BOARD's wire format to carry it** — that would couple unrelated controls to the board-name lifecycle and obscure the timing constraint. The two escape hatches are: - -1. **Add a second vendor Improv RPC** (`SET_<control>` analogue of SET_BOARD) and dispatch it from the orchestrator BEFORE `SEND_WIFI_CREDENTIALS`. One RPC per pre-association control keeps the timing contract explicit. -2. **Bake the value into firmware** via a board-specific sdkconfig fragment. Works when the value is truly board-static (country code per region) and not user-configurable. - -Pick (1) for user-configurable controls; (2) for truly fixed-per-board values. Avoid the implicit option C ("just push it earlier in SET_BOARD") — it tangles the lifecycle. - -## Lessons from the ESP32-S3 N16R8 (DevKitC) enablement branch - -Three non-obvious failures showed up while adding native-USB S3 support, all with the same diagnostic shape ("symptom looks like X, root cause is somewhere completely different"). Record them so a future S3 / native-USB addition doesn't repeat the dig. - -1. **USB-Serial-JTAG ≠ UART0 on ESP32-S3.** The ESP32-S3 N16R8 Dev's USB-C port wires through the ESP32-S3's built-in USB-Serial-JTAG peripheral, NOT through an external USB-Serial bridge to UART0. ESP-IDF's secondary-console feature (`CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG=y`, on by default for S3) mirrors stdio out BOTH paths, so the developer sees boot logs and assumes UART0 is wired and working. It isn't. Improv-listener-on-UART0 was deaf because the host was talking to USB-Serial-JTAG. **Fix:** install BOTH drivers and read from both transports — `#if SOC_USB_SERIAL_JTAG_SUPPORTED` keeps ESP32-classic builds free of the JTAG path. Don't make this compile-time-per-board ("the DevKit firmware") — the same binary should work on a board with either wiring. - -2. **CORS preflight is silent on the client side.** Every cross-origin POST from a browser with `Content-Type: application/json` (everything the web installer does) triggers an OPTIONS preflight. If the device's HTTP server returns 405 to OPTIONS, the browser **silently drops the subsequent POST** — no error surfaces in client code, no network tab line, no console message in the install log. The symptom is "the API write didn't happen" with no diagnostic to follow. Burned an entire session diagnosing what looked like a board-injection-fan-out bug; the root cause was an unhandled HTTP verb. **Fix:** always implement OPTIONS in any device-side HTTP server that's reachable cross-origin. Return 204 with `Access-Control-Allow-Origin: *`, `Access-Control-Allow-Methods`, `Access-Control-Allow-Headers: Content-Type`. Verify with `curl -X OPTIONS -H "Origin: ..." -H "Access-Control-Request-Method: POST" -H "Access-Control-Request-Headers: content-type" http://device/api/control` — must return 204 with the headers, not 405. - -3. **Cached "last applied" state goes stale when the underlying stack restarts.** The `txPowerSetting` cap used an `appliedTxPowerSetting_` field to skip redundant `esp_wifi_set_max_tx_power` calls. When ESP-IDF stops the WiFi stack (AP→STA cascade, STA reconnect, AP shutdown), it resets the radio's TX-power state — but our cached "applied" value stayed equal to the desired value, so `syncTxPower()`'s equality check short-circuited and the cap never re-landed on the restarted radio. The user saw "the cap doesn't actually work after the first power-cycle." Classic cache-invariant bug: the cache was correct vs the *driver state we set*, but stale vs the *underlying hardware state* that an external event reset. **Fix:** every callsite that calls `wifiStaStop()` / `wifiApStop()` also invalidates the cache (`appliedTxPowerSetting_ = -1`). Generalisable: any "skip if already applied" optimisation against a hardware peripheral needs an invalidation hook tied to every event that resets the peripheral, not just the obvious explicit ones. - -## Core/light type boundary: light_types.h split + preview decouple - -`src/core/types.h` had grown into a junk-drawer of light-domain types (`nrOfLightsType`, `CoordCallback`, `defaultGridSize`, `HEAP_RESERVE`, `lengthType`) alongside `Dim`. Split it so each symbol lives with its owner. Done in two passes: - -**Pass 1 — the symbols with no core consumer:** -- **`nrOfLightsType`, `CoordCallback`, `defaultGridSize`** → `src/light/light_types.h`. -- **`HEAP_RESERVE`** → `platform.h` (it's a platform memory constraint guarding stack/HTTP/WiFi headroom — not a light type and not Layer's, even though Layer was its only caller; ownership follows concept, not call count). - -**Pass 2 — `lengthType`, by removing its core consumers rather than declaring them load-bearing:** -The apparent blocker was "core uses `lengthType`." On inspection the uses were incidental: `Control.h` only *mentioned* it in a comment; `HttpServerModule`'s `put16` only took it because it serialised `PreviewFrame`. The real tie was `PreviewFrame` itself — a light-produced struct sitting in `src/core/` that core's `HttpServerModule` read to build the preview WS frame. We severed it properly: -- Introduced `BinaryBroadcaster` (core interface, ~6 lines): "send these bytes to all WS clients." `HttpServerModule` implements it via `broadcastBinary` — the old `broadcastPreviewFrame` body minus all preview specifics. -- `PreviewFrame.h` → `src/light/`. `PreviewDriver` now owns the 13-byte header packing and **pushes** the bytes to the broadcaster; the HTTP server no longer reads `PreviewFrame` or knows the format. Push replaced the old `PreviewFrame::ready` poll (flag deleted). -- `lengthType` → `light/light_types.h`, zero core users left. - -**Result:** core has zero light dependency in the preview path; light owns the preview end to end; the binary-frame *transport* stays in core (reusable by any future binary feed — the leverage that justifies the new interface). - -**Pass 3 — `Dim`, and the deletion of `core/types.h`.** `Dim` (the effect/modifier dimensionality enum) was the last light symbol in core. It looked load-bearing: `ModuleFactory::registerType<T>` probes a type for `dimensions()` to capture the UI's 📏/🟦/🧊 chip, and the probe named the enum — `requires { { t.dimensions() } -> std::same_as<Dim>; }`. But the very next line did `static_cast<uint8_t>(probe.dimensions())` — the factory only ever wanted a byte. The `Dim` name in the constraint was incidental, not essential. Loosened the probe to `requires { static_cast<uint8_t>(t.dimensions()); }` — detects the method and reduces it to a byte without naming the type. `Dim` then moved to `light/light_types.h`, and with it gone **`core/types.h` was empty and was deleted.** Verified the probe still captures correctly: `/api/types` reports dim 3 for NoiseEffect, 2 for CheckerboardEffect, 0 for GridLayout/ArtNetSend — unchanged. (Safe because only EffectBase/ModifierBase declare `dimensions()`; nothing else matches the loose constraint.) - -**The rule that held:** don't accept "core uses X" at face value — check whether the use is *essential* or *incidental*. Every tie here turned out incidental and severable: a comment (`Control.h`), a serializer following a misplaced struct's field type (`PreviewFrame`/`put16`), and a SFINAE constraint that named a type it immediately discarded (`Dim`). The fixes: move the real owner and give core a domain-neutral seam — `BinaryBroadcaster` for the preview bytes, a return-type-agnostic probe for `dimensions()`. End state: **no `core/types.h`; core names zero light types.** - -## The layer buffer persists frame-to-frame; effects own their background (fade/fill/read-prior) - -The render buffer is **not** cleared each frame — `Layer::loop()` leaves last frame's pixels in place, the FastLED / WLED / MoonLight convention (their `leds[]` / segment / VirtualLayer buffers all persist; none auto-clears). An earlier design cleared the buffer before every effect frame ("the buffer is the effect's to fill, every time"), but that silently broke every persistence effect: a scroll (FreqMatrix) reads the prior column via `draw::get` and shifts it — reading a wiped buffer, so only the freshly-painted pixel survived; a trail effect calls `fadeToBlackBy` to decay the previous frame — fading zeroes, so the trail never formed; Game-of-Life reads its prior cell state — gone. The symptom that surfaced it: FreqMatrix lit only one row, and ~13 effects' `fade` controls did nothing. - -**The model (matches the three reference frameworks):** -- **Persistence is the default and universal** — the buffer holds the previous frame; there is no per-effect "persist" flag (a flag would be bespoke and change no framework behaviour, unlike `dimensions()` which drives `extrude`). The buffer is zeroed **once** on allocation/resize, then persists. -- **Each effect owns its background inside its own `loop()`:** a full-grid effect overwrites every pixel; a **trail** effect calls `layer()->fadeToBlackBy(amt)`; a **sparse** effect that wants a clean frame calls `draw::fill(buf, {0,0,0})` itself (e.g. RubiksCube, whose `drawCube` writes only surface voxels). Multiple effects on one layer deliberately *interact* through the shared persistent buffer — that's a feature, not a defect. -- **`fadeToBlackBy` is a Layer operation, collected once per frame** (MoonLight's `VirtualLayer::fadeToBlackBy`): effects register an amount, the Layer keeps the **MIN** across them (the gentlest fade wins, so the longest requested trail is honoured), and applies **one** buffer pass at the next frame's start, then resets. N fading effects cost one pass, not N, and never fade each other's fresh pixels. `unit_Layer_persistence` pins persistence + the MIN-collect + reset. - -**GoL corollary (still holds, now for the right reason):** separate *when the simulation advances* from *when the effect paints*. `GameOfLifeEffect` steps one generation per `bpm` beat but its state lives in the persistent buffer, so between beats it simply leaves the buffer untouched (the automaton holds). Time-gate the state update (the `dt*bpm` accumulator, same shape as `CheckerboardEffect`); the sim runs at `bpm`, the buffer persists at frame rate. A `unit_GameOfLifeEffect` case pins it. - -**Two adjacent traps the same effect hit:** -- **First-frame `dt`.** `lastElapsed_` starts at 0, so the first `now - lastElapsed_` is the whole device uptime — a huge `dt` that pins the step accumulator above the beat threshold *permanently* (max rate forever, `bpm` ignored). Bootstrap `lastElapsed_` on the first call and take `dt = 0` that frame. -- **Width of the change delta.** The stagnation check narrowed `alive - lastAlive_` to `uint16_t`; at the 512×512 max grid on a PSRAM board (`nrOfLightsType == uint32_t`, 262144 cells) that truncates and triggers false re-seeds. Counters derived from cell counts must be `nrOfLightsType`, not a fixed width. - -## A static "current instance" pointer needs re-election, not just claim/vacate - -`AudioModule::latestFrame()` hands effects the active mic via a process-wide `static AudioModule* active_`. The original design was "setup() claims the seat, teardown() vacates it" — which silently breaks with **two** mics: removing the one that holds `active_` leaves the seat null while a second, still-running mic sits captured-but-unread, and every audio effect goes silent. The fix is a three-part protocol: the **first** live module claims in `setup()`, `teardown()` vacates, and any running module **re-claims an empty seat in `loop()`** — so the survivor takes over on its next tick for *any* add/remove order (the robustness rule). Rule: a singleton-accessor backed by a static pointer must have a *self-election* path in the periodic loop, not only claim-on-setup — otherwise it's only correct for exactly one instance. (`unit_AudioModule` pins the two-mic first-wins + re-election.) Tempting CodeRabbit "fix": gate the claim on `inited_` — rejected, because a claimed-but-uninited module publishes valid *silence* (the documented contract), and a mic-less board running `simulate` publishes synth frames without being `inited_`; gating on init breaks both. - -## An "Effect-role" module is not guaranteed to be an `EffectBase` - -DemoReel hosts a child effect and needs its `dimensions()` to extrude it. The first cut did `static_cast<EffectBase*>(current_)->dimensions()` every frame on the `MoonModule*` child — which **crashed** (SIGBUS in RTTI/vtable) when the eligible list included a test `EffectStub` that registers with `ModuleRole::Effect` but is a bare `MoonModule`, not an `EffectBase`. Two lessons: (1) **role is a registration property, not a type guarantee** — a module can carry any `role()` without deriving from the matching base, so a downcast keyed on role is undefined behaviour; and (2) the data was already available without the cast — the factory probes `dimensions()` at registration via `if constexpr` and stores it, so `ModuleFactory::typeDim(index)` gives the dimension RTTI-free (ESP32 builds `-fno-rtti`, so `dynamic_cast` isn't even an option). Reach for the factory's probed metadata instead of a cross-tree downcast whenever you need a registered type's declared property. - -## uint16 intermediate overflow blanks the display — and a status check doesn't prove the render works (MultiplyModifier) - -A high fan-out modifier (`MultiplyModifier` at 8×8×4) black-screened the no-PSRAM Olimex while the desktop showed it working. Root cause: `Layer::rebuildLUT` computed `maxDest = logicalCount * mod->maxMultiplier()` in `nrOfLightsType`. On no-PSRAM that's `uint16_t`, and `256 * 256 = 65536` **wraps to 0** — the LUT was sized to ~nothing, so almost every light mapped nowhere and the frame went black. On desktop (`nrOfLightsType == uint32_t`) the product fits, so the bug was invisible there. **Fix:** compute the product in `uint64_t`, clamp to the ceiling, then narrow back. This is the same family as the GameOfLife delta-width trap above, but for an *intermediate product*, not a stored counter — **any `nrOfLightsType * nrOfLightsType` (or `× a multiplier`) can overflow uint16 even when both operands are individually small; do the arithmetic in a wider type before narrowing.** - -The harder lesson is about *verification*: the agent's hardware test asserted the Layer **status string** ("16×16×1") and called it a pass — but the status reflects `logicalDimensions`, which was correct; the *render* (driven by the corrupted LUT) was black. The product owner caught it by eye. **A status/dimension assertion is not proof the pipeline renders.** A correctness test for a mapping/effect must assert the **buffer or LUT is non-empty with the expected coverage** (the regression added here counts LUT destinations == physical light count), not just that the declared dimensions look right. And: a bug that depends on `nrOfLightsType` width is **invisible on the uint32 desktop build** — width-sensitive paths need either a uint16-typed unit test or hardware confirmation, not desktop-only. - -## Lessons from the repo-transfer + v1.0.0 release branch - -Moving the repo `ewowi/projectMM → MoonModules/projectMM` and cutting the first stable release surfaced three CI failures that had nothing to do with the transfer's code changes — they came from the *infrastructure around* the release (GitHub's hosted runners, GitHub Pages environment rules). Same diagnostic shape as the S3-DevKit branch: the red X appears on a release/build, but the cause is a platform behaviour we pinned against, not our diff. Record them so the next release doesn't re-dig. - -1. **Don't pin GitHub-runner toolchain specifics — the `windows-latest` image migrates underneath you.** Mid-release, GitHub began redirecting `windows-latest` from the VS 2022 image to `windows-2025-vs2026` (the run's own annotation warned: "redirected to windows-2025-vs2026 by June 15, 2026"). Two breakages followed on the *same source tree* that had been green the day before: (a) `package_desktop.py` hard-coded `-G "Visual Studio 17 2022"`, and the new image has no VS 2022 → CMake failed at configure ("could not find any instance of Visual Studio"), a 21-second fast-fail before any compile; (b) once the generator pin was dropped (let CMake auto-detect the installed VS), the build reached compilation and the *new* VS 2026 MSVC STL emitted **C5285** ("specializing `std::tuple` is forbidden") on the vendored `doctest.h`'s tuple forward-declaration, and `/WX` made it fatal. **Fixes:** drop the generator pin (auto-detect survives image migration); add `/wd5285` to the existing MSVC suppression list (it's third-party header code we don't own; GCC/Clang never warn). **Generalisable:** anything pinned to a hosted-runner's bundled toolchain version (CMake generator, compiler path, SDK version) is a time-bomb — the image rotates on GitHub's schedule, not yours. Prefer auto-detection; when a pin is unavoidable, expect to update it and don't treat a sudden Windows-only failure on an unchanged tree as your regression. - -2. **Don't gate asset-publishing behind a Pages-only `environment:` — a tag fails the gate before any step runs, silently dropping all release assets.** The `release` job did two things (upload release binaries *and* deploy GitHub Pages) under one `environment: github-pages`. That environment's protection rule allowed only `main`. When publishing v1.0.0 created the `v1.0.0` *tag*, it re-triggered the workflow on `refs/tags/v1.0.0`; GitHub evaluates the environment protection rule **at job start, before the first step** — the tag failed it, so the *entire job was rejected in 2 seconds*, including the "Publish GitHub release" step. Result: a published release with **zero binaries**, and a red X whose message ("Tag v1.0.0 is not allowed to deploy to github-pages") pointed at Pages, not at the asset upload that actually got skipped. The two symptoms (missing assets + red X) had one root cause: coupling. **Fix:** split into a `release` job (no environment, `contents: write`, uploads assets — runs on tags) and a `deploy-pages` job (`needs: [release]`, `if: ref==main`, carries the `github-pages` environment). Tags publish assets; `main` deploys Pages. **Generalisable:** an `environment:` on a job gates the *whole job* via a ref-based protection rule evaluated up front — never put work that must run on refs the environment forbids (tag-triggered asset upload) in the same job as the environment-gated work (production deploy). Recovery without re-tagging: `gh workflow run release.yml -f tag=vX.Y.Z` replays the fixed job and uploads assets onto the existing release. - -3. **A failure that looks like the change is often the environment — verify before assuming a regression.** Recurring across this branch: a scenario "failed" its 120µs tick contract at 536µs (machine load from concurrent builds + MoonDeck + a preview server — re-ran isolated at 118µs, passed); Improv reported `UNABLE_TO_CONNECT` while the device was *already provisioned and reachable* (async-confirmation timeout, not a join failure); MoonDeck showed `0/0 online` while the device served HTTP 200 (the active network record had an empty subnet and a duplicate record held the device). None were code or transfer defects. **The rule:** when something fails right after a change, first prove the failure is *about* the change — re-run isolated, probe the actual end state (ping/curl the device, read the real CI error line, check the env), and only then edit code. Several hours here would have been saved by checking the device was reachable *before* debugging the "WiFi failure". - -## Lessons from the LCD_CAM WS2812 driver bench debug (LcdLedDriver, S3) - -Bringing the 8-lane LCD_CAM driver from "compiles and ticks" to "strip actually animates" took three stacked root causes, each masked by the one before it. The through-line: **every layer of indirection between "the code ran" and "the LED lit" hid a failure the layer above couldn't see.** Record the chain so the next parallel-output driver (16-lane LCD, P4 PARLIO, Teensy FlexIO) skips the dig. - -1. **The i80 peripheral requires ALL `bus_width` data GPIOs — a partial bus never exists.** `esp_lcd_new_i80_bus` validates every `data_gpio_nums[0..bus_width)` and rejects `GPIO_NUM_NC` entries (`esp_lcd_panel_io_i80.c`, "configure GPIO failed"). A 1-pin config therefore never initialized: no bus, no transmit, dark strip — while the UI showed a configured, enabled driver. **Fixes:** the driver demands exactly 8 pins and reports `LCD bus needs exactly 8 pins` in the status slot (unused lanes take `0` in `ledsPerPin` and idle LOW); the loopback builds its private bus full-width from the driver's real pin set. **Generalisable:** when a peripheral claims a *group* of pins, surface the group contract in the control's validation — don't let a config that the hardware layer will reject look valid in the UI. - -2. **"Capacity still fits" is not "config unchanged" — a resize-optimisation early-return swallowed pin changes.** `reinit()` skipped the bus rebuild when the existing DMA buffer was big enough — correct for grid resizes, wrong for pin edits: moving lane 0 from GPIO 13 to 18 keeps the frame size identical, so the bus kept clocking out on the OLD pins and the strip pin carried nothing. **Fix:** record the pin set (data + WR + DC) the live bus was built with and compare on every reinit; any difference forces the rebuild. **Generalisable:** an "is the existing resource still good?" fast path must compare *identity* (what the resource is bound to), not just *capacity* (how big it is). Same family as the S3-DevKit branch's stale-cache lesson: the cached check was true about the wrong invariant. - -3. **Gate the LCD driver on `CONFIG_SOC_LCDCAM_I80_LCD_SUPPORTED`, NOT `CONFIG_SOC_LCD_I80_SUPPORTED` — the names look interchangeable and are not.** `lcdLanes` (and the whole driver wiring) first gated on `SOC_LCD_I80_SUPPORTED`, on the assumption it meant "has the S3-style LCD_CAM i80 bus." It doesn't: **the classic ESP32 also defines `SOC_LCD_I80_SUPPORTED=1`** for its unrelated *I2S-LCD* peripheral. So on the classic chip `lcdLanes` became 8, the `LcdLedDriver` got wired at boot, and `esp_lcd_new_i80_bus()` hung the watchdog trying to init an LCD_CAM bus the chip lacks → **boot-loop on every classic ESP32** shipped with that gate. The correct macro is **`SOC_LCDCAM_I80_LCD_SUPPORTED`**, defined only on chips with the actual LCD_CAM peripheral (S3 / P4) — which is what `esp_lcd`'s i80 driver requires. Bisect-proven (the prior commit booted, the LCD-driver commit looped) and hardware-verified after the fix (classic boots with LcdLed absent, S3/P4 keep it). The gate lives in `src/platform/esp32/platform_config.h` (`lcdLanes`) and the `#if` in `platform_esp32_lcd.cpp`. **Generalisable:** SOC capability macros with near-identical names can describe *different peripherals* on different chips — verify the macro is actually defined where you expect (and only there) before gating on it; a "supported" flag that's true on a chip you didn't mean is worse than a missing one, because it silently activates code on the wrong hardware. - -4. **A self-test that passes while the device fails is telling you what the test doesn't cover — close the gap before theorising.** The first loopback transmitted a 3-byte synthetic pattern through a 136-byte transfer and PASSED while the real strip washed out max-white. The differences between test and reality were the suspect list: frame size (5.4 KB, multi-descriptor GDMA chain), sustained back-to-back cadence, and pulse timing as seen by a real WS2812 rather than an RMT RX capture. Upgrading the test to transmit the driver's REAL frame (same size, same chain, repeated like the render loop) and bit-verify the WHOLE capture (RMT RX with the DMA backend, ~1536 symbols) eliminated the first two suspects in one run — leaving timing as the only remaining difference, which was the answer. **Generalisable:** when test and reality disagree, enumerate what the test abstracts away and make the test transmit the genuine article; each closed gap either finds the bug or eliminates a theory with proof instead of speculation. - -5. **Modern WS2812B reads a 412 ns "0" as "1" — T0H max is ~380 ns on newer revisions, and 3.3 V direct drive eats the remaining margin.** The classic 3-slots-at-2.4 MHz encoding (hpwit / FastLED lineage, slot ≈ 416 ns) produced a waveform the RMT RX decoded perfectly — and the strip rendered as max white with flicker: most "0" pulses sampled as "1" (mostly-ones ≈ white; the animation's actual 1-bits flicker through). The same strip on the same pin ran clean from the RMT driver's 350 ns zeros, which isolated timing as the only variable. **Fix:** pclk 2.67 MHz → 375 ns slots ("0" = 375 ns, "1" = 750 ns, bit 1125 ns), inside every WS2812B revision's window; latch pad resized to keep ≥300 µs. The lineage gets away with 416 ns because those rigs typically sit behind a 74HCT level shifter that restores threshold margin. **Generalisable:** datasheet timing windows shrank across WS2812B revisions — design new encoders against the NEWEST revision's T0H max (≤380 ns), and treat "RMT-captured waveform is correct but the strip disagrees" as a threshold/margin problem, not a logic problem. - -Bench-procedure notes worth keeping: a board that drops STA mid-session falls back to softAP silently — poll the device's reachability before interpreting an unanswered API call as a wedge (one STA beacon-timeout drop was observed seconds after the LCD bus first went live; not reproduced since, watch for recurrence). And the differential test that cracked the case twice: drive the same pin/strip with the already-proven RMT driver — it exonerates wiring, power, and the strip in one move. - -## Diagnosing LED flicker: eliminate firmware with hardware tests before blaming (or fixing) the wire - -A classic ESP32 driving a WS2812 strip on RMT showed random wrong colours on LEDs the effect left black ("blue flicker", later "random flicker"). The temptation is to guess — WiFi interference, a buffering bug, timing — and start changing code. The bench session that resolved it instead ran a four-step elimination, each step a *measurement*, and the answer fell out: - -1. **Capture the source/preview buffer** — it held zero stray colour. The effect output is clean, so the corruption is downstream of the logical buffer (not an effect or correction bug). -2. **Run the whole-frame loopback self-test** through a short jumper — bit-exact `PASS`, repeatedly, even under WiFi load. The RMT encode + transmit emit a correct WS2812 waveform on real silicon, so the firmware/peripheral is innocent. -3. **Sweep `txPowerSetting` 20 → 1 dBm** while watching the strip — the flicker was *constant*. A ~50× drop in radiated RF changed nothing, so it is **not** WiFi coupling into the data line (the standing hypothesis, disproven by the experiment). -4. **Check the pulse timing** — 350/700/1250 ns, spec-exact, and the loopback confirms the wire carries them. Not a timing-margin bug like the LCD T0H case. - -With every firmware cause eliminated *by test*, the remaining cause is the physical data path — and "constant regardless of TX power" specifically fingers electrical signal integrity over radio. On a 3.3 V part driving WS2812 directly, the dominant cause is the missing 3.3 → 5 V level shift (the LED's logic-high threshold sits above what the GPIO drives, so marginal bits flip under any noise). Fix is hardware: level shifter, series resistor, shorter/grounded wire — documented in [LED signal integrity](../usecases/led-signal-integrity.md). - -**The transferable lesson** is the order: when hardware output looks wrong, prove the firmware is *not* the cause with a measurement at each layer (buffer → encode/transmit loopback → environment sweep) before either editing code or buying parts. Two strong hypotheses here (a buffering regression, WiFi interference) were both wrong, and only the measurements said so — guessing would have burned a level shifter's worth of time on the wrong layer, or "fixed" code that was never broken. The whole-frame loopback self-test exists precisely so step 2 is a one-click answer instead of a scope session. - -**Red-herring note:** the flicker's *appearance* shifted (blue-only → random) after an unrelated change (the `lightPreset` default went RGB→GRB, remapping which channel carries which colour) plus a pin swap. The underlying electrical fault was identical; only the colour mapping over it changed. A changed symptom is not proof a code change caused it — confirm the mechanism, not the surface. - -## ESP32-P4 support, round 1 — per-board Ethernet pin config, and the P4's WiFi reality - -Adding the Waveshare ESP32-P4-NANO (round 1 of 4: board + Ethernet-only; later rounds add the Parlio LED driver, C6-co-processor WiFi, and the Parlio loopback). Two findings worth keeping: - -**The P4 has no native WiFi.** `SOC_WIFI_SUPPORTED` is absent on esp32p4 (it has EMAC, RMT, LCD_CAM i80, and Parlio, but no radio). WiFi on these boards comes from an on-board **ESP32-C6 co-processor over SDIO** via the `esp_wifi_remote` / esp-hosted stack — which is a managed component, **not in mainline IDF v6.1-dev**. So round 1 ships Ethernet-only (`MM_ETH_ONLY`, WiFi components excluded), and round 3 will introduce a WiFi abstraction seam so the P4 routes to the remote stack while classic/S3 stay on native `esp_wifi`. The C6 SDIO pins on the P4-NANO (recorded for round 3, and so round-2 Parlio avoids them): CLK 18, CMD 19, D0-D3 14-17, C6 reset 54. - -**Ethernet pins became a per-target compile-time config, not scattered #ifdefs.** `ethInit()` had the Olimex RMII/PHY pins baked in as literals. The P4-NANO needs different ones (IP101 PHY addr 1, MDC 31, MDIO 52, reset 51, and crucially an *external* 50 MHz RMII clock fed IN on GPIO50 — `EMAC_CLK_EXT_IN`, the opposite of Olimex's `EMAC_CLK_OUT`). Rather than `#ifdef` the pins inside `ethInit`, an `EthPinConfig` struct + a `constexpr ethPins = isEsp32P4 ? {…} : {…}` lives in `platform_config.h`, and `ethInit` reads it. This keeps the platform-boundary rule (compile-time branching is `if constexpr` on config flags, not `#ifdef` in domain code), turns the Olimex magic numbers into a named config, and is the seam future eth boards extend. Full *runtime* PHY/pin selection stays a 2.0 backlog item — this is compile-time-per-target, which is all the board variants need. - -**The IP101 PHY driver is a managed component in IDF v6.** IDF v6 moved every per-PHY driver out of `esp_eth` core into the component registry (`espressif/ip101`). The generic PHY (Olimex LAN8720) stays in core, so only the P4 build pulls `espressif/ip101` (added to `idf_component.yml`); the IP101 ctor is behind `if constexpr (ethPins.isIp101)` so non-P4 builds never reference the symbol. A subtle build-script trap fixed alongside: the eth-fragment detector matched only `.eth`, so the new `sdkconfig.defaults.esp32p4-eth` (ending `-eth`) would have silently set `MM_NO_ETH` and stubbed Ethernet out — the matcher now accepts both `.eth` and `-eth`. - -## ESP32-P4 round 2 — Parlio LED driver: a simpler peripheral, the same encoder - -Adding `ParlioLedDriver` (the P4's parallel WS2812 path, round 2 of 4) turned out to be mostly *subtraction* from the LCD driver, not addition — worth recording why, so the next parallel-output backend (16-lane, Teensy FlexIO) starts from the right base. - -**Parlio is a simpler peripheral than the LCD_CAM i80 bus, so the driver is simpler too:** -- **No sacrificial WR/DC pins.** The i80 bus mandates two real GPIOs (pixel-clock + data/command) that WS2812 ignores — `LcdLedDriver` carries `clockPin`/`dcPin` controls just to feed them. Parlio generates the clock internally (`output_clk_freq_hz`), so `ParlioLedDriver` has neither control. -- **No exactly-8-pins rule.** The i80 layer rejects a partial bus (`esp_lcd_new_i80_bus` requires a real GPIO on every data line), which is why `LcdLedDriver` demands exactly 8 pins. Parlio takes `data_gpio_nums[]` with unused = `-1`, so the driver runs on any 1–8 lanes. The "exactly 8" validation simply isn't there. - -**The encoder is shared, not duplicated.** `LcdSlots.h::encodeWs2812LcdSlots` outputs "one bus byte per slot, bit L = data line L" — and a Parlio bus byte is byte-identical to an i80 bus byte. So `ParlioLedDriver` reuses the same encoder and the same `unit_LcdLedEncoder.cpp` test; no new encode code, no new encode test. (This is the "reuse a recognisable shape" rule paying off: a second parallel peripheral cost ~0 encoder lines.) - -**One Parlio API constraint to remember:** `data_width` must be a *power of two* (`(w & (w-1)) == 0`), ≤ `SOC_PARLIO_TX_UNIT_MAX_DATA_WIDTH`. We always create the unit at `data_width = 8` (matching the encoder's 8-bit bus byte) and set unused lanes' GPIOs to `-1` — rather than narrowing the bus to the lane count, which would mismatch the encoder's byte layout. So 1–8 driver lanes all map onto an 8-wide Parlio bus; the unused high bits are simply not wired. - -**Clock math is identical to the LCD driver:** Parlio's default `PLL_F160M` (160 MHz) ÷ 60 = 2.67 MHz = the same WS2812 slot rate (375 ns) the LCD driver settled on for the T0H-margin reason — so the timing decision carries straight across with no rework. - -**Default pins must dodge the P4's strapping GPIOs.** The first cut of `ParlioLedDriver` defaulted `pins` to `36,37,38,...` — but **GPIO 34-38 are the ESP32-P4's strapping pins** (boot-mode control). It inited fine on the bench (nothing drove them during the boot window), but defaulting LED *output* onto strapping pins is a latent footgun: a driven level at the wrong moment can change boot mode. Fixed the default to 8 strapping-safe pins (`pins="20,21,22,23,24,25,26,27"`) with `ledsPerPin="64"` putting all 64 lights on lane 0 — a serpentine 8×8 panel is one 64-LED strand, and the other 7 lanes idle LOW at zero cost (the parallel DMA transfer time is set by the *longest* lane, not the lane count, so 1-of-8 is the same speed as 8-of-8; reassigning `ledsPerPin` adds strips later with no pin change). The clear GPIOs on the P4-NANO, after Ethernet (28-31, 49-52), C6 SDIO (14-19, 54), I2C (7-8) and **strapping (34-38)**, are 20-27, 32-33, 39-48. **Generalisable:** when picking default output pins for a new chip, pull the strapping-pin list from the datasheet first — "it booted on the bench" doesn't prove a strapping pin is safe to drive, only that nothing drove it at the wrong instant. - -## Audio input (INMP441 mic) — ship the manual core, design fresh from the datasheet - -The first audio-reactive capability: `AudioModule` (a SystemModule Peripheral) reads an INMP441 I²S mic and publishes an `AudioFrame` (level + 16-band spectrum + dominant peak) that `AudioVolumeEffect` and `AudioSpectrumEffect` consume. Decisions worth keeping: - -**Two seams, everything else host-tested.** Only the I²S read and the FFT *kernel* sit behind the platform boundary (`platform_esp32_i2s.cpp`: IDF `i2s_std` + esp-dsp's float `dsps_fft2r_fc32`). All the signal math — DC strip, RMS, the Hann window, the magnitude→16-band log mapping — is pure header-only domain code (`AudioLevel.h`, `AudioBands.h`), the `RmtSymbol.h`/`LcdSlots.h` shape. The desktop `audioFft` stub is a real (naive O(n²)) DFT, so the *whole* pipeline (window → FFT → bands) runs end-to-end in CI on synthesized sines with no hardware. esp-dsp **float** (not fixed-point) is the right call on an FPU chip — Espressif's own benchmark fact. - -**Effects reach the producer via a static accessor, not a boot setter.** The normal producer/consumer wiring passes a `const Foo*` to the consumer once in `main.cpp` (PreviewDriver→HttpServer). That works when both ends are boot-wired singletons — but an audio effect can be *added through the UI after boot*, and a boot setter only ever wired the boot instance. So `AudioModule::latestFrame()` is a static accessor: the active mic registers itself in `setup()`, clears the pointer in `teardown()`, and returns a static silent frame when there's no mic. Any add/remove order yields the live frame or valid silence, never null. Reach for this whenever a *user-addable* consumer needs a *singleton* producer's data. - -**Found on hardware, pinned by tests.** Two bugs the desktop build couldn't surface, both now regression-tested: (1) a missing `registerType<AudioModule>` made `create("AudioModule")->markWiredByCode()` deref null and **boot-loop** — the fix is the registration plus a null-guard, pinned by a "registered + createable" test; (2) the I²S read **blocked the render tick** ~7.7 ms at a 20 ms timeout — fixed to a non-blocking read plus a cross-tick sample accumulator (a full 512-sample block takes ~23 ms at 22 kHz, longer than one tick), dropping AudioModule to ~400 µs. The INMP441 is also on the **left** I²S slot here (the right reads empty) — one config line, the first bench suspect when level floors with sound present. - -**Shipped the manual core; the adaptive conditioner was prototyped and removed.** The mic, FFT, log/dB scale, two effects, and a `floor`/`gain` manual control surface are the solid, host-tested increment that landed. A self-calibrating conditioner (auto noise-floor + AGC + smoothing, goal "sound off → dark, sound on → vivid") was built and then *deleted* — it needs bench tuning in a **quiet** room, and the development environment (a campground van with strong, *varying* low-frequency engine/inverter rumble) was the adversarial worst case that kept it from settling: a per-band auto-floor removes *constant* tones but can't track *varying* broadband ambient; global AGC pumps the residual to full in silence; a relative per-band floor fixes treble over-cut but flattens the spectrum. Lesson — **land the manual core, treat adaptive auto-tuning as its own increment, and tune it where the noise floor is real-quiet, not in the field.** Also: `level` is overall RMS loudness (independent of the FFT) — don't derive it from the bands, or it stops fluctuating with volume. - -**Designed fresh from the datasheet + textbook DSP, not from a prior project.** Per the product owner: don't trace WLED-MM (or any existing controller) for naming, structure, or functionality — build something independent. The concrete DSP choices and *why* (Hann/RMS/geometric-bands/argmax, and why a flat ±3 dB mic needs no per-frequency correction table) are documented in the module spec, [AudioModule.md](../moonmodules/core/moxygen/AudioModule.md), where an integrator looks for them. The lesson worth keeping here is consistent with the repo's *Industry standards, our own code* principle (study with respect, don't copy): **reference proven behaviour, don't trace structure.** Here the datasheet made even the behaviour-reference unnecessary — a flat ±3 dB mic has no per-frequency error to correct, so the hand-tuned band-correction table years of prior-project work produced was the wrong tool, and the textbook defaults were enough. Read prior art to understand *what* works and *why*; let the hardware datasheet and standard DSP decide *how*, so the result is independent by construction rather than a renamed trace. - -## ESP32-P4 round 3 — WiFi via the C6: the abstraction the earlier round feared wasn't needed - -Round 1 recorded that round 3 "will introduce a WiFi abstraction seam so the P4 routes to the remote stack while classic/S3 stay on native `esp_wifi`." The actual implementation was far smaller, and *why* is the lesson. - -**`esp_wifi_remote` is API-compatible, so there was no seam to build.** From the P4's side you still call `esp_wifi_init()`, `esp_wifi_connect()`, `esp_wifi_scan_start()`, `esp_netif_create_default_wifi_sta()` — identical signatures; the component forwards them to the C6 over SDIO. So the entire existing WiFi platform layer (`wifiStaInit`/`wifiApInit`/scan/tx-power, ~230 lines) compiles and runs **unchanged** on the P4. The only genuinely new code is a **two-call prelude** — `esp_hosted_init()` + `esp_hosted_connect_to_slave()` — that must run *before* `esp_wifi_init()`, added to `ensureWifiInit()` behind `if constexpr (platform::usesRemoteWifi)` (= `isEsp32P4 && hasWiFi`). Lesson: before designing an abstraction for "two backends," check whether the vendor already made them API-identical — here the "seam" was a 15-line prelude, not a routing layer. *Concrete first* would have caught this even without the foresight; the round-1 note over-scoped from a position of less information. - -**Init ordering is the whole risk.** Espressif's docs and the community starters are emphatic: esp_hosted must be fully up before anything touches the WiFi stack, and wrong ordering surfaces as **NVS errors / asserts / a silent hang**, not a clean error — so it reads like an unrelated bug. The prelude-before-`esp_wifi_init` placement is the mitigation; it's the first thing to check if a P4-wifi build misbehaves at boot. - -**Pulled P4-only via a `rules` gate.** `esp_wifi_remote` + `esp_hosted` are added to `idf_component.yml` with `rules: - if: "target == esp32p4"`, so classic/S3 and the eth-only P4 build never fetch or compile them. This is the clean answer to "managed components are per-project" — the gate makes the pull per-target. (The pre-existing `ip101` entry could use the same gate but doesn't; not worth churning.) - -**A deliberate v6.0-floor exception.** These components live outside mainline v6.0, so the build steps below the [v6.0 floor](../building.md#esp-idf-version). That floor has an explicit-exception clause exactly for cases like this: the product owner accepted it consciously, it's documented at every introduction site (the yml, the sdkconfig fragment, the platform flag, building.md), and it's scoped to the P4 — the other targets keep the v6.0 fallback intact. Lesson: a floor rule with a *documented-exception* path beats a rigid one; the exception stays honest because it's recorded, not silent. - -## RMT timeout: don't cancel a stuck transfer — the cancel crashes classic ESP32 - -A deferred "fuller error handling" item for `rmtWs2812Show`/`rmtWs2812Wait` (🐇 CodeRabbit PR#17) turned out to be mostly *already done* and partly *actively harmful* — a good case of *default to subtraction*. - -**Most of it had already landed** in the multi-pin (2a) work: `rmtWs2812Transmit` already returns the `rmt_transmit` result, and `RmtLedDriver::loop()` already tracks `started[]` so it only waits on channels whose transmit succeeded (a failed one gives no done-callback, so waiting would burn the full 1 s timeout). And there's no mid-transmit corruption risk because the symbol buffer is re-encoded from scratch *before* any transmit each tick. So two of the three "to-do" items were no-ops. - -**The remaining item — cancel the in-flight transfer on timeout via `rmt_disable()` — is a trap.** `rmt_disable()` while a transmission is still active triggers an **interrupt-WDT panic on classic ESP32** (espressif/esp-idf#17692; classic-only, S3/C6/P4 unaffected). That trades a self-healing dropped frame for a crash on a shipping target — strictly worse, and a direct *"crashed is not acceptable"* violation. So the right change was to **not** add the cancel, and instead replace the vague "deferred" comment with a sourced explanation of why we deliberately leave a timed-out transfer alone (it self-heals: next tick re-encodes and re-transmits; a still-busy channel just fails its `rmt_transmit` cleanly, and `started[]` skips the wait). Lesson: a deferred-improvement note is a hypothesis, not a spec — verify the improvement is real *and* safe on every target before implementing it; sometimes the finished work is "document why the current code is already right." - -## Pin defaults: assign one only when it cannot do harm - -A mic-less **classic ESP32** boot-looped (TG1WDT_SYS_RESET at ~736 ms, no panic backtrace — a silent hang). Bisect: clean-built the known-good commit (still looped → recent driver work innocent), then disabled the AudioModule wiring in `main.cpp` → booted clean → **AudioModule was the cause.** Root cause: AudioModule was **auto-wired** (`addChild` + `markWiredByCode()` in `main.cpp`, gated on `platform::hasI2sMic`), so on every boot it ran `setup()` → `reinit()` → `platform::audioMicInit()` → the IDF `i2s_channel_enable()`, which on the classic's older I²S driver **blocks forever** when no mic is clocking the pins. The watchdog fired on the stuck init. (The P4 was never affected: its newer I²S either returns a silent frame or fails cleanly without blocking — same mic-less condition, different driver behaviour. Two independent code paths, one symptom only on classic.) - -**The fix, per the product owner, was a design fix not a band-aid:** (1) **don't auto-wire AudioModule** — register it in the factory (`registerType<AudioModule>`) so it's user-addable like an effect, but only when the user with a mic adds it; (2) **default the mic pins to unset (0)**, not to bench values; (3) `reinit()` **no-ops on any unset pin** (`if (wsPin==0||sdPin==0||sckPin==0) { setStatus("set …"); return; }`) so even an added-but-unconfigured module never touches I²S. Classic then boots 191 FPS, 0 WDT resets, and an added mic works once its real GPIOs are entered. - -**The generalisable rule the product owner drew from it: _assign a pin default only when it cannot do harm._** The test is *who fixes the pin*: -- **Chip-/board-fixed pins → default them** (and you *must*): the RMII **Ethernet** pin map is silicon-/PCB-wired, so a default cannot do harm — and *omitting* it does, because a no-WiFi board with un-defaulted Ethernet pins can never connect to be configured (a chicken-and-egg lockout). This is why `platform::ethPins` is a compile-time-per-target constant, never a user-blank control — and it stays that way. -- **User-soldered pins → leave them empty**: a MEMS mic or an LED strand goes wherever the user ran the wire, so any default is a guess that can drive a pin the user committed to something else. Empty until set; idle with a "set pins" status meanwhile (the robustness rule: degraded is fine, crashed is not). - -The LED drivers (Rmt/Lcd/Parlio) are the same Device-level case — user-soldered, so their pin defaults follow the same rule. This rule is the runtime face of the three-level **MCU → Board → Device** config-provenance model (architecture.md § Config provenance): a pin may be defaulted only at the level that actually fixes it, and the empty Device-level defaults are the correct baseline a saved device profile later *fills* rather than *overrides*. Lesson: a hard-coded pin default is a claim about the user's hardware — make that claim only when the hardware, not the user's soldering iron, decides the pin; and never auto-run a peripheral whose init can block on absent hardware. - -## Live reconfiguration falls out of the prepare-pass for free — MoonLight's "initless" goal, a different mechanism - -projectMM has a property most LED-controller firmware lacks: **every module reconfigures live the instant a control changes — pins, leds-per-pin, output protocol, mic pin/rate — with no reboot, immediately reactive on the next render tick.** The design note for *why* this exists lives in [architecture.md § Live reconfiguration](../architecture.md#live-reconfiguration-every-change-applies-without-a-reboot); the lineage and the *how-it-differs* are the lesson worth keeping here. - -**The lineage is MoonLight's "initless drivers."** The product owner's earlier project ([MoonLight nodes.md § Initless drivers](https://github.com/ewowi/MoonLight/blob/main/docs/develop/nodes.md)) set the same no-reboot goal at the LED-driver level, named *initless*: a driver with **no `addLeds` (FastLED) / `initLed` (Parallel LED Driver) step** — it reads a mutable Context at `show()` time, so pin allocation, leds-per-pin, RGB/RGBW and light type all change live without a restart or recompile. - -**projectMM reaches the same outcome by a different mechanism, so the word doesn't transfer.** Our drivers *do* have an explicit rebuild — `RmtLedDriver::reinit()` re-creates the RMT channels, the i80/Parlio drivers rebuild the DMA bus — so they are not "initless" in MoonLight's no-`initLed` sense. What makes the behaviour universal here is that the rebuild is driven by the **generic tier-3 `onBuildState()` sweep** ([§ Event triggering](../architecture.md#event-triggering-between-modules)), not hand-built per driver: any module that returns `true` from `controlChangeTriggersBuildState` inherits live-reconfig for free, which is why it spans drivers, the audio peripheral, effects, layouts, modifiers and network I/O alike. Lesson: credit the lineage for the *idea* (MoonLight's initless drivers), but name the property by what the user sees (*live, no-reboot reconfiguration*) when the mechanism differs — overloading a prior project's term onto a different implementation misleads. And: a generic prepare-pass buys breadth a per-driver technique can't — the same three tiers that rebuild a mapping LUT also re-target a GPIO, so the property generalised itself. - -## Lessons from the catalog-driven installer branch (3-layer device model) - -The installer was reworked so a board catalog (`boards.json`) sets a device up for its hardware at install time. The mechanics and schema live in the installer README; these are the hard-won principles worth keeping. - -**Inject from data, don't bury in code (vs MoonLight's `ModuleIO.h`).** MoonLight hardcoded ~20 boards' pin presets in firmware C++ (`setBoardPresetDefaults()`, behind `#ifdef CONFIG_IDF_TARGET_*`); adding a board is a recompile, and every binary ships every board's table. projectMM instead **injects** the same information from the catalog after flash — the firmware is a generic engine that knows nothing about QuinLED/Serg/Olimex, and the *data* specialises it. Adding a board is a JSON edit, binaries carry no board tables, and board definitions become community-contributable data. This is the *domain-neutral core* principle: the specialisation is data, not code. - -**Investigate before building — the device side was already done.** The original plan was a new `POST /api/preset` batch endpoint with a device-side consumer. Investigation overturned it: three install clients already fan a catalog's controls out as `POST /api/control` calls. The *real* gap was that the fan-out can't configure a module that doesn't exist on a fresh flash (a control write 404s) — solved by the **already-existing, idempotent `POST /api/modules`** the clients simply weren't driving. Lesson: the reflex to add an endpoint hid that the mechanism existed; a day of mapping the existing code replaced a new core endpoint (and a forbidden JSON-array parser) with a small client change. *Default to subtraction.* - -**`loopbackTxPin` — verify the claim against the code before deciding twice.** A driver loopback self-test transmits on `pins[0]`, and the bench's loopback TX jumper is a *different* pin from the operational LED pins — so the test forced retyping `pins`. A `loopbackTxPin` override control was proposed, then **dropped** on the reasoning "it only fits RMT's single pin, not Parlio/Lcd's lane array," then **re-added** after reading the code: the Parlio/Lcd loopback only drives **lane 0** with the test pattern, so a single TX override substituting for lane 0 works uniformly on all three drivers. Lesson: the "doesn't fit lane arrays" objection was an assumption about the loopback, not a fact — checking `ParallelLedDriver::runLoopbackSelfTest` (lane-0-only) settled it. Verify the mechanism before a design call that rests on how it behaves. - -**Board vs Device is a completeness spectrum, not two schemas.** The carrier/shield pattern (a PCB an ESP32 *module* plugs into) is a **Board** — it fixes the LED/relay pins, the MCU and strips are chosen separately. A vendor-finished all-in-one with peripherals soldered (QuinLED Dig-2-Go, Dig-Next-2 with built-in mics) is a **Device**. But they are the *same* catalog entry shape — a "Device" is just a Board entry with more of its optional `modules`/`controls` filled in (verified: every `boards.json` entry shares one schema). So **no *separate-schema* `devices.json`**: one entry type, not two. (This rejects a *second schema*, not a future *same-schema* grouping — architecture.md leaves the door open to splitting the flat list into a `devices.json` / `kind:` tag purely for organisation once it's large enough to be unwieldy; that's a file-layout choice under the sequencing rule, not a second entry type.) A board's pins fall into three categories, not two: *always-fixed* (LED outputs, status LED — default freely), *board-optional* (a populated-or-not W5500/IR/power-monitor — an opt-in peripheral block, can't default blind because the same board name ships with and without), and *user-soldered* (always unset). The optional-peripheral case (e.g. SE 16 with a W5500) is the runtime SPI-PHY mechanism, not a new one. - -**Per-board capability spec'n'test loop.** Each board's pin-layout `image` and product-page `url` are the *inputs* to a repeatable build loop, not just decoration: (1) read the board's capabilities off the image + link (LEDs, mic, IR, power monitor, …); (2) for a capability we already offer (an I²S mic → `AudioModule`), wire it into the entry's `modules`/`controls` with real pins; (3) for one we don't offer yet (IR receive), write a proposal against the architecture, spec it, **create test scripts** (host unit + scenario, hardware loopback where applicable), iterate until it works, *then* add it; (4) the entry grows only as far as each capability is spec'n'tested — an un-implemented capability is a recorded proposal, not a half-wired control. A fully-implemented board entry is the *output* of running this loop; the image/link tell you what to aim for. This is *Specs before code* at board granularity. - -**Drivers became catalog-added, with an OTA nuance.** LED/network drivers stopped being boot-wired (only `Preview` stays, since it needs the HTTP broadcaster the catalog can't supply); each board declares its driver(s) in `modules`. A fresh-erased board boots with `Drivers = [Preview]` only — the deliberate explicit-add model. **The nuance hardware surfaced:** an OTA update *without* erase keeps a device's previously-persisted drivers (they're saved config the new firmware reloads as user-added children), so the clean Preview-only state applies only to a fresh/erased flash. That's correct — an update shouldn't wipe a user's configured drivers — but it means "out-of-box" and "after-update" differ, which isn't obvious until you flash a non-erased board. - -**A persistence overlay must distinguish "key absent" from "value 0".** The runtime-Ethernet-PHY work moved pin/PHY config from a compile-time `constexpr ethPins` into persisted NetworkModule controls (`ethType`, pin GPIOs, …) with **non-zero per-chip defaults** (P4 IP101 = `ethType` 2). That exposed a latent bug in `applyControlValue` (the persistence load path): it used `json::parseInt(json,key)`, which returns 0 for an *absent* key — indistinguishable from a real 0 — and then wrote that 0 into the control under the Clamp policy. So loading an older/partial `<Module>.json` that omitted a key **clobbered the control's default with 0**. On the ESP32-P4 this zeroed `ethType` (2 → 0 = none), so `ethInit()` dispatched to "no Ethernet": link LEDs on, but no DHCP. It was invisible on classic/Olimex (their eth defaults are mostly 0 anyway) and on `main` (which still read the `constexpr ethPins` directly), so it only bit once eth config became persisted controls with meaningful non-zero defaults. **Fix:** a `json::hasKey()` guard in `applyControlValue` — an absent key leaves the control untouched (preserves its default); a present key (even value 0) still applies. Lesson: any "control resets to its default/0 after reboot" symptom is a persistence-overlay smell, not a control-init bug; a flat JSON parser that returns a zero sentinel for missing keys MUST be paired with a presence check before the value is applied as authoritative. The decisive debugging move was a `std::printf` of the runtime struct over the P4's *secondary* USB-Serial-JTAG console (stdout reaches USB even when ESP_LOG/UART is on GPIO 37/38), after a `git worktree` bisect (round-1 ✓, main ✓, uncommitted ✗) proved it was our code, not hardware or IDF. - -**A GPIO pin is its own control type (`ControlType::Pin`), not an overloaded int16.** Pins were first added as `addInt16` with a `-1..48` range, which the UI rendered as a *slider* — meaningless for a GPIO, and the cap wrongly excluded the P4's high pins (MDIO 52, clk 50). Dropping the range didn't help: the UI's `int16` case *always* draws a slider (an unbounded int16 falls back to a −100..200 percentage slider that position/region controls rely on), so int16 couldn't be made to mean both "position slider" and "pin number." The fix is a dedicated `Pin` type: `int8_t` storage (one byte — a GPIO never exceeds ~54, and on a DRAM-scarce ESP32 the per-pin byte matters across many pin controls), −1 = unused, the UI always renders a plain number input keyed off the `"pin"` type string, and min/max are a server-side write-clamp guard only. Serializes/parses as a plain integer (same as int16). This also serves every future pin control (LED-driver clockPin/dcPin, GyroDriver SDA/SCL, board pins) — they migrate to `addPin` for free. Lesson: when one control type is doing two jobs with different UX (slider vs number), that's the smell for a new type, not a range hack; and pick the smallest storage that fits the domain (int8 for a pin). - -**`deviceName` (identity) vs `deviceModel` (product) vs board (bare PCB) — one term was doing three jobs.** "Board" had been overloaded to mean the per-unit network identity, the hardware product/catalog key, AND the bare PCB. Untangling it: `deviceName` is the **per-unit identity** — one string that drives mDNS (`<deviceName>.local`), the SoftAP name, and the DHCP hostname, so the device shows up under one name everywhere; it's RFC-1123-coerced (`sanitizeHostname`) because it becomes a hostname. `deviceModel` is the **hardware product** (the `deviceModels.json` catalog key, e.g. "projectMM testbench S3") — display-form, spaces allowed, never a hostname. "Device" is the umbrella noun; "board" now means **only the bare PCB**. This drove the BoardModule→SystemModule fold (the identity is core unit state, not a separate module), the `board`→`deviceModel` rename across catalog/installer/Improv (SET_BOARD→SET_DEVICE_MODEL, byte 0xFE unchanged), and the eth pin-map clarification (driver = firmware, pin map = firmware-seeded but **deviceModel-authoritative** so an Olimex entry can override). Lesson: when one noun answers three different questions ("what do I call this unit on the network?", "what product is it?", "what's the bare board?"), that's a naming smell — split it into the qualified terms, pick one umbrella word, and make the split visible in every layer (control names, RPC symbols, catalog keys, docs) so the three concepts can't re-merge. - -**"Improv = REST over serial" — one apply-core, two transports, and the testability that follows from extracting the hard part.** The deployed HTTPS installer couldn't configure a flashed device: a browser blocks an HTTPS page from POSTing to an `http://` device (mixed-content), and the `?deviceModel=` pull/handoff that replaced it only ran if the user opened that exact link. The fix reframed the problem — the installer already owns the USB serial port during provisioning, so push the config over it as the *same REST operations the HTTP API runs*: a new `APPLY_OP` (0xFC) Improv vendor RPC whose payload is `{"op":"add|set|clearChildren",…}`, the same JSON a `POST /api/modules`/`/api/control` body carries. On the device the op routes to **one transport-free apply-core** (`HttpServerModule::applyAddModule/applySetControl/applyClearChildren/applyOp`) the HTTP handlers also call, so a network REST call and a serial APPLY_OP execute identical code; the handlers became thin `switch(applyX())` → status-code mappers. This **deleted** the whole browser handoff (device-side catalog fetch, `?deviceModel=` decoration, the inject button) — a net subtraction — and works on Ethernet-only firmware once the Improv listener is decoupled from WiFi (the vendor RPCs compile in unconditionally; only `WIFI_SETTINGS`/`GET_WIFI_NETWORKS` stay `#ifndef MM_NO_WIFI`). Lesson 1: when a push is blocked by the *medium* (mixed-content on HTTPS), look for a medium you already control (the serial port mid-flash) instead of bolting on a fragile pull. Lesson 2 (the one with legs): the way to make it *provable* was to **extract the hard part into a pure core primitive** — the chunk reassembly + out-of-order/duplicate sequence guard moved from the ESP32-only handler into `src/core/ImprovOpReassembler.h` (header-only state machine, returns `Continue/Ready/Error`), and the JS frame builders into `docs/install/improv-frame.js` so `node:test` imports them without the orchestrator's browser deps. Both are *Complexity lives in core; domain modules stay simple* applied for testability: the device handler keeps only its serial I/O, the algorithm gets unit-tested on the desktop, and a format implemented three times (device C++, Python, JS) is pinned by **one shared golden vector** asserted in `test/python` + `test/js` — a contract test is the right answer to *forced* duplication no shared compilation target can remove. The reflex worth keeping: a hard mechanism buried in a platform `.cpp` that "can only be tested on hardware" is a smell — extract its pure core, and "rock solid proven" becomes a unit test instead of a bench session. - -**A periodic re-broadcast to let late joiners "catch up" is a hack wearing a keepalive costume.** The 3D preview sends a coordinate table (positions) once, then per-frame colour. The original implementation re-sent the *whole table every ~1 second* "so a client that connected after the last rebuild catches it." It looked fine — a fresh page recovered within a second — so it shipped and sat there. But it's a workaround, not the mechanism: it rebuilt the full table from the layout **every tick-second forever**, on the hot path, whether or not anyone had connected and whether or not anything changed — and it papered over a missing request/response with polling. The correct construct is event-driven: send the table **when it actually changes** (`onBuildState` — grid/layout/LUT rebuild) and **when a client asks** (a new WS connection bumps `BinaryBroadcaster::clientGeneration()`, which `PreviewDriver::loop()` watches and re-sends on change). That's strictly *less* code than the timer and zero idle cost. How it sneaked past review: the workaround *worked* in casual testing and its cost was invisible until a later change made each rebuild heavier and the per-module tick was profiled. Lessons: (1) "re-send periodically so it eventually syncs" is the polling-instead-of-events smell — ask "what's the event that should trigger this?" and trigger on *that*; (2) a recurring rebuild on the hot path must justify itself every tick, so "every second, just in case" fails *Data over objects / fastest hot path* on sight; (3) this is *Continuous refactor, no hacks* — the fix isn't a scheduled cleanup, it's "the moment you see a keepalive timer standing in for a request, replace it." The guard is a test that advances the clock several seconds with no client change and asserts the table is **not** re-sent (the old timer would have). - -**When a working seam regresses after your "fix," suspect the fix — and measure with a tool faithful to what the user sees.** The resumable preview send (`sendBufferedFrame`/`drainPreviewSend` — stream the producer buffer a memory-adaptive chunk per `loop20ms`, drop-new backpressure, downsample + display cap) shipped working on all three boards. A later attempt to *also* route the coordinate table + downsampled colour frame through the resumable path (removing the synchronous `sendAllOrClose` spin-and-close) looked principled but **regressed every board into an intermittent stall**, through several variants. Three lessons compounded: (1) **Stop at the first failed fix on a working path.** Each "improvement" to a seam that already worked added a new failure; the discipline (CLAUDE.md *Anti-stalling*) is to revert to the known-good state at attempt two, not re-engineer. The committed synchronous coord/downsampled path *closes a wedged client past a spin budget and lets it reconnect* — not elegant, but proven; the elegant unification wasn't worth a regression nobody could pin. (2) **A measurement tool must be faithful to the real client or it invents and hides bugs.** A plain one-shot WebSocket probe *gave up on close* where a browser *reconnects*, so it reported stalls users never saw AND missed blips they did — it sent the debugging in circles for a whole session. The fix was a browser-faithful probe (`moondeck/diag/preview_health.py`: reads binary frames, sends the 25 s keepalive ping, auto-reconnects with backoff — exactly `app.js`'s `connectWs`); only then did probe and eyes agree. (3) **A stale process masquerades as a code bug.** "No preview on the PC build" with a corrupt-looking on-the-wire coordinate count survived every fresh rebuild — because a **38-hour-old desktop binary** still held port 8080; the freshly-built one couldn't bind and the browser/probe hit the stale one. The tell was the *uptime*, spotted by the product owner, not the diff. Lesson: before bisecting a "rebuild didn't fix it" bug, confirm the artifact under test is the one you built (check the process uptime / `build` timestamp / what's actually bound to the port). The faithful probe is now the standing way to measure preview health on any target (PC + the three boards), wired into MoonDeck's Live tab. - -**Don't hold a vendor library's async handle across your own event loop — it races the library's internal timers.** A UI refresh intermittently crashed the device (`assert failed: xQueueSemaphoreTake queue.c:1709 (( pxQueue ))` — a null FreeRTOS queue — inside the espressif mDNS component's `mdns_query_async_get_results`, plus an `Interrupt wdt timeout`). The mDNS *browse* (discovering peers for the "Your devices" list — distinct from mDNS *advertise*, which serves `<deviceName>.local` and was never the problem) used the async API: `mdns_query_async_new` returns a handle that `DevicesModule` held across ticks, polling it each `loop1s` with a 0 ms timeout. The trap: the mDNS component's **own task** owns that handle's queue and **frees it when the query's window (3 s) expires** — so a poll landing in the gap after expiry asserts on a freed queue. It was intermittent and grid-size-sensitive (a bigger grid lengthens the tick, widening the gap) and looked like "refresh crashes it" only because a refresh's activity coincided with the poll. **First fix attempt was wrong:** I assumed a *service-table mutation* (live rename re-registering `_http._tcp`) tore the handle down and added a cancel-before-mutate guard — it didn't fix it, because the freeing party is the component's expiry timer, not our code. **Real fix:** stop using the async-handle API entirely — replace the start/poll/stop trio with one synchronous `mdnsBrowse()` (`mdns_query_ptr`) that queries, delivers results, and frees everything in a single call, holding **no handle across ticks**, so the race window can't exist. The catch that synchronous introduced: `mdns_query_ptr` blocks the *full* timeout (it waits the whole window for late responders, no early return) and `loop1s` is charged to the tick — an 80 ms query tanked the tick. So **throttle**: browse one service type every ~8th tick with a ~60 ms timeout — one brief hiccup every ~8 s, invisible for discovery, FPS untouched in between. Lessons: (1) a library's async/iterator handle is only valid between *its* lifecycle events — if you can't see/where those fire (here, an internal expiry timer on another task), don't hold the handle across your loop; prefer a self-contained synchronous call that owns the whole lifecycle. (2) An *intermittent, load-dependent* crash whose backtrace sits in a vendor component is a **lifecycle race**, not a component bug — but find the *actual* concurrent actor before "fixing" (my first guess at the actor was wrong and the fix did nothing). (3) Trading async for synchronous trades a race for a blocking cost — budget it (throttle + bound the timeout) so the cure isn't a tick-killer. (4) Desktop stubs these mDNS calls to no-ops, so it's a hardware-only fix the unit suite can't reach; the reproduction (concurrent WS churn at a large grid → crash before, stable after, uptime climbing) is the proof, in the commit, not a desktop test. - -**A dead control that was always meant to be functional belongs in the mechanism that already expresses it, not where it happens to sit.** Six persisted-but-ignored Layer controls (percent region carving) were due to be wired into `rebuildLUT`; the product owner's question — *can a modifier already do this?* — was the better path. `ModifierBase`'s two virtuals (`logicalDimensions`, `mapToPhysical`) express carving exactly, so it shipped as a modifier and the Layer controls were deleted. Lessons: (1) before wiring an inert control where it lives, check whether an existing mechanism already covers it. (2) "make the default fastest" is best met by making the default the *absence* of the feature — full coverage = no modifier = the existing fast path, zero added cost — not a guarded branch through it. - -One sub-decision the implementation forced: the boundary rounding. The original spec said inclusive-ceil ("start 33/end 66 on a 4-wide axis → pixels 1..3"), which on a 128-wide axis makes `end=50` land on pixel 64 *inclusive* — so two abutting layers (0..50, 50..100) **overlap by one pixel** at the seam. The product owner chose **half-open `[start, end)`** instead: `end=50` → pixels 0..63, and 0..50 + 50..100 tile a 128 axis into 0..63 / 64..127 exactly, no overlap, no gap (with a min-1-pixel floor so tiny panels still get a non-zero region). Lesson: when a region/range feature will be used to *tile* a space, half-open intervals are the textbook choice (same reason `[begin, end)` is the C++ iterator convention) — inclusive bounds double-count the seam. - -Lessons: (1) a persisted-but-inert control is a feature with no home yet — before wiring it where it sits, ask whether an existing mechanism already expresses it (the modifier interface did, completely). (2) "make it the fastest at the default" is often best met by making the default the *absence* of the feature, not a fast branch inside it. (3) a feature framed as "a Layer property" may really be "a composable transform" — the modifier framing also unlocked stacking for free. (4) reach for half-open intervals whenever regions abut. - -## Composable modifiers — invert the map build (physical→logical), don't bolt fan-out onto the old interface - -Modifiers needed to chain (Region then Multiply then Rotate), but the old interface — `mapToPhysical(logicalCoord) → [physical indices]`, a virtual→physical **fan-out** — didn't compose: stages emitted flat indices, not coordinates, and chaining would need a product-of-`maxMultiplier` fan-out ceiling (the exact overflow class that caused the multiplyZ black-screen). The fix was to **invert the build to physical→logical**, adopting MoonLight's proven model (the product owner's prior engine) in projectMM's own code: each modifier becomes an in-place coordinate fold, and the Layer walks the *physical* lights, folding each through the enabled chain to its logical cell. Three hooks — `modifyLogicalSize` (fold the box), `modifyLogical` (fold a coord, return false to reject), `modifyLive` (per-frame remap for Rotate). - -Why the inversion was the right call, not just the chaining bolt-on: -- **Fan-out becomes free.** N physical lights folding onto one logical cell *is* the fan-out — no fan-out list, no product ceiling, no overflow. `destinationCount ≤ driverCount` is now a hard invariant. The whole `maxMultiplier`/scratch-buffer/`buildBoxToDriver`/`buildSparseIdentityLUT` machinery deleted. -- **The hot path is untouched.** Our `MappingLUT` is a CSR keyed by logical index; the inverted build is a scatter onto arbitrary logical keys, which doesn't fit `setMapping`'s in-order contract — so the build is a textbook **counting-sort CSR construction** (count, prefix-sum, scatter, replay) entirely on the cold path. The per-frame `forEachDestination` read is byte-identical. -- **Static vs dynamic split correctly.** Mask/tile/crop fold forward at build time (`modifyLogical`); rotation gathers backward per frame (`modifyLive`) — each in its natural direction, so rotation keeps its clean inverse-sample (no gaps), and a static-only chain pays *nothing* per frame (the live pass is gated on `hasModifyLive()`). - -Lessons: (1) when a feature "doesn't compose," check whether the *interface direction* is wrong before adding machinery to force it — inverting the build deleted more code than it added. (2) A proven external model (MoonLight) is worth adopting wholesale when it's the textbook approach (backward mapping + LUT bake), but write it fresh against your own structures (our CSR, our names) rather than porting. (3) Matrices compose the *affine* subset cleanly (Rotate is written as an explicit 2×2 matrix, the codebase's matrix reference) but can't express masks/tiles — so the coordinate fold is the general composition model, with a matrix-backed modifier as a special case the same interface hosts. - -## MoonLive: build around expressions + host-bound functions, not statement shapes - -The MoonLive live-script compiler (IR rung) was first built around the *statement shape* `setRGB(idx, r, g, b)` — the parser had per-slot rules (index could be `random16`, colours were literal-only) and the IR had an RGB-specific `Store` op baked into the **core**. Three product-owner remarks exposed the same root flaw: (1) `random16` only worked in the index slot, not any argument; (2) `random16(255)` capped at a byte because the index/colour validators conflated ranges; (3) the core compiler was light-domain-specific (`setRGB`/`fill`/`Store` hardcoded), violating *Domain-neutral core*. - -The fix was the ESPLiveScript / ARTI / doc-§3.4 model: **the core knows only expressions + a generic call mechanism; the host registers its functions in a builtin table.** Every argument parses as an expression (a literal or a nested call), so `setRGB(random16(256), random16(256), 30, 0)` works and a number is a uint16. `setRGB`/`fill`/`random16` — the LED *names* and the RGB meaning — live only in the light-domain registration (`MoonLiveBuiltins_light.h`); the core sees a neutral `BuiltinTable` of `{name → Call(fn ptr) | Inline(opcode tag)}`. A buffer writer is `Kind::Inline` (lowers to stores — the hot-path fast path, no per-pixel call); a pure helper is `Kind::Call`. A mechanised test pins the neutrality: with an empty table the core knows *no* functions, and a host can register an arbitrary name (`paint`) against the same machinery. - -Two codegen lessons surfaced fixing it: (1) **the live-vreg-across-call contract must hold for ANY expression, not a hand-ordered one** — once arguments can be calls, a value computed before one call can be live across a *second* call; the assembler's `call()` must save/restore the whole caller-saved register set (host: a full stp/ldp frame; Xtensa: s32i/l32i of the rotate-out registers a8/a9/a11, with the result stashed in a non-saved reg across the restore). (2) **register budget is real on the MCU** — fold the address into a dead vreg (WriteRGB writes into the index register after `index *= cpl`) rather than reserving fresh scratch, so a multi-call statement fits the small windowed register file. - -Lessons: (1) when a language "can't express X in slot Y," the fix is almost always *real expressions*, not a per-slot special case — build the general grammar once. (2) Domain-neutrality is testable: assert the core, given an empty host table, knows nothing — if it compiles a domain function, the domain leaked in. (3) The bound-function table is the same seam for *speed* and *neutrality*: the descriptor carries how it lowers (inline store vs call), so the core stays LED-free while the hot path stays inline. - -### MoonLive RISC-V backend + vreg reuse (the third ISA, and what it exposed) - -Bringing up the ESP32-P4 (RISC-V) backend — a third per-ISA assembler + lowering behind the same neutral IR — was mechanical *and* revealing. Mechanical: RV32 is uniform 4-byte instructions and a standard (non-windowed) call ABI, so the assembler is simpler than Xtensa's; every encoding was verified by disassembling the assembler's own output with `riscv32-esp-elf-as`/`objdump` before flashing (the same discipline that caught every Xtensa encoding bug). Revealing: the P4 was the first target where multi-call statements *failed to compile*, exposing two limits the host (14 regs) and Xtensa (12 regs, but the heaviest test was 2 calls) had masked. - -First, the **register file**. The front-end allocated a fresh virtual register per sub-expression and never reused it, so `setRGB(random16(..), random16(..), random16(..), 0)` needed more live vregs than the 12-register device pool — "codegen failed." The fix is the textbook tree-walk register stack: a free-list allocator where each argument temp is returned to the pool the moment its call consumes it, so a chain of N calls peaks at a handful of registers instead of growing 2N. `vregsUsed` (the lowering's reservation) is the high-water mark, which now *shrinks* because freed vregs are reused at low indices. This is the "concrete-first, the allocator arrives when a real script exhausts registers" point the design anticipated — and a 4-call statement is exactly that script. - -Second, the **code arena**. RISC-V's `call()` saves the full caller-saved set around each host call (~140 bytes), so four calls in one statement is ~600 bytes — past the original 256-byte staging cap. Rather than grow per-script (a moving target), the arena is sized once for the heaviest *realistic* single statement (four-arg-all-calls on the bulkiest ISA). Exec memory is cheap; a fixed worst-case cap is simpler and more predictable than dynamic growth. - -Lessons: (1) a third backend is the cheap insurance that the IR seam is real — if adding RISC-V had needed front-end changes, the seam was a lie; it needed none (only the new register/byte *limits* surfaced, which are target properties, not design leaks). (2) Register allocation is where "it works on my 14-register host" quietly diverges from "it works on the 12-register device" — bring up the smallest register file early. (3) Verify every emitted instruction against the real toolchain's disassembler before trusting it on hardware; it is faster than debugging a `StoreProhibited` on-device, every time. - -### MoonLive: size the exec block to the program, and bound every fixed table the codegen fills - -Two follow-on lessons after the engine landed, both about *fixed-capacity buffers in a code generator*. (1) **Allocate the live exec block at the emitted length, not the worst-case cap.** The first cut allocated `allocExec(kCodeCap)` (the 768-byte ceiling sized for a four-call statement) for *every* script, so a `fill(0,0,255)` — ~60 bytes of native code — reserved 768. The fix: `place()` allocates `allocExec(len)` (word-rounded for the IRAM 32-bit-store rule), so per-effect heap scales with the script; the staging *buffer* stays worst-case (it's transient stack), only the *retained* block is right-sized. Paired with it: the effect must **report that block** (`setDynamicBytes(codeLen())`) or the JIT'd code is invisible to the memory accounting — the UI card showed only `sizeof(MoonLiveEffect)` and none of the native code it allocated. (2) **A code generator's label/fixup tables need the same bounds check as its code buffer.** The assemblers guarded `emit()` against `kCap` from the start but let `newLabel()`/the branch-fixup enqueue write `labelPos_[]`/`fixups_[]` unbounded — a script with enough branches would corrupt memory past the arrays. The fix routes both through the same `overflow_` signal `emit()` already uses (a guarded `addFixup()` helper, a bounds check in `newLabel()`), so *every* fixed table the codegen fills fails cleanly, not just the byte buffer. - -Lessons: (1) when a JIT retains a fixed buffer per unit, size the *retained* allocation to the actual output and keep only the *scratch* at worst-case — and report the retained bytes, or the memory accounting lies. (2) a bounds check on the code buffer is not enough: audit *every* fixed-capacity array the generator appends to (labels, fixups, relocation tables) and route them all through one overflow signal — robustness is per-table, not per-generator. (3) these surfaced from a deliberate memory review at the small-language stage, before loops/expressions multiply the branch count — cheaper to pin the discipline now than after a real script first overruns a table. - -### MoonLive Stage 1: live controls via a 5th-argument data arena (not baked immediates) - -A scripted effect became *tunable*: `uint8_t speed = 50; // @control 0..99` surfaces a real uint8 MoonModule control. The load-bearing decision was **how a control value reaches the native code each tick without recompiling on every slider move**. A baked-in `Const` immediate is wrong (a slider change would need a re-emit); the answer is the **`kArg3`/`t` pattern, one slot over** — pass a pointer to a control-values *arena* as a 5th function argument (`CtrlFn = void(*)(buf, nLights, cpl, t, const uint8_t* ctrls)`), add an IR `LoadCtrl{offset}` that lowers to one byte load (`ldrb [x4]` / `l8ui a6` / `lbu (a4)`), and let the binding update an arena byte when a slider moves. The next `run()` reads it — live, no recompile. `t` already proved this exact mechanism works across all three ABIs, so a control pointer was a mechanical addition, not a new design. - -Three sub-decisions held against the principles: (1) **the arena lives in the engine as a stable-address, grow-capacity-only heap block** (`platform::alloc`, PSRAM-first), NOT a fixed member array (won't scale to big effects) and NOT a core sweep change (re-ordering `rebuildControls`→`onBuildState` to allow realloc-and-rebind would touch *every* module to serve one — violating *Core grows slower than the domain*). Because the sweep runs `onBuildControls` on every change, re-binding the control pointers there is automatic; because the arena only grows (never moves on a normal recompile), `&arena[i]` stays valid, and a kept control keeps its live value across a source edit. (2) **The binding owns the projectMM-facing details** — the engine exposes a *neutral* `DeclaredControl{name, type, min, max, def, offset}` list and the binding turns each into `controls_.addUint8` bound to the arena slot, keeping the engine domain-neutral; the binding copies the (non-NUL-terminated) name spans into a stable pool because the control descriptor borrows the name pointer. (3) A scripted control's value change returns **false** from `controlChangeTriggersBuildState` (no recompile); only `source` returns true. - -Two things surfaced building it. **The device JSON string reader (`parseString`) decoded only `\"` and `\\`, not `\n`** — yet the writer emits `\n` escapes — so a multi-line script arrived with a literal backslash-n and the `// @control` comment swallowed the statement. The fix is symmetry: the reader now decodes the standard escapes its own writer produces (`\n \r \t \b \f`), which benefits any multi-line text control, not just scripts. And **the `ok()` predicate**: adding the `ctrl_` (5-arg) compiled-source path meant `ok()` had to include it — the classic "added a code path, forgot the predicate that gates it" (compile returned true while ok() returned false, so every source compile looked failed). Pinned by the live-read unit test thereafter. - -Lessons: (1) when a value must change live without re-emitting code, pass it through *data* (an arena read each tick), never *code* (a baked immediate or self-modifying patch) — the read is one instruction and the arena is the seam the next capability (script state, palettes) reuses. (2) Reach for the mechanism the system already proves: `t` was the template, so the 5th-arg control pointer was low-risk by construction. (3) A reader that decodes a subset of the escapes its writer emits is a latent asymmetry bug — make them symmetric. (4) Adding a parallel code path means revisiting every predicate that gates the old paths: the `ok()` bug — a host-side compile-validation gap where the new `ctrl_` path compiled fine but `ok()` still tested only `fn_`/`anim_`, so every source compile read as failed — was caught by the host live-read unit test (the runtime test path's job), not by disassembly. (5) Separately, on the device codegen: register-map shifts (stealing the next arg register) are the riskiest part; disassembling every emitted instruction against the real toolchain before flashing is what confirmed `l8ui a6`/`lbu (a4)` land correctly. The reader-symmetry and runtime-predicate fixes are host test-path lessons; the register-encoding evidence is device-ISA — distinct, not the same fix. - -### Adding a new chip (ESP32-S31) + bumping the IDF: where the cost actually was - -Adding the S31 (a RISC-V preview chip) cost almost nothing in *our* code — it inherited the RISC-V MoonLive backend and all four LED drivers for free, because both gate on SOC capabilities (`__riscv`, `SOC_RMT_SUPPORTED`, `SOC_PARLIO_SUPPORTED`, …), not a chip-family flag. The real work, and the lessons, were in the **toolchain and the platform/board layer**: - -- **An IDF pin bump is a re-test pass, not a `git pull`.** Moving from an early v6.1-dev snapshot to the `release/v6.1` tip (the line carrying the S31 preview target) jumped ~4800 commits, which surfaced HAL *graduation* drift: the RMT HAL moved into its own `esp_hal_rmt` component, deleting the public `SOC_RMT_TX_CANDIDATES_PER_GROUP` soc-cap (now `RMT_LL_TX_CANDIDATES_PER_INST`), and Parlio renamed `sample_edge`→`shift_edge`. None of this is caught by building one target — a **re-verify gate that rebuilds every existing target on the new IDF** is what found them before they reached CI. The pin also needs the toolchain re-installed (`install.sh`) and the env's stale Python venvs refreshed; the build picks the most-recently-modified venv, which can be the wrong one. -- **A chip's *minimum-revision* default can silently brick field silicon.** v6.1 defaults the P4 minimum to rev 3.1; the bench (and field) P4 is v1.3, so the stock build refused to boot ("requires chip revision [v3.1-v3.99]"). The fix (`SELECTS_REV_LESS_V3` + `REV_MIN_0`) was only *found by flashing real hardware* — it builds clean either way. Hardware re-test, not CI, is what catches a min-rev trap. -- **Newest esptool-js ≠ best for flashing.** esptool-js 0.6.0 (the latest) deterministically corrupts a compressed flash — a P4 web-flash aborts at the *same* block (seq 38) every time, where 0.4.7/0.5.7 and the CLI all succeed (a fixed-seq failure, not random, rules out a transient). We pinned **0.5.7 — the version ESP Web Tools (ESPHome/WLED) ship** — over the newest, because *Common patterns first* applies to dependency versions too: match the battle-tested one the ecosystem runs, not the highest number. (0.5.x also moved `hardReset` off `ESPLoader`; driving DTR/RTS through the transport directly is version-agnostic.) -- **A new chip can be CLI-flashable but not browser-flashable**, and that's an *upstream* gap, not ours: esptool.py knows the S31 (since v5.2.0), esptool-js has no S31 chip class in any version. Worse, the S31's ROM magic collides with the classic ESP32's, so esptool-js would *mis-identify* it — a version bump alone won't fix browser flashing, it needs the secondary detection esptool.py already has. The catalog ships the firmware (`ships:True`) anyway: even with no web-flash, the published release asset is the device's **OTA** update channel, which is the more-used path (flash once via CLI, OTA forever). - -## Device discovery: UDP presence (we control it), mDNS advertise-only; the transport is per-ecosystem - -The DevicesModule refactor first reached for a UDP beacon, then swung to "mDNS is the standard, use it for discovery too", and finally — after bench measurement — landed on **UDP presence for discovery, mDNS for advertise-only**. The swing is the lesson: the "use the standard" instinct was right about *announce-to-foreign-apps* and wrong about *discover-peers*, and only measuring the wire separated the two. The lessons: - -- **Discovery transport is per-ecosystem, owned by the plugin — not one mechanism for all.** Use UDP where we control both ends (projectMM↔projectMM) or a foreign system already broadcasts (WLED's 44-byte packet on UDP 65506, on by default). Use mDNS only where a foreign *app* requires it (the native WLED app discovers us *only* via mDNS `_wled._tcp` — UDP can't reach it). The `DevicePlugin` seam reshaped from mDNS-shaped (`service()`/`classify(MdnsHost)`) to transport-agnostic (`discoveryPort()`/`classifyPacket(data,len,srcIp)`) so each plugin carries its own transport. -- **Two directions of one interop can need different transports.** We discover WLED over UDP 65506; WLED's *app* discovers us over mDNS. Same ecosystem, opposite directions, different mechanisms — the plugin owns both halves. Trying to force one transport for both is what kept the design fighting itself. -- **The mDNS query was the bug, and the fix was to stop querying mDNS, not to pace it better.** A blocking PTR query for a service the device *also advertises* exhausts the IDF mDNS pool (`mdns_querier: Cannot allocate memory` *with megabytes free*) and the device's own advertisement vanishes from peers. An earlier "throttle the query" / "go passive-browse" detour each half-worked (a passive browser can't discover — nothing re-announces unsolicited; a 75 s capture proved it). Moving discovery to UDP makes the self-query-disturbs-advertise bug **structurally impossible**: mDNS shrinks to advertise-only (`mdnsInit` announces `_http`+`mm=1` / `_wled`+`mac=`; the browse/query seam is deleted). -- **Misattributing a self-inflicted bug to "the network" or "the chip" wastes hours — read the device's own serial, and the prior art's actual behaviour, first.** Chasing this I twice concluded wrongly ("the network filters multicast", "ESP32↔ESP32 mDNS is an IDF limitation") — both were our own query destabilising the boards. The product owner's logic cut through it (*the WLED app discovers WLED devices on classic ESP32s, so classic mDNS demonstrably propagates — an invisible projectMM device is OUR bug*). And reading the prior art's behaviour, not assuming it, is what made the rest safe + precise: a WLED that receives our 65506 packet *lists* us, it does not *sync* to it (confirmed from MoonLight's working code); the WLED-app interop's missing `wifi{}` field was a one-line fix once we read the WLED-Android app's actual Moshi model, not a guess. -- **Commands split three ways by two questions — must it arrive? is it latency-critical? — not "all REST".** Discovery → **UDP presence**. Must-arrive + latency-tolerant config (brightness, presets, OTA) → **REST/TCP** (~10–50 ms, delivery guaranteed). Latency-critical + lossy-OK (time sync, live pixels) → **UDP stream** (~0.5–1 ms, broadcast to N; REST would be 10–100× too slow and visibly de-sync). The trap is routing a sync pulse over REST because "it's a command". -- **A plugin seam makes "interop" our own industry-standard hook-in point.** Foreign systems hook in as `DevicePlugin`s (the adapter pattern), not hardcoded branches — one file per ecosystem, no core edit. The discovery half ships now (projectMM + WLED); the command half is a reserved extension (*concrete first, abstract later*). The seam allows **hub-shaped** plugins (Hue: a bridge → child resources + auth), not just flat devices — designing that room in now (without building it) lets the harder tiers land additively. The differentiator: one UI across WLED + ESPHome + Hue, incrementally. - -**A blocking socket op on the single render loop freezes EVERYTHING — and an intermittent freeze hides as the wrong cause.** The desktop web UI went "slow like a cow in the mud" and FPS pinned at ~18 regardless of grid size. The HTTP/WebSocket server runs *inside* the desktop render loop (`HttpServerModule::loop20ms`), and the accepted client socket carried a **2 s `SO_RCVTIMEO`**: whenever a request's bytes hadn't landed the instant `accept()` returned (the GET line a TCP segment behind the handshake), `recv()` **blocked the whole loop up to 2 s**. The two symptoms were one bug — a 2 s stall per unlucky request (the sluggishness) and a fixed ~50 ms-per-connection drag that capped the tick at ~18 FPS (the "FPS independent of grid" tell, while the Noise effect itself measured 222 FPS — proof the cost was non-render). **Fix:** the accepted socket is persistently non-blocking, `read()` returns -1 immediately when nothing's pending, and `writeSome` stops toggling it back to blocking; ~2 s → ~13 ms, FPS tracks real render speed. Lessons: (1) **a single-threaded loop that services I/O must never make a blocking call** — a timeout isn't a fix, it's the size of the freeze; non-blocking + poll is the only safe shape. (2) **"FPS constant across workload" means a fixed non-workload cost dominates** — compare the effect's own FPS to the tick FPS; a large gap points away from rendering. (3) An *intermittent* stall (timing race on byte arrival) masquerades as environment noise — it sent the debug down "zombie process / CPU busy-spin" dead ends before a profiler (thread stuck in the loop, not in render) + a "does plain curl also stall?" test isolated it. (4) It was **pre-existing on main**, surfaced only because this branch added a second per-tick socket read (the WS poll) that made the race fire more often — a latent single-loop-blocking bug waits for the load that exposes it. - -## Docs from source: generate the technical layer from `///` comments, keep a thin hand-written summary — and the traps that cost real time - -The docs overhaul first tried to *shrink* the per-module `.md` files, then inverted to **generate** them: the `.h` is the single home of technical content (`///` comments), a Doxygen→moxygen pipeline renders one page per module, and a hand-written summary page (a 4-column table) is the only prose that stays. The old `.md` files are absorbed into the headers and deleted. The recognisable prior art is docs.rs / Sphinx-autodoc / Doxygen — a generated API reference under a hand-written guide — which is why it survived the *Common patterns first* bar. The lessons are mostly the traps found building it: - -- **A per-file external-tool invocation doesn't scale — batch it.** The first cut ran `npx moxygen` once per header (132×); each `npx` cold-start is ~1 s, so a build took ~150 s and pinned `mkdocs serve` unusable. One Doxygen pass over all headers + one `moxygen --classes` call, recombined per-header via the XML `<location>` map, is ~7 s (22×). The rule: when a build step shells out to an external tool per item, the per-invocation startup cost *is* the build time — invoke once over the whole set, split the output, never loop the process. -- **Writing generated files into a watched dir is an infinite-rebuild loop — write only on change.** The generated `.md` live under `docs/`, which `mkdocs serve` watches. An unconditional write bumps the mtime every build → the watcher rebuilds → which regenerates → which writes again. The serve churned forever and every page load landed mid-rebuild (~7 s). Fix: compare content and skip the write when identical, so mtime stays put and the watcher goes quiet. Any generator that emits into a watched tree needs this. -- **An unbalanced backtick in a `///` comment silently deletes the class from the docs.** One header (`ImprovProvisioningModule`) generated *no* page. Cause: `` `connected: `ssid`` `` — nested/odd backticks — made Doxygen report "end of C comment inside a `` ` `` block" and stop associating the comment with the class, so `HIDE_UNDOC_CLASSES` dropped it. The symptom (a missing page) was three steps from the cause (a typo in prose). Sweep every enriched header for `WARN`-level Doxygen output, not just build success; a comment typo is a *content* bug that no compiler catches. -- **A bare `<tag>` in a doc comment renders as live HTML and eats the rest of the page.** `<textarea>` / `<value>` in a `///` became an unclosed HTML element that swallowed everything after it in the rendered table. Wrap every `<…>` in backticks. (Same class of bug as the backtick one: prose that's valid English but invalid markup, invisible until rendered.) -- **A generator that's *present but failing* must fail loud, not return empty.** `gen_api.generate()` first returned `{}` on any error — so a transient `npx` registry hiccup would ship a docs site with **zero** API pages and no red X. Split the two cases: toolchain *absent* → graceful `{}` (a contributor without Doxygen still builds the rest); toolchain *present but failing* (or under-producing vs a floor) → raise. Silent degradation in a generated artifact is worse than a hard failure, because nobody notices until a reader hits a 404. -- **Enrichment by parallel agents needs an adversarial read, not just a build.** Worker agents enriched 19 headers; all compiled and generated. But the pre-merge reviewer + CodeRabbit still found real content bugs a build can't catch: a future-tense roadmap sentence (violates present-tense), a core-affinity claim describing plumbing that doesn't exist (inherited verbatim from the old `.md` — the inaccuracy pre-dated the enrichment), and an intentional default-value change (`TextEffect hue 0→128`) riding in unremarked and untested. "Compiles + generates" is table stakes; prose accuracy needs a human/reviewer pass, and an *intentional* behaviour change (even a good one) needs a test pinning it. -- **A migration is only net-subtractive once the old thing is deleted — stage the deletion, but don't call it done early.** Moving the old `.md` to `archive/` (instead of deleting) kept them for cross-check but left the branch net-positive and shipped a temporary migration banner on every generated page. That's fine as an explicit *stage*, but the banner and archive are debt with a name and a removal plan, not a resting state. Mark such scaffolding "temporary / removed at Stage N" at every site so the cleanup is mechanical. -- **A doc path duplicated in *code* drifts silently when the doc moves — gate it, don't trust it.** Each module's `main.cpp` `registerType("Name", "docPath")` builds the in-UI help link, but nothing validated the docPath resolved. A docs rename (deleting per-driver pages, flattening `light/effects/effects.md` → `light/effects.md`) left ~10 of them 404-ing in the running UI — invisible to the desktop build, ctest, and the docs build alike, caught only by a reviewer eyeballing the diff. The fix is a `check_specs.py` gate that parses every docPath and asserts it resolves to a real page **and** `#anchor`. Rule: a path (or any fact) that lives in *code* but points at *docs* is a cross-boundary duplication with no compiler guarding it — the moment you can write a check that resolves it against ground truth, that check is cheaper than the drift it prevents. -- **Verify a generated anchor against the *built HTML*, never a re-derived slug.** Auditing those docPath anchors, an agent computed each `#anchor` with the stock python-`markdown` `toc` slugify and declared 52 links "broken" (`#fire` should be `#fire-2d`, etc.). All 52 were actually *correct* — the catalog-table build hook emits its own `<a id="fire">` anchors from the card name, overriding the heading slug the standalone slugify guessed. Applying the agent's "fixes" would have broken 52 working links. Rule: when a build hook post-processes a page, the only authority for what anchors exist is the *rendered output* (`grep id=` the built `.html`), not a reimplementation of the slug algorithm — the hook is free to disagree with the default, and here it did. - -## ESP32-S31 RGMII Ethernet bring-up: four bugs between "code compiles" and "link up" - -Bringing up the S31's on-chip 1 Gb RGMII Ethernet took four fixes, and none of them was the C++ — each was a layer *below* the feature code that a build can't catch. The general lesson: **for a hardware bring-up, "it compiles" tells you almost nothing; the truth is only in the boot log on the actual board.** - -- **A `MM_NO_ETH`-style capability gate keyed on a *filename* silently omits a new board.** `build_esp32.py` decided "does this firmware have Ethernet?" by pattern-matching the sdkconfig fragment *filename* for `.eth`. The S31 enables its EMAC in `sdkconfig.defaults.esp32s31` (no `.eth` in the name), so the gate said "no Ethernet" → compiled the `ethInit()` stub → the board booted with **zero** eth log and fell back to WiFi. The symptom (silent WiFi fallback) was maximally far from the cause (a filename heuristic in a build script). Rule: a gate that asks "does X enable feature Y?" must read what X *contains* (`CONFIG_ETH_USE_*=y`), never what X is *named*. Names drift out of the heuristic's blind spot the moment a new case doesn't follow the naming convention. - -- **A schematic/reference pin table is a hypothesis until the boot log confirms it — and ours was systematically off by one.** The reference-doc pin table (hand-transcribed from the schematic) had MDC/MDIO/TXD all one GPIO low. Wrong data pins failed loudly (`invalid TX_CTL GPIO number`), but wrong *MDIO* failed as "No PHY device detected" — three inference-steps from the typo. The chip's own IDF IO_MUX table (`esp32s31/emac_periph.c`) is the authority for RGMII data pins (they're fixed pads, not arbitrary GPIOs); trust it over any transcribed table, and verify the SMI pins against IDF's `ETH_ESP32_EMAC_DEFAULT_CONFIG` for that chip. Use PHY-addr auto-detect (`-1`) so a wrong strap assumption can't mask a working bus. - -- **A shared clock is a contended resource — the RGMII 125 MHz Tx clock can't come from a PLL that's already spoken for.** The default (AUTO) sourced the Tx clock from the MPLL, but PSRAM already ran the MPLL at 400 MHz (no integer path to 125), and CPLL couldn't synthesise 125 MHz on the 40 MHz XTAL grid either. Only the *fractional* APLL (built for exact frequencies) works. The lesson beyond Ethernet: on an SoC where one PLL feeds multiple peripherals, "pick a clock source" is a *conflict-resolution* decision, not a default — check what already owns each PLL before claiming one. - -- **Changing a Kconfig *choice* in a defaults fragment needs a clean build.** Editing `CONFIG_ETH_EMAC_RGMII_TX_CLK_SRC_*` in the fragment and doing an incremental build silently kept the *old* choice (the build dir's `sdkconfig` already had a value; defaults don't override an existing one). Two flash cycles were spent "testing APLL" that were actually still CPLL. When a sdkconfig *choice* (not a plain `=y`) changes, `rm -rf` the build dir or the test is a lie. - -## W5500 SPI Ethernet (LightCrafter / SE16): interrupt service, IDF-v6 config, and a per-board reset pin - -Bringing up the two Limpkin ESP32-S3 boards' external W5500 (SPI) Ethernet surfaced three traps between "the driver installs" and "the link is up" — each far from its cause, each a general rule. - -- **Interrupt mode needs `gpio_install_isr_service()` — miss it and the IRQ silently degrades to slow polling, not an error.** The device model gives the W5500 a real INT pin (SE16 GPIO18, LightCrafter GPIO45, per the board schematics). But the IDF W5500 driver registers its handler with `gpio_isr_handler_add()`, which requires the per-pin ISR service already installed — and nothing in the platform layer installed it. The `add` failed with a one-line `E gpio: GPIO isr service is not installed`, the interrupt never fired, and RX was serviced only on the driver's slow fallback → a ~1 s *sawtooth* ping latency (packets serviced in descending batches, the poll-cycle signature). The symptom read like a *network* problem; the cause was a missing one-liner. Rule: a driver that takes a GPIO interrupt needs the ISR service installed first — do it once before the driver init (`ESP_ERR_INVALID_STATE` = already installed = fine). The `int_gpio=-1` polling fallback is a *workaround* that hides this, not the fix. - -- **IDF v6's W5500 driver rejects `int_gpio_num < 0` unless a poll period is also set.** The "no interrupt" fallback isn't just `int_gpio_num = -1` — that alone returns `esp_eth_mac_new_w5500(): invalid configuration argument combination` and the whole eth init fails, cascading the board to WiFi. You must *also* set `w5500_config.poll_period_ms` (e.g. 10). The pair is the API contract; half of it is a config error. - -- **A failed Ethernet init that cascades to WiFi *masks* the eth bug — "why are we on WiFi?" is the tell.** Both bugs above manifested as the board quietly running on WiFi with a healthy HTTP server and no eth log at the default verbosity. The robustness cascade (eth fails → WiFi) is correct behaviour, but it turns a loud init failure into a silent capability loss. When a board that *should* be on Ethernet is on WiFi, that mismatch is the signal to read the eth init log, not a benign state. - -- **A `0x00` "version mismatched" read from an SPI PHY means the chip is held in reset — check for a board reset-release pin.** The LightCrafter's WIZ850IO module holds its own `nRST` until GPIO3 is driven HIGH (it also gates RS485_DE / VBUS_DET on that board). Without releasing it, the ESP32 read `0x00` from the W5500's version register (`expected 0x04, got 0x00`) → "verify chip ID failed". The SE16's W5500 self-resets, so it needed no such pin — which is exactly why this was board-specific and easy to miss. Setting `ethRstGpio` (→ the IDF PHY driver's `reset_gpio_num`, which runs the assert-low/release-high sequence) fixed it. Rule: an all-zeros register read from an SPI peripheral is "the chip isn't talking", and the first suspect after wiring is a reset/enable line the board expects the host to drive. - -## Extract the generic "set a control" primitive to the Scheduler, not per-input seams - -Adding IR remote control raised "how does one module drive another module's control?" The wrong answer is a bespoke seam per target (`Drivers::adjustBrightness`, then `Drivers::setPalette`, … — N one-offs, the "N disparate inputs" the LightsControl backlog warns against). The right answer already existed, half-hidden: the WLED-app bridge sets brightness via `applySetControl(module, control, value)` — the generic find-module → validate → apply → onUpdate → persist → conditional-buildState path. It lived *private on HttpServerModule*. - -The move: **lift `setControl` onto the `Scheduler`** (which owns the module tree + the persistence hook), and make HttpServer's `applySetControl` a thin result→HTTP-status mapper. Now IR, HTTP, Improv, and the WLED bridge all compose against one control-agnostic primitive — brightness, palette, or any future control is the same call with different arguments; adding an input never adds plumbing. HttpServer shrank ~54 lines (dedup, not addition). The reach mechanism is the `FilesystemModule::instance_` singleton pattern (`Scheduler::instance()`), so a factory-created module (IrModule) gets it without a per-module inject. General rule: when a second caller wants a capability that a transport layer implemented privately, the capability belongs in core (the tree owner), and the transport keeps only its status-mapping — the same shape as the DevicesModule/AudioModule "core owns the hard part, the module leans on it" split. - -## An RMT/IR done-callback runs in ISR context: signal only, decode on the task - -The first NEC-decoder version crash-looped the board (USB dropped, port re-enumerated) because the RMT `on_recv_done` callback both **decoded** the frame (calling non-`IRAM_ATTR` helpers) and **re-armed** `rmt_receive()` — from interrupt context. On ESP32, an ISR that jumps to flash-resident code while the cache is disabled, or re-enters a driver call, faults. The existing working RX capture (`rmtWs2812RxCapture`) already showed the safe shape and I didn't follow it at first: **the ISR only records the symbol count and signals a queue; the decode + re-arm happen on the task** (in `irRead`, called from the render loop). Rule: an RMT/GPIO done-callback is ISR context — it may touch only its stack, the passed event data, and an ISR-safe FreeRTOS primitive (`xQueueSendFromISR`); everything else (parsing, logging, re-arming the peripheral) moves to the waking task. When in doubt, copy the codebase's already-proven callback, don't re-derive it. - -## An MQTT (or any integration) topic identity must be a stable hardware id, never the editable name - -The MQTT module first derived its topic prefix from the user-editable `deviceName` (`projectMM/<deviceName>`) — attractive because it reads well and a rename "follows" on the device. On the bench a rename (ShellyOne → ShellyTwo) instantly repointed every topic and orphaned the Homebridge config: the hub kept publishing to the old topics, the device now listened on new ones, and the light went "not responding." Researching WLED (which projectMM already emulates), Tasmota, ESPHome, and Home-Assistant MQTT discovery found the finding unanimous: **the machine-facing identity is anchored to a stable hardware id (MAC / chip-id), and the human display name is a separate field.** WLED's default device topic is `wled/<last6-of-MAC>`, rename-stable; the friendly `serverDescription` is decoupled. HA discovery *requires* a stable `unique_id` precisely so a friendly-name change doesn't break automations, and explicitly forbids device name / hostname as identity (they change). - -The fix: topics derive from `projectMM/<last6-of-MAC>` (stable for the chip's life), and the friendly `deviceName` rides a separate retained `<prefix>/name` topic. A rename now updates only the display label; control never breaks. General rule: **when a config value is a control-plane identity that external systems bind to (a topic, a discovery id, an API key path), derive it from something immutable and keep the human-readable name a separate, published-but-non-identifying field.** The convenience of "the name is the identity" is a trap — it couples a display concern to a wire contract. - -## Don't promote one specific board's pins to "the chip default" - -`NetworkModule::ethType` defaulted to a per-chip `ethConfigDefault`, and the classic-ESP32 / P4 / S31 entries in that table were, on inspection, **one specific board's wiring each** (the classic default was "e.g. Olimex," the P4 default was the Waveshare NANO's pins, the S31 default was the Function-CoreBoard's) dressed up as a chip property. Two failures fell out: a WiFi-only board (the LOLIN-D32 Shelly) wasted an Ethernet-PHY init on hardware it lacks, and — worse — the QuinLED Dig-Octa, which sets custom MDC/MDIO but has *no software eth reset* (GPIO5 is an LED output on that board, per its published pinout), would have inherited the classic default reset=GPIO5 and driven an **LED pin as an Ethernet reset**. The reviewer's instinct ("are these industry standards, or a device we happened to use?") was exactly right. - -The fix: `ethType` defaults to **None** — Ethernet is opt-in, and every board that has it declares its own `ethType` + board-wiring pins explicitly in `deviceModels.json`; a validator (`check_devices.py`) enforces "ethType set → board-wiring pins present." The pin *defaults* still seed a board that opts in, but nothing relies on them silently. General rule: **a "default" that is really one board's values is bespoke masquerading as standard (a § Principles violation); make the capability opt-in and require each consumer to state its own values, so a missing declaration fails loudly instead of inheriting a stranger's wiring.** - -## A new firmware default only reaches a factory-fresh device; persistence wins on an upgrade - -After changing the `ethType` firmware default to None and reflashing an already-provisioned board, the device *still* reported the old value — because persistence (`FilesystemModule` overlays every saved control at boot) restores the value captured during the earlier provisioning, and a plain `flash write` keeps the filesystem. The same shape bit the MQTT prefix: an old-firmware device carried a stale stored prefix a reflash didn't clear. This is *correct* persistence behavior (saved config wins over a constructor default), but it means **a default change is invisible to any device that has already run** — only an erase-flash or an explicit re-set surfaces it. General rule: when you change a default to fix a bug, remember the fix lands only on factory-fresh devices; an already-provisioned fleet needs an explicit migration (re-inject, an erase, or a one-shot "clear if stale" on upgrade) — and when debugging "my fix didn't take," check the persisted `.config/*.json` before the firmware default. - -## Fixing "blocking in the tick" at the connect misses the same violation at the write - -The MQTT client's first version blocked the render loop twice — a synchronous `connect()` (with DNS) *and* an unbounded blocking `write()` retry — both inside `loop1s`, which runs in `Scheduler::tick`. The first review caught the connect and it was reworked to non-blocking `connectStart`/`connectPoll`; the *re-review* caught that `conn_.write()` was still there, one layer up: a zero-window broker (accepts TCP, stops reading) fills the send buffer and the next `write` never returns — the render loop freezes permanently, and the keepalive timeout can't even fire because control never comes back. The fix mirrored the connect one: sends go through a non-blocking `writeSome` with an all-or-reset rule (control packets are ≤256 B, so atomic-send is realistic). General rule: **a "no blocking in the hot path" audit must sweep every syscall the path can reach, not just the obvious one** — connect, DNS, read, *and* write. Fixing the loudest blocker while an equally-blocking sibling survives is a partial fix that reads as complete; the re-review exists to catch exactly that. diff --git a/docs/history/lessons.md b/docs/history/lessons.md new file mode 100644 index 00000000..b7af8ef9 --- /dev/null +++ b/docs/history/lessons.md @@ -0,0 +1,236 @@ +# Lessons + +Hard-won debugging lessons and gotchas, recorded with the code that proved them. The PR-merge *carry-forward* gate writes new entries here (CLAUDE.md § Lifecycle Events). This is the **lesson** record: a bug, its root cause, and the fix. Three neighbours hold the other genres, and a lesson belongs in whichever fits: + +- A genuine architectural **decision** (chose approach A over B/C) is an [ADR](../adr/README.md), immutable and kept. +- A durable **rule** that graduated from a lesson lives in [CLAUDE.md](../../CLAUDE.md) or [coding-standards.md](../coding-standards.md), not here. +- The forward-looking **design intent** of a feature is a [plan](plans/README.md). + +Entries run oldest-first by branch. A lesson fully absorbed into a rule doc or the code is pruned (per *Mandatory subtraction*): the git history is the permanent record, this file is the working narrative on top. + +## Foundational lessons (early projectMM, pre-branch-log) + +The debugging lessons from the first iterations that a present-tense doc can't hold — the architecture-shaping retrospectives they came with have been absorbed into [CLAUDE.md](../../CLAUDE.md) and [architecture.md](../architecture.md): + +- **Verify a protocol against the spec's test vectors, not internal consistency.** A one-character typo in the RFC 6455 WebSocket magic GUID passed every internal check (the SHA-1 was self-consistent) but the browser rejected the handshake — silently. When implementing a wire protocol, assert against the RFC's published test vectors. +- **`freeHeap()` vs `freeInternalHeap()` on PSRAM devices.** `freeHeap()` returns internal + PSRAM combined, but the `HEAP_RESERVE` guard must use `freeInternalHeap()` — stack, HTTP, and WiFi need *internal* RAM, not PSRAM. Checking the combined figure lets internal RAM exhaust while the reserve check still passes. +- **Delete copy/move on any class owning raw memory (Rule of Five).** `MoonModule` and `ControlList` owned raw pointers but had implicit copy/move — a double-free waiting to happen. A class that owns a raw resource declares (or deletes) all five special members. +- **`setName` must copy, not store a pointer.** HTTP module creation stored a pointer to a stack-local name buffer; after the call returned the name was garbage. Owned strings are `char[N]` + `memcpy`, never a borrowed pointer. + +## Lessons from the next-iteration branch (plans 08-12) + +The branch covering SystemModule/NetworkModule, persistence, the UI rewrite, the eth-only build, and the side-nav. + +- **Plan-09's persistence abandonment was a success.** The first attempt (~1700 LOC) didn't pay for itself; ~700 LOC of foundations (partition scheme, platform fs API, MoonModule additions) were kept, the rest dropped, and plan-10 succeeded with a smaller control-list-driven design. +- **ESP-IDF v6.x removes WiFi via `EXCLUDE_COMPONENTS`, not Kconfig.** `CONFIG_ESP_WIFI_ENABLED=n` is silently ignored in v6.x. The eth-only profile excludes `esp_wifi`/`wpa_supplicant`/`esp_coex` (NOT `esp_phy`, the EMAC needs it) plus an `MM_NO_WIFI` define gating `if constexpr` branches. +- **The render tick collapsed on a blocking 49 KB preview WebSocket write** spinning `vTaskDelay` until lwIP drained. Fix: non-blocking scatter-gather write + downsample to fit the send buffer. +- **WiFi UDP is ~4× the Ethernet per-packet cost** (WiFi CSMA/CA, retries, rate adaptation, not a code regression): ArtNet at 16K lights is ~7 FPS on WiFi vs ~19 on Ethernet. +- **Two no-op wrappers were found and removed** by the Reviewer (`HttpServerModule::parseJsonString` re-namespacing `mm::json::*`; `NetworkModule::rebuildLocalControlsAndPipeline` whose name contradicted its body). +- **An unescaped control value containing `"` or `\` produced malformed JSON** in `/api/state` and the persisted config; caught by a round-trip test, not the reviewer. +- **A mid-implementation design change (password length-only → XOR+base64) left one header's comment stale**, it documented a security property the code no longer provided. + +## Lessons from this branch (plans 13-16) + +Plan-13 (nest child cards), plan-14 (replace-type button), plan-15 (stream `/api/state`), plan-16 (Layouts/Layers/Drivers reshape), plus mid-branch fixes (effect freeze, Int16 zero-corruption, Layouts disable crash, status slot, layers reorg, FilesystemModule + Scheduler split). + +- **`ControlDescriptor.min/max` are `uint8_t` and can't bound wider widths.** Applying them to `Int16`/`Uint16` clamps every value into `[0..0]`; `addInt16`/`addUint16` leave bounds at `0,0` because the slot can't represent the wider range. The load path silently zeroed every Int16 control on every reboot ("Layouts cannot be activated after reboot"); reverted with a comment in `Control.h` on why bounds stay 0,0. +- **Per-tick integer division rounded four effects' animation rate to zero on fast devices.** `phase += dt * bpm * 256 / 60000` truncates to 0 when `dt < 234/bpm` ms (desktop `dt ≈ 0..1ms`; ESP32 at 16K LEDs is fine). Fix: keep the raw `dt * bpm` numerator in the accumulator, divide at the read site (NoiseEffect's pattern). Now CLAUDE.md § Hard Rules: "effects must run at every grid size and tick rate." +- **`Layer::onAllocateMemory` early-returned on empty layouts without resetting its LUT or buffer**; Drivers then reallocated the output buffer to 0 bytes while the stale LUT pointed at 16K destinations, and `blendMap` dereferenced null. "Nothing to do" branches must still reach a consistent zero state. +- **HTML5 `dragstart`'s `e.target` is always the draggable element**, so `e.target.closest(".child-class")` in `dragstart` never matches (`e.target === card`). Use the *mousedown* target instead; toggle `draggable` on mousedown, with a `touchstart` mirror. Shipped silently because the exclusion list happened to cover `<input>`; surfaced only with a `<details>`/`<summary>` control. +- **A single `warning` slot conflates info, degradation, and failure.** Three levels (`Status`/`Warning`/`Error`) earn their keep once more than one module produces non-degradation messages; wire format mirrors the C++ enum lowercased (`"status"`/`"warning"`/`"error"`), the field-name/severity `status` collision documented at the introduction site. +- **The lifecycle events are "commit" and "merge"; "push" has no work of its own.** A separate Push event for the Reviewer bred the "address-reviewer" noise-commit anti-pattern; the Reviewer at Commit cost 5-7 min per commit. Final shape: reviewer at PR-merge over the whole branch diff, on-demand pre-commit as the safety valve. +- **The reviewer agent flags real findings AND wrong ones (~30% valid across three passes here).** Wrong ones misread line numbers, repeated an accepted finding, or proposed re-introducing a fixed bug (the `c.min`/`c.max` Int16 finding twice). "Skip with one-line reason in the commit body" is the honest response, and the reason becomes the audit trail. +- **Top-level system docs are flat (`architecture.md`, `coding-standards.md`, `building.md`, `testing.md`); per-module specs live under `docs/moonmodules/`.** The earlier `architecture.md` + `architecture-light.md` pair was asymmetric and pulled toward more suffixes; merging into one `architecture.md` with `# Core`/`# Light domain` sections matches every well-known project. Subfolders only kick in under `moonmodules/` where there's a plural of each kind. +- **`util/` and `modules/` buckets were rejected.** A `util/` bucket groups by file-shape not concern (each header already names what it does); a `modules/` bucket needs a plural of each kind to earn its keep, but the four system services are singletons. The right cleanup was header-only → `.h`+`.cpp` splits, done lazily. +- **`.claude/scheduled_tasks.lock` is harness runtime state**, not project content. Ignore `.claude/*.lock` (a broader pattern than per-file); the single-file `.claude/settings.local.json` ignore was too narrow. + +## Lessons from this branch (plans 17-23) + +The plan-18 branch landed plans 17 + 18 + six unplanned follow-ups (19, 19.1, 20, 21, 22, 23). + +- **GitHub Pages CDN returns no `Access-Control-Allow-Origin`, so cross-origin fetches of release-asset `.bin` files fail**, CORS-on-static-files isn't fixable from your side. Plan-18 pivoted to self-hosting the last N releases' binaries on Pages content (same-origin), rather than a third-party CORS proxy. +- **Chrome's mixed-content policy blocks an HTTPS Pages page from `fetch("http://192.168.1.X/…")`** even with `Access-Control-Allow-Origin: *`; the block happens before the request leaves the browser. Plan-20's Diagnose feature moved to the device UI (same-origin, mixed-content moot). +- **ESP Web Tools' rich panel ("Visit Device" / "Configure Wi-Fi") is in-browser-session-only**, the device URL is browser-side memory, not asked-back from the device. A device-side `GET_CURRENT_STATE` → URL follow-up (ESPHome pattern) surfaces the data but does NOT change the third-party tool's UI. +- **`improv_provision` returns `ERROR_UNABLE_TO_CONNECT` when WiFi STA is already connected, by design** (protects large installs from a scan-induced ArtNet drop). The browser shows "Unknown error (255)" because Improv's error mapping carries no human reason; document the rejection at every layer the user meets it. +- **Per-board build directories land as `build/<board>/`, not `<chip>/build/<board>/`.** `build/esp32-<board>/` keeps every target under one root shared with desktop targets (`build/macos/`, …); the doubled `esp32-<board>` prefix is intentional. One root, one cleaner. +- **`idf.py -B <dir>` needs `-DSDKCONFIG=<dir>/sdkconfig` to isolate per-build-dir sdkconfigs.** Without it, idf.py writes `<project>/sdkconfig` shared across boards, tripping "project sdkconfig was generated for target X, CMakeCache contains Y" on the second board build. Retrofitted to build_esp32/flash_esp32/collect_kpi. +- **Adding an Improv child to NetworkModule (plan-21) reverted, then resolved by the Peripheral-role branch.** The base lifecycle *does* propagate every callback to children; a child misses one only when the parent overrides that method and forgets to chain to base. Fix is per-parent (SystemModule chains `setup()`/`loop1s()`), not a scheduler refactor. `unit_SystemModule` pins it; the general rule lives in [coding-standards.md § Override-and-chain convention](../coding-standards.md#override-and-chain-convention). +- **Split `platform_esp32.cpp` at public-API boundaries, not section banners.** Improv + OTA + LittleFS each own private state and talk back only through `platform.h`, so they split cleanly; Network stayed put because Eth + WiFi + sockets + mDNS share eight file-scope variables. +- **Desktop's `platform_desktop.cpp` is correctly asymmetric with ESP32's**, its OTA/Improv/FS sections are 6-line stubs, so per-subsystem files would be all overhead. Symmetry across platforms is a heuristic, not a rule. +- **Nightly builds live in their own workflow** (`nightly.yml` tags `nightly-YYYY-MM-DD`, `release.yml` builds via tag push): zero duplication of the build matrix. The skip-on-no-change check costs ~2s on quiet days. +- **`workflow_dispatch` reads the workflow YAML from the default branch, not the dispatched branch.** Cost a CI cycle on RC2: dispatched against `plan-18` with a tag invalid against `main`'s older `release.yml`. `inputs.tag` arrives correctly but the consuming logic is whatever main has. +- **The reviewer agent's PR-merge job is architectural drift across N commits, not line-level bugs** (CodeRabbit's job). Two agents, two scopes. +- **A 13-commit branch is the upper end of what one merge should carry**, the merge train is heavy and the reviewer's job gets harder as commits stack. Aim for "ship 3-4 plans, merge, start the next branch." This one worked because the plans were mostly independent. +- **MoonModule's asymmetric lifecycle propagation was historical, not principled.** `setup`/`teardown`/`onBuildControls`/`onAllocateMemory` propagated to children, but `loop`/`loop20ms`/`loop1s` defaulted to empty no-ops, so every container duplicated a 5-line per-child block. A shared `tickChildren` helper (gated by `!respectsEnabled() || enabled()`, per-child timing accumulated) closed it; leaf modules pay one predicted-not-taken branch. PC tick stayed 55-160 µs. The parked Plan-21 move became four lines. +- **Override-and-chain: option A for `loop` (parent prepares, then chain so children read fresh state, `Drivers::loop` runs `blendMap` before children read `outputBuffer_`); option B for `setup` (chain first so children init before the parent depends on them); `teardown` chains late.** The conventions live in [coding-standards.md § Override-and-chain convention](../coding-standards.md#override-and-chain-convention) and [architecture.md § Lifecycle propagation to children](../architecture.md#lifecycle-propagation-to-children). +- **Control-change reactions are a three-tier split; the coarse-grained rebuild debt is closed.** `handleSetControl` now: (1) always calls `MoonModule::onUpdate(controlName)`; (2) calls `scheduler_->buildState()` only when `controlChangeTriggersBuildState(controlName)` is true (default false, overridden on `LayoutBase`/`ModifierBase`); (3) the sweep reaches each `onBuildState()`. Slider drag no longer triggers a tree-wide realloc. Mirrors MoonLight's `onUpdate`/`requestMappings`/`onSizeChanged` split (confirmed against MoonLight source); the verb is "build" not "rebuild" (idempotent), so `onAllocateMemory` became `onBuildState`. +- **Output correction (brightness/reorder/white) is a per-driver stage shared via the Drivers container.** ArtNet was sending raw bytes (no brightness/order/white), a gap. Drivers owns a `Correction` (256-entry brightness LUT + channel-order table + derive-white flag) and hands each child a `const Correction*`; brightness applies before white derivation. The field is `briLut` not `gammaLut` so gamma folds in as a fill, not a rename. Follows MoonLight's driver-edge per-channel LUTs. +- **Three call sites carried their own 50-60-line `switch (c.type)` over `ControlType`** (`HttpServerModule::writeControls`, `FilesystemModule::writeValue`, `scenario_runner`), which drifted (the scenario runner stopped recognising new types). Extracted to free functions in `Control.cpp` (`writeControlValue`/`applyControlValue`/…); `JsonSink` gained a fixed-buffer mode so the FS path shares the serializer without a per-value alloc, and an `ApplyPolicy` param (`Strict`/`Clamp`) preserved tolerant load. Codified in [coding-standards.md § Per-type behaviour lives with the type](../coding-standards.md#per-type-behaviour-lives-with-the-type). +- **Local Improv testing closed a high-friction dev loop.** Before, verifying Improv end-to-end meant tag a release → CI → deploy Pages → flash from the live installer, burning a release tag per iteration. `preview_installer.py`'s flash-ready mode stages local `build/esp32-*/projectMM.bin` under `releases/latest/`, generates Pages-relative manifests, and serves at `localhost:8000` (Web Serial works on localhost without the secure-origin gate). Paired with `improv_smoke_test.py` (probe + provision + LAN reachability). + +## Lessons from this branch (Board injection follow-ups) + +- **`src/ui/release-picker.js` is now `src/ui/install-picker.js`** (symbol `installPicker`, C array `installPickerJs`), renamed once the picker grew from "pick a release" to "pick release + board + firmware + install". A wide but mechanical ~20-file sweep. Recorded so a search for the old name lands here. +- **Why the web installer dropped ESP Web Tools for a custom orchestrator.** ESP Web Tools 10.x's `<esp-web-install-button>` held the SerialPort exclusively across flash + provision and fired `state-changed` inside its dialog's shadow DOM, so post-PROVISIONED board injection was structurally impossible and `devices.js`'s auto-add broke. Owning the SerialPort end-to-end in `install-orchestrator.js` lets both fixes land in one place, and each future injectable adds one vendor command ID + one dispatcher case. + +## Lessons from the ESP32-S3 N16R8 (DevKitC) enablement branch + +Three non-obvious failures adding native-USB S3 support, all with the shape "symptom looks like X, root cause is elsewhere." + +1. **USB-Serial-JTAG ≠ UART0 on ESP32-S3.** The Dev's USB-C wires through the built-in USB-Serial-JTAG peripheral, not an external bridge to UART0. IDF's secondary console (`CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG=y`, default for S3) mirrors stdio both paths, so boot logs appear and the developer assumes UART0 works. The Improv-listener-on-UART0 was deaf. Fix: install BOTH drivers and read both transports; `#if SOC_USB_SERIAL_JTAG_SUPPORTED` keeps classic builds clean. Don't make it compile-time-per-board, the same binary should work with either wiring. +2. **CORS preflight is silent on the client side.** A cross-origin POST with `Content-Type: application/json` triggers an OPTIONS preflight; if the device returns 405 to OPTIONS, the browser silently drops the POST, no error, no network line, no console message. Burned a session diagnosing what looked like a fan-out bug. Fix: always implement OPTIONS (204 + `Access-Control-Allow-Origin: *`, `-Methods`, `-Headers: Content-Type`); verify with `curl -X OPTIONS …` returning 204 not 405. +3. **Cached "last applied" TX-power went stale when the WiFi stack restarted.** `appliedTxPowerSetting_` skipped redundant `esp_wifi_set_max_tx_power` calls, but an AP→STA cascade / reconnect / AP shutdown resets the radio's TX-power while the cached value stayed equal to the desired one, so `syncTxPower()` short-circuited and the cap never re-landed. Fix: every `wifiStaStop()`/`wifiApStop()` also invalidates the cache (`= -1`). + +## Core/light type boundary: light_types.h split + preview decouple + +`src/core/types.h` had grown into a junk-drawer of light-domain types (`nrOfLightsType`, `CoordCallback`, `defaultGridSize`, `HEAP_RESERVE`, `lengthType`) alongside `Dim`. Split so each symbol lives with its owner, done in three passes: + +- **Pass 1 (no core consumer):** `nrOfLightsType`/`CoordCallback`/`defaultGridSize` → `src/light/light_types.h`; `HEAP_RESERVE` → `platform.h` (a platform memory constraint, not Layer's, even though Layer was its only caller, ownership follows concept, not call count). +- **Pass 2 (`lengthType`, by removing its incidental core consumers):** `Control.h` only *mentioned* it in a comment; `HttpServerModule::put16` only took it because it serialised `PreviewFrame`, a light struct sitting in core. Introduced `BinaryBroadcaster` (core interface, ~6 lines: "send these bytes to all WS clients"); moved `PreviewFrame.h` → `src/light/`, where `PreviewDriver` now owns the 13-byte header and *pushes* bytes (replacing the old `PreviewFrame::ready` poll). `lengthType` → `light/light_types.h`, zero core users left. +- **Pass 3 (`Dim`, and the deletion of `core/types.h`):** `ModuleFactory::registerType<T>` probed `dimensions()` naming `Dim` in the constraint, but the next line did `static_cast<uint8_t>(...)`, the name was incidental. Loosened the probe to `requires { static_cast<uint8_t>(t.dimensions()); }`; `Dim` moved to `light/light_types.h`, `core/types.h` was empty and **deleted**. Verified `/api/types` still reports dim 3/2/0 unchanged. + +Every tie here turned out incidental and severable: a comment, a serializer following a misplaced struct's field type, and a SFINAE constraint that named a type it immediately discarded. End state: no `core/types.h`; core names zero light types. + +## A static "current instance" pointer needs re-election, not just claim/vacate + +`AudioModule::latestFrame()` hands effects the active mic via a process-wide `static AudioModule* active_`. "setup() claims, teardown() vacates" silently breaks with **two** mics: removing the one holding `active_` leaves the seat null while a second running mic sits captured-but-unread, and every audio effect goes silent. Fix is a three-part protocol: the first live module claims in `setup()`, `teardown()` vacates, and any running module **re-claims an empty seat in `loop()`**, so the survivor takes over on its next tick for any add/remove order. (`unit_AudioModule` pins the two-mic first-wins + re-election.) A tempting CodeRabbit "fix" (gate the claim on `inited_`) was rejected: a claimed-but-uninited module publishes valid *silence* (the documented contract), and a mic-less board running `simulate` publishes synth frames without being `inited_`. + +## An "Effect-role" module is not guaranteed to be an `EffectBase` + +DemoReel hosts a child effect and needs its `dimensions()` to extrude it. The first cut did `static_cast<EffectBase*>(current_)->dimensions()` each frame on the `MoonModule*` child, which **crashed** (SIGBUS in RTTI/vtable) when the eligible list included a test `EffectStub` that registers with `ModuleRole::Effect` but is a bare `MoonModule`. Two lessons: role is a registration property, not a type guarantee (a downcast keyed on role is UB); and the data was already available RTTI-free, the factory probes `dimensions()` at registration via `if constexpr` and stores it, so `ModuleFactory::typeDim(index)` gives it (ESP32 builds `-fno-rtti`, so `dynamic_cast` isn't even an option). + +## uint16 intermediate overflow blanks the display, and a status check doesn't prove the render works (MultiplyModifier) + +`MultiplyModifier` at 8×8×4 black-screened the no-PSRAM Olimex while desktop showed it working. `Layer::rebuildLUT` computed `maxDest = logicalCount * mod->maxMultiplier()` in `nrOfLightsType`; on no-PSRAM that's `uint16_t`, and `256 * 256 = 65536` **wraps to 0**, so the LUT was sized to ~nothing and almost every light mapped nowhere. On desktop (`uint32_t`) the product fits, so the bug was invisible. Fix: compute in `uint64_t`, clamp to the ceiling, narrow back. + +The harder lesson is verification: the agent's hardware test asserted the Layer **status string** ("16×16×1"), which was correct, while the *render* (driven by the corrupted LUT) was black; the product owner caught it by eye. A correctness test for a mapping/effect must assert the **buffer or LUT is non-empty with expected coverage** (the regression counts LUT destinations == physical light count), not just the declared dimensions. A width-sensitive bug is invisible on the uint32 desktop build; such paths need a uint16-typed unit test or hardware confirmation. + +## Lessons from the repo-transfer + v1.0.0 release branch + +Moving `ewowi/projectMM → MoonModules/projectMM` and cutting v1.0.0 surfaced three CI failures from the *infrastructure around* the release, not the diff. + +1. **`windows-latest` migrated from VS 2022 to `windows-2025-vs2026` mid-release**, breaking a green tree two ways: (a) `package_desktop.py` hard-coded `-G "Visual Studio 17 2022"` → CMake couldn't find VS 2022; (b) once the pin was dropped, the new VS 2026 MSVC STL emitted **C5285** ("specializing `std::tuple` is forbidden") on vendored `doctest.h`, fatal under `/WX`. Fixes: drop the generator pin (auto-detect survives migration); add `/wd5285` (third-party header, GCC/Clang never warn). +2. **A Pages-only `environment: github-pages` on the `release` job made a tag fail the gate before any step ran, dropping all release assets.** The job did asset upload AND Pages deploy under one environment whose protection rule allowed only `main`; the `v1.0.0` tag failed it, so the whole job (including "Publish GitHub release") was rejected in 2 seconds, a published release with zero binaries, and a red X blaming Pages. Fix: split into a `release` job (no environment, `contents: write`, runs on tags) and a `deploy-pages` job (`needs: [release]`, `if: ref==main`). Recovery without re-tagging: `gh workflow run release.yml -f tag=vX.Y.Z`. +3. **Three "failures" were the environment, not the change**, a 120µs tick contract "failed" at 536µs under concurrent-build load (118µs isolated); Improv reported `UNABLE_TO_CONNECT` while the device was provisioned and reachable (async-confirmation timeout); MoonDeck showed `0/0 online` while the device served HTTP 200 (an active network record with an empty subnet). None were defects. + +## Lessons from the LCD_CAM WS2812 driver bench debug (LcdLedDriver, S3) + +Bringing the 8-lane LCD_CAM driver from "compiles and ticks" to "strip animates" took three stacked root causes, each masked by the one before; every layer of indirection hid a failure the layer above couldn't see. + +1. **The i80 peripheral requires ALL `bus_width` data GPIOs, a partial bus never exists.** `esp_lcd_new_i80_bus` rejects `GPIO_NUM_NC` entries, so a 1-pin config never initialized: no bus, dark strip, while the UI showed an enabled driver. Fix: demand exactly 8 pins and report `LCD bus needs exactly 8 pins`; unused lanes take `0` and idle LOW. +2. **"Capacity still fits" is not "config unchanged."** `reinit()` skipped the bus rebuild when the DMA buffer was big enough, correct for grid resizes, wrong for pin edits (moving lane 0 from GPIO 13 to 18 keeps the frame size, so the bus kept clocking the OLD pins). Fix: record the pin set the live bus was built with, compare on every reinit, any difference forces the rebuild. Same family as the S3-DevKit stale-cache lesson: the cached check was true about the wrong invariant. +3. **Gate on `CONFIG_SOC_LCDCAM_I80_LCD_SUPPORTED`, NOT `CONFIG_SOC_LCD_I80_SUPPORTED`.** The classic ESP32 also defines `SOC_LCD_I80_SUPPORTED=1` for its unrelated I2S-LCD peripheral, so gating on it wired `LcdLedDriver` at boot on classic, and `esp_lcd_new_i80_bus()` hung the watchdog → **boot-loop on every classic ESP32**. The correct macro is defined only on chips with the real LCD_CAM (S3/P4). Bisect-proven, hardware-verified. +4. **A self-test that passes while the device fails is telling you what the test doesn't cover.** The first loopback transmitted a 3-byte synthetic pattern through a 136-byte transfer and PASSED while the strip washed out max-white. Upgrading the test to transmit the driver's REAL frame (5.4 KB, multi-descriptor GDMA chain, sustained cadence) and bit-verify the whole capture eliminated two suspects in one run, leaving timing. +5. **Modern WS2812B reads a 412 ns "0" as "1", T0H max is ~380 ns on newer revisions, and 3.3 V direct drive eats the margin.** The classic 2.4 MHz encoding (slot ≈ 416 ns) decoded perfectly on RMT RX yet rendered max white with flicker; the RMT driver's 350 ns zeros ran clean, isolating timing. Fix: pclk 2.67 MHz → 375 ns slots ("0"=375, "1"=750, bit 1125 ns), inside every WS2812B revision's window; latch pad resized to keep ≥300 µs. The lineage gets away with 416 ns behind a 74HCT level shifter. + +Bench notes: a board that drops STA mid-session falls back to softAP silently, poll reachability before reading an unanswered API call as a wedge. The differential test that cracked it twice: drive the same pin/strip with the proven RMT driver, it exonerates wiring, power, and strip in one move. + +## Diagnosing LED flicker: eliminate firmware with hardware tests before blaming (or fixing) the wire + +A classic ESP32 driving WS2812 on RMT showed random wrong colours on LEDs the effect left black. Instead of guessing (WiFi, buffering, timing), a four-step elimination each a *measurement*: (1) capture the source/preview buffer, clean, so corruption is downstream of the logical buffer; (2) whole-frame loopback self-test through a jumper, bit-exact PASS even under WiFi load, so the firmware/peripheral is innocent; (3) sweep `txPowerSetting` 20→1 dBm, flicker constant, so it's NOT WiFi coupling; (4) check pulse timing, 350/700/1250 ns spec-exact, not a timing-margin bug. Every firmware cause eliminated by test, the remaining cause is the physical data path, and "constant regardless of TX power" fingers signal integrity, a missing 3.3→5 V level shift. Fix is hardware (documented in [LED signal integrity](../usecases/led-signal-integrity.md)). + +**Red-herring note:** the flicker's *appearance* shifted (blue-only → random) after an unrelated `lightPreset` RGB→GRB default change plus a pin swap. The electrical fault was identical; only the colour mapping changed. A changed symptom is not proof a code change caused it. + +## ESP32-P4 support, round 1, per-board Ethernet pin config, and the P4's WiFi reality + +Adding the Waveshare ESP32-P4-NANO (round 1 of 4: board + Ethernet-only). + +- **The P4 has no native WiFi.** `SOC_WIFI_SUPPORTED` is absent on esp32p4 (it has EMAC, RMT, LCD_CAM, Parlio, no radio). WiFi comes from an on-board ESP32-C6 over SDIO via `esp_wifi_remote`/esp-hosted (a managed component, not in mainline v6.1-dev). Round 1 ships Ethernet-only. C6 SDIO pins on the P4-NANO (so round-2 Parlio avoids them): CLK 18, CMD 19, D0-D3 14-17, C6 reset 54. +- **Ethernet pins became a per-target compile-time config, not scattered `#ifdef`s.** `ethInit()` had Olimex RMII/PHY pins baked as literals; the P4-NANO needs IP101 PHY addr 1, MDC 31, MDIO 52, reset 51, and an *external* 50 MHz RMII clock fed IN on GPIO50 (`EMAC_CLK_EXT_IN`, opposite of Olimex's `EMAC_CLK_OUT`). An `EthPinConfig` struct + `constexpr ethPins = isEsp32P4 ? {…} : {…}` lives in `platform_config.h`; `ethInit` reads it. Keeps the platform-boundary rule (`if constexpr`, not `#ifdef` in domain code). +- **The IP101 PHY driver is a managed component in IDF v6** (`espressif/ip101`); the generic PHY (Olimex LAN8720) stays in core, so only the P4 build pulls it, behind `if constexpr (ethPins.isIp101)`. A build-script trap: the eth-fragment detector matched only `.eth`, so `sdkconfig.defaults.esp32p4-eth` would have set `MM_NO_ETH`, the matcher now accepts both `.eth` and `-eth`. + +## ESP32-P4 round 2, Parlio LED driver: a simpler peripheral, the same encoder + +Adding `ParlioLedDriver` (round 2 of 4) was mostly *subtraction* from the LCD driver. + +- **Parlio is simpler than the i80 bus:** no sacrificial WR/DC pins (Parlio clocks internally via `output_clk_freq_hz`, so no `clockPin`/`dcPin`), and no exactly-8-pins rule (Parlio takes `data_gpio_nums[]` with unused = `-1`, so 1-8 lanes work). +- **The encoder is shared, not duplicated.** `LcdSlots.h::encodeWs2812LcdSlots` outputs one bus byte per slot, and a Parlio bus byte is byte-identical to an i80 one, so `ParlioLedDriver` reuses the same encoder and `unit_LcdLedEncoder.cpp` test, zero new encode code. +- **One Parlio constraint:** `data_width` must be a power of two, ≤ `SOC_PARLIO_TX_UNIT_MAX_DATA_WIDTH`. Always create at `data_width = 8` (matching the encoder's 8-bit byte), unused lanes' GPIOs `-1`. +- **Clock math is identical:** Parlio's `PLL_F160M` ÷ 60 = 2.67 MHz = the same 375 ns slot the LCD driver settled on for the T0H-margin reason. +- **Default pins must dodge the P4's strapping GPIOs.** The first cut defaulted `36,37,38,…` but GPIO 34-38 are strapping pins (boot-mode control). It inited fine on the bench (nothing drove them during boot) but defaulting LED *output* onto strapping pins is a latent footgun. Fixed to `pins="20,21,22,23,24,25,26,27"` with all 64 lights on lane 0 (the other 7 idle LOW at zero cost, the parallel DMA transfer time is set by the *longest* lane, not the lane count). Clear P4-NANO GPIOs after Ethernet, C6 SDIO, I2C, and strapping: 20-27, 32-33, 39-48. Pull the strapping-pin list from the datasheet before picking defaults for a new chip. + +## Audio input (INMP441 mic), ship the manual core, design fresh from the datasheet + +The first audio-reactive capability: `AudioModule` (a SystemModule Peripheral) reads an INMP441 I²S mic and publishes an `AudioFrame` (level + 16-band spectrum + peak) that `AudioVolumeEffect`/`AudioSpectrumEffect` consume. + +- **Two seams, everything else host-tested.** Only the I²S read and the FFT *kernel* sit behind the platform boundary (`i2s_std` + esp-dsp's float `dsps_fft2r_fc32`). All signal math (DC strip, RMS, Hann window, magnitude→16-band log map) is header-only domain code. The desktop `audioFft` stub is a real naive O(n²) DFT, so the whole pipeline runs in CI on synth sines. esp-dsp float (not fixed-point) is right on an FPU chip. +- **Effects reach the producer via `AudioModule::latestFrame()` (a static accessor), not a boot setter**, because an audio effect can be added through the UI after boot and a setter only wired the boot instance. The active mic registers in `setup()`, clears in `teardown()`, returns a static silent frame when there's no mic. +- **Two hardware-only bugs, now pinned:** a missing `registerType<AudioModule>` made `create(...)->markWiredByCode()` deref null and boot-loop; the I²S read blocked the tick ~7.7 ms at a 20 ms timeout (fixed to non-blocking + a cross-tick sample accumulator, dropping to ~400 µs). The INMP441 is on the **left** I²S slot here (right reads empty), the first bench suspect when level floors with sound present. +- **Shipped the manual core; the adaptive conditioner was prototyped and removed.** Auto noise-floor + AGC + smoothing needs tuning in a *quiet* room; the dev environment (a campground van with strong varying low-frequency rumble) was the adversarial worst case that kept it from settling. Land the manual core, treat adaptive auto-tuning as its own increment, tune it where the noise floor is real-quiet. Also: `level` is overall RMS (independent of the FFT), don't derive it from the bands or it stops fluctuating with volume. +- **Designed fresh from the datasheet + textbook DSP, not a prior project.** The datasheet made even a behaviour-reference unnecessary: a flat ±3 dB mic has no per-frequency error to correct, so the hand-tuned band-correction table years of prior-project work produced was the wrong tool; the textbook defaults sufficed. The DSP choices and *why* live in [AudioModule.md](../moonmodules/core/moxygen/AudioModule.md). + +## ESP32-P4 round 3, WiFi via the C6: the abstraction the earlier round feared wasn't needed + +Round 1 predicted round 3 would need a WiFi abstraction seam; the actual implementation was far smaller. + +- **`esp_wifi_remote` is API-compatible, so there was no seam to build.** You still call `esp_wifi_init()`/`esp_wifi_connect()`/…; the component forwards them to the C6 over SDIO. The entire WiFi platform layer (~230 lines) compiles unchanged. The only new code is a two-call prelude (`esp_hosted_init()` + `esp_hosted_connect_to_slave()`) run *before* `esp_wifi_init()`, behind `if constexpr (platform::usesRemoteWifi)` (= `isEsp32P4 && hasWiFi`). Before designing a two-backend abstraction, check whether the vendor already made them API-identical. +- **Init ordering is the whole risk.** esp_hosted must be fully up before anything touches the WiFi stack; wrong ordering surfaces as NVS errors / asserts / a silent hang, not a clean error. The prelude-before-`esp_wifi_init` placement is the mitigation and the first thing to check if a P4-wifi boot misbehaves. +- **Pulled P4-only via a `rules` gate** (`if: "target == esp32p4"` in `idf_component.yml`), so other targets never fetch or compile the managed components. +- **A deliberate v6.0-floor exception:** these components live outside mainline v6.0. The floor rule's documented-exception path applies, accepted consciously, recorded at every introduction site, scoped to the P4. A floor rule with a documented-exception path beats a rigid one. + +## RMT timeout: don't cancel a stuck transfer, the cancel crashes classic ESP32 + +A deferred "fuller error handling" item for `rmtWs2812Show`/`rmtWs2812Wait` (🐇 CodeRabbit PR#17) was mostly already done and partly actively harmful. Most had landed in the multi-pin work: `rmtWs2812Transmit` already returns the `rmt_transmit` result, and `RmtLedDriver::loop()` already tracks `started[]` so it only waits on channels whose transmit succeeded. The remaining item, cancel the in-flight transfer on timeout via `rmt_disable()`, is a trap: `rmt_disable()` while a transmission is active triggers an **interrupt-WDT panic on classic ESP32** (espressif/esp-idf#17692; classic-only). That trades a self-healing dropped frame for a crash, strictly worse. The right change was to NOT add the cancel and document why the current code is already right (next tick re-encodes and re-transmits; a still-busy channel fails its `rmt_transmit` cleanly, `started[]` skips the wait). A deferred-improvement note is a hypothesis, not a spec, verify the improvement is real AND safe on every target before implementing. + +## Pin defaults: assign one only when it cannot do harm + +A mic-less **classic ESP32** boot-looped (TG1WDT_SYS_RESET at ~736 ms, silent hang). Bisect (clean-built known-good still looped; disabling AudioModule wiring booted clean) fingered AudioModule: auto-wired via `addChild` + `markWiredByCode()` (gated on `platform::hasI2sMic`), it ran `setup()` → `reinit()` → `audioMicInit()` → `i2s_channel_enable()`, which on the classic's older I²S driver **blocks forever** when no mic clocks the pins. The P4 was never affected (newer I²S returns silence or fails cleanly). + +The fix (a design fix, not a band-aid): don't auto-wire AudioModule (register it in the factory so it's user-addable); default the mic pins to unset (0); `reinit()` no-ops on any unset pin (`setStatus("set …"); return;`). Classic then boots 191 FPS, 0 WDT resets. This drove the general rule now in [architecture.md § Config provenance](../architecture.md#config-provenance-mcu-devicemodel) and [coding-standards.md § Defaults](../coding-standards.md#defaults): chip-/board-fixed pins (RMII Ethernet) *must* default (omitting them is a chicken-and-egg lockout); user-soldered pins (mic, LED strands) stay empty until set. Never auto-run a peripheral whose init can block on absent hardware. + +## Live reconfiguration falls out of the prepare-pass for free, MoonLight's "initless" goal, a different mechanism + +projectMM reconfigures every module live the instant a control changes (pins, leds-per-pin, output protocol, mic pin/rate), no reboot. The design note lives in [architecture.md § Live reconfiguration](../architecture.md#live-reconfiguration-every-change-applies-without-a-reboot). The lineage is MoonLight's "initless drivers" (a driver with no `addLeds`/`initLed` step, reading a mutable Context at `show()` time). projectMM reaches the same outcome by a different mechanism: our drivers *do* have an explicit rebuild (`RmtLedDriver::reinit()`, the i80/Parlio DMA-bus rebuild), but it's driven by the generic tier-3 `onBuildState()` sweep, not hand-built per driver, so any module returning `true` from `controlChangeTriggersBuildState` inherits live-reconfig, spanning drivers, audio, effects, layouts, modifiers, and network I/O alike. Credit the lineage for the idea, but name the property by what the user sees when the mechanism differs. + +## Lessons from the catalog-driven installer branch (3-layer device model) + +The installer was reworked so a board catalog sets a device up for its hardware at install time. The mechanics live in the installer README; these are the principles. + +- **Inject from data, don't bury in code (vs MoonLight's `ModuleIO.h`).** MoonLight hardcoded ~20 boards' pin presets in firmware C++ behind `#ifdef CONFIG_IDF_TARGET_*`; adding a board is a recompile and every binary ships every board's table. projectMM injects the same info from the catalog after flash, the firmware is a generic engine, the data specialises it. Adding a board is a JSON edit; board definitions become community-contributable data. +- **Investigate before building, the device side was already done.** The original plan was a new `POST /api/preset` endpoint; investigation found three install clients already fan a catalog's controls out as `POST /api/control` calls, and the real gap (a control write 404s on a fresh flash where the module doesn't exist) was solved by the already-existing idempotent `POST /api/modules` the clients weren't driving. A day of mapping the existing code replaced a new core endpoint (and a forbidden JSON-array parser). +- **`loopbackTxPin`, verify the claim against the code before deciding twice.** A `loopbackTxPin` override was proposed, dropped on "it only fits RMT's single pin, not Parlio/Lcd's lane array," then re-added after reading `ParallelLedDriver::runLoopbackSelfTest`: the Parlio/Lcd loopback only drives lane 0, so a single TX override substituting for lane 0 works on all three drivers. The "doesn't fit lane arrays" objection was an assumption, not a fact. +- **Board vs Device is a completeness spectrum, not two schemas.** A carrier/shield PCB an ESP32 module plugs into is a Board; a vendor-finished all-in-one (QuinLED Dig-2-Go) is a Device, but they're the *same* catalog entry shape, a Device is just a Board entry with more optional `modules`/`controls` filled in. So no separate-schema `devices.json`. A board's pins fall into three categories: *always-fixed* (LED outputs, status LED, default freely), *board-optional* (a populated-or-not W5500/IR, can't default blind because the same board name ships both ways), and *user-soldered* (always unset). +- **Per-board capability spec'n'test loop.** Each board's pin-layout `image` and product `url` are *inputs* to a repeatable loop: read capabilities off the image + link; for a capability we offer (I²S mic → `AudioModule`) wire it into `modules`/`controls` with real pins; for one we don't (IR receive) write a proposal, spec it, create test scripts, iterate, *then* add it. An un-implemented capability is a recorded proposal, not a half-wired control. *Specs before code* at board granularity. +- **Drivers became catalog-added, with an OTA nuance.** LED/network drivers stopped being boot-wired (only `Preview` stays, since it needs the HTTP broadcaster the catalog can't supply); each board declares its driver(s) in `modules`. A fresh-erased board boots with `Drivers = [Preview]` only. The nuance: an OTA update *without* erase keeps a device's previously-persisted drivers (saved config the new firmware reloads), so "out-of-box" and "after-update" differ. +- **A persistence overlay must distinguish "key absent" from "value 0".** The runtime-Ethernet-PHY work moved pin/PHY config into persisted controls with non-zero per-chip defaults (P4 IP101 `ethType` 2). `applyControlValue` used `json::parseInt`, which returns 0 for an absent key, indistinguishable from a real 0, and wrote that 0 under the Clamp policy, so loading an older `<Module>.json` clobbered the default. On the P4 this zeroed `ethType` (2→0=none): link LEDs on, no DHCP. Invisible on classic (eth defaults mostly 0) and on `main` (still read `constexpr ethPins`). Fix: a `json::hasKey()` guard in `applyControlValue`. Any "control resets to 0 after reboot" symptom is a persistence-overlay smell. The decisive move was a `std::printf` over the P4's secondary USB-Serial-JTAG console after a `git worktree` bisect. +- **A GPIO pin is its own control type (`ControlType::Pin`), not an overloaded int16.** Pins added as `addInt16` `-1..48` rendered as a *slider* (meaningless for a GPIO) and the cap excluded the P4's high pins (MDIO 52, clk 50); dropping the range didn't help because the UI's `int16` case always draws a slider. Fix: a dedicated `Pin` type (`int8_t` storage, a GPIO never exceeds ~54; −1 = unused; the UI renders a plain number input keyed off the `"pin"` type string; min/max are a server-side write-clamp only). Every future pin control migrates to `addPin` for free. +- **`deviceName` (identity) vs `deviceModel` (product) vs board (bare PCB), one term was doing three jobs.** "Board" had meant per-unit network identity, the catalog key, AND the bare PCB. Untangled: `deviceName` is the per-unit identity (drives mDNS/AP/DHCP, RFC-1123-coerced); `deviceModel` is the hardware product (catalog key, spaces allowed, never a hostname); "device" is the umbrella; "board" means only the bare PCB. Drove the BoardModule→SystemModule fold, the `board`→`deviceModel` rename (SET_BOARD→SET_DEVICE_MODEL, byte 0xFE unchanged), and the eth pin-map clarification (driver = firmware, pin map = firmware-seeded but deviceModel-authoritative). +- **"Improv = REST over serial", one apply-core, two transports.** The HTTPS installer couldn't POST to an `http://` device (mixed-content), and the `?deviceModel=` pull only ran if the user opened that exact link. Fix: the installer owns the USB serial port during provisioning, so push config over it as the same REST ops, a new `APPLY_OP` (0xFC) Improv vendor RPC whose payload is the same JSON a `POST /api/modules`/`/api/control` body carries. On device it routes to one transport-free apply-core (`HttpServerModule::applyAddModule/applySetControl/…`) the HTTP handlers also call, so serial and network execute identical code. This deleted the whole browser handoff. The hard part (chunk reassembly + out-of-order/duplicate sequence guard) was extracted to `src/core/ImprovOpReassembler.h` (header-only state machine) so it unit-tests on desktop, and the frame builders (device C++, Python, JS) are pinned by one shared golden vector in `test/python` + `test/js`. A hard mechanism buried in a platform `.cpp` that "can only be tested on hardware" is a smell, extract its pure core. +- **A periodic re-broadcast to let late joiners catch up is a hack wearing a keepalive costume.** The 3D preview re-sent the whole coordinate table every ~1 s "so a client that connected after the last rebuild catches it", rebuilding the full table from the layout every tick-second forever, on the hot path, whether or not anyone connected or anything changed. The correct construct is event-driven: send the table when it changes (`onBuildState`) and when a client asks (a new WS connection bumps `BinaryBroadcaster::clientGeneration()`, which `PreviewDriver::loop()` watches), strictly *less* code and zero idle cost. It sneaked past review because it worked in casual testing and its cost was invisible until a later change made each rebuild heavier. "Re-send periodically so it eventually syncs" is the polling-instead-of-events smell. Guard: a test that advances the clock several seconds with no client change and asserts the table is NOT re-sent. +- **When a working seam regresses after your "fix," suspect the fix, and measure with a tool faithful to what the user sees.** A later attempt to route the coordinate table + downsampled frame through the resumable send path (removing the synchronous spin-and-close) regressed every board into an intermittent stall through several variants. Three compounding lessons: (1) stop at the first failed fix on a working path (revert to known-good at attempt two, per *Anti-stalling*); (2) a plain one-shot WebSocket probe *gave up on close* where a browser *reconnects*, so it reported stalls users never saw and missed blips they did, the fix was a browser-faithful probe (`moondeck/diag/preview_health.py`: reads binary frames, 25 s keepalive ping, auto-reconnect, exactly `app.js`'s `connectWs`); (3) a **38-hour-old desktop binary** still held port 8080 so the freshly-built one couldn't bind, before bisecting a "rebuild didn't fix it" bug, confirm the artifact under test is the one you built (check process uptime / `build` timestamp / what's bound to the port). +- **Don't hold a vendor library's async handle across your own event loop, it races the library's internal timers.** The mDNS *browse* (peer discovery, distinct from advertise) used `mdns_query_async_new`, whose handle `DevicesModule` held across ticks and polled with a 0 ms timeout. The mDNS component's own task frees that handle's queue when the 3 s window expires, so a poll landing after expiry asserts on a freed queue (`xQueueSemaphoreTake queue.c:1709`), intermittent and grid-size-sensitive (a bigger grid widens the gap). A first fix (a cancel-before-mutate guard, assuming a service-table mutation tore it down) did nothing, because the freeing party is the expiry timer, not our code. Real fix: replace start/poll/stop with one synchronous `mdnsBrowse()` holding no handle across ticks. Synchronous `mdns_query_ptr` blocks the full window (~80 ms, charged to the tick), so throttle: browse one service type every ~8th tick with a ~60 ms timeout. A library's async/iterator handle is only valid between *its* lifecycle events; an intermittent load-dependent crash in a vendor backtrace is a lifecycle race, not a component bug (find the *actual* concurrent actor before fixing). Hardware-only; the reproduction (concurrent WS churn at a large grid) is the proof. +- **A dead control that was always meant to be functional belongs in the mechanism that already expresses it.** Six persisted-but-ignored Layer controls (percent region carving) were due to be wired into `rebuildLUT`; the product owner's question, *can a modifier already do this?*, was the better path. `ModifierBase`'s two virtuals (`logicalDimensions`, `mapToPhysical`) express carving exactly, so it shipped as a modifier and the Layer controls were deleted. The implementation forced the boundary rounding: inclusive-ceil made two abutting layers (0..50, 50..100) overlap by one pixel at the seam, so the product owner chose half-open `[start, end)` (with a min-1-pixel floor). "Make the default fastest" is best met by making the default the *absence* of the feature (full coverage = no modifier = the existing fast path). Reach for half-open intervals whenever regions tile a space. + +## ESP32-S31 RGMII Ethernet bring-up: four bugs between "code compiles" and "link up" + +Bringing up the S31's on-chip 1 Gb RGMII Ethernet took four fixes, none of them the C++, each a layer *below* the feature code that a build can't catch. For a hardware bring-up, "it compiles" tells you almost nothing; the truth is only in the boot log on the actual board. + +- **A `MM_NO_ETH`-style gate keyed on a *filename* silently omits a new board.** `build_esp32.py` decided "does this firmware have Ethernet?" by pattern-matching the sdkconfig fragment *filename* for `.eth`; the S31 enables its EMAC in `sdkconfig.defaults.esp32s31` (no `.eth`), so the gate compiled the `ethInit()` stub and the board fell back to WiFi with zero eth log. A gate asking "does X enable feature Y?" must read what X *contains* (`CONFIG_ETH_USE_*=y`), never what it's named. +- **A schematic/reference pin table is a hypothesis until the boot log confirms it, ours was systematically off by one.** MDC/MDIO/TXD were all one GPIO low. Wrong data pins failed loudly (`invalid TX_CTL GPIO number`), but wrong MDIO failed as "No PHY device detected", three inference-steps from the typo. The chip's own IDF IO_MUX table (`esp32s31/emac_periph.c`) is the authority for RGMII data pins (fixed pads); verify SMI pins against `ETH_ESP32_EMAC_DEFAULT_CONFIG`; use PHY-addr auto-detect (`-1`). +- **A shared clock is a contended resource, the RGMII 125 MHz Tx clock can't come from a PLL already spoken for.** The default (AUTO) sourced it from the MPLL, but PSRAM already ran the MPLL at 400 MHz (no integer path to 125) and CPLL couldn't synthesise 125 on the 40 MHz XTAL grid; only the fractional APLL works. On an SoC where one PLL feeds multiple peripherals, "pick a clock source" is a conflict-resolution decision, check what already owns each PLL. +- **Changing a Kconfig *choice* in a defaults fragment needs a clean build.** Editing `CONFIG_ETH_EMAC_RGMII_TX_CLK_SRC_*` and building incrementally kept the *old* choice (the build dir's `sdkconfig` already had a value; defaults don't override an existing one). Two flash cycles were spent "testing APLL" that were still CPLL. `rm -rf` the build dir when a sdkconfig *choice* changes. + +## W5500 SPI Ethernet (LightCrafter / SE16): interrupt service, IDF-v6 config, and a per-board reset pin + +Bringing up the two Limpkin ESP32-S3 boards' external W5500 (SPI) Ethernet surfaced three traps between "the driver installs" and "the link is up." + +- **Interrupt mode needs `gpio_install_isr_service()`, miss it and the IRQ silently degrades to slow polling.** The W5500 has a real INT pin (SE16 GPIO18, LightCrafter GPIO45), but the IDF driver's `gpio_isr_handler_add()` requires the per-pin ISR service already installed, which nothing did. The `add` failed with a one-line log, the interrupt never fired, and RX serviced only on the slow fallback → a ~1 s sawtooth ping latency. Install the ISR service once before driver init (`ESP_ERR_INVALID_STATE` = already installed = fine). The `int_gpio=-1` polling fallback is a workaround that hides this. +- **IDF v6's W5500 driver rejects `int_gpio_num < 0` unless a poll period is also set.** `int_gpio_num = -1` alone returns `invalid configuration argument combination`; you must also set `w5500_config.poll_period_ms` (e.g. 10). The pair is the API contract. +- **A failed Ethernet init that cascades to WiFi masks the eth bug, "why are we on WiFi?" is the tell.** Both bugs above manifested as the board quietly running on WiFi with a healthy HTTP server and no eth log. The robustness cascade is correct behaviour but turns a loud init failure into a silent capability loss; a board that *should* be on Ethernet but is on WiFi is the signal to read the eth init log. +- **A `0x00` "version mismatched" read from an SPI PHY means the chip is held in reset.** The LightCrafter's WIZ850IO holds its own `nRST` until GPIO3 is driven HIGH; without releasing it the ESP32 read `0x00` from the W5500's version register (`expected 0x04, got 0x00`). The SE16's W5500 self-resets, which is why this was board-specific. Setting `ethRstGpio` (→ the driver's `reset_gpio_num`) fixed it. An all-zeros register read from an SPI peripheral is "the chip isn't talking"; the first suspect after wiring is a reset/enable line the host must drive. + +## An RMT/IR done-callback runs in ISR context: signal only, decode on the task + +The first NEC-decoder version crash-looped the board (USB dropped, port re-enumerated) because the RMT `on_recv_done` callback both **decoded** the frame (calling non-`IRAM_ATTR` helpers) and **re-armed** `rmt_receive()` from interrupt context. On ESP32, an ISR that jumps to flash-resident code while the cache is disabled, or re-enters a driver call, faults. The existing working RX capture (`rmtWs2812RxCapture`) already showed the safe shape: the ISR only records the symbol count and signals a queue; the decode + re-arm happen on the task (`irRead`, from the render loop). An RMT/GPIO done-callback may touch only its stack, the passed event data, and an ISR-safe FreeRTOS primitive (`xQueueSendFromISR`); parsing, logging, and re-arming move to the waking task. When in doubt, copy the codebase's already-proven callback. + +## Don't promote one specific board's pins to "the chip default" + +`NetworkModule::ethType` defaulted to a per-chip `ethConfigDefault`, and the classic / P4 / S31 entries were, on inspection, **one specific board's wiring each** (classic = Olimex, P4 = Waveshare NANO, S31 = Function-CoreBoard) dressed up as a chip property. Two failures: a WiFi-only board (LOLIN-D32 Shelly) wasted an Ethernet-PHY init on hardware it lacks; and the QuinLED Dig-Octa (custom MDC/MDIO, no software eth reset, GPIO5 is an LED output there) would have inherited reset=GPIO5 and driven an **LED pin as an Ethernet reset**. Fix: `ethType` defaults to **None**, Ethernet is opt-in, every board declares its own `ethType` + wiring pins in `deviceModels.json`, and `check_devices.py` enforces "ethType set → wiring pins present." A "default" that is really one board's values is bespoke masquerading as standard; make the capability opt-in and require each consumer to state its own values, so a missing declaration fails loudly instead of inheriting a stranger's wiring. + +## A new firmware default only reaches a factory-fresh device; persistence wins on an upgrade + +After changing the `ethType` firmware default to None and reflashing an already-provisioned board, the device *still* reported the old value, persistence (`FilesystemModule` overlays every saved control at boot) restores the value captured during earlier provisioning, and a plain `flash write` keeps the filesystem. The same shape bit the MQTT prefix (a stale stored prefix a reflash didn't clear). This is correct persistence behaviour (saved config wins over a constructor default), but a default change is invisible to any device that has already run, only an erase-flash or explicit re-set surfaces it. When you change a default to fix a bug, the fix lands only on factory-fresh devices; an already-provisioned fleet needs an explicit migration, and when debugging "my fix didn't take," check the persisted `.config/*.json` before the firmware default. + +## Fixing "blocking in the tick" at the connect misses the same violation at the write + +The MQTT client's first version blocked the render loop twice, a synchronous `connect()` (with DNS) *and* an unbounded blocking `write()` retry, both inside `loop1s` (which runs in `Scheduler::tick`). The first review caught the connect (reworked to non-blocking `connectStart`/`connectPoll`); the re-review caught that `conn_.write()` was still there: a zero-window broker (accepts TCP, stops reading) fills the send buffer and the next `write` never returns, freezing the render loop permanently, and the keepalive timeout can't even fire because control never comes back. Fix: sends go through a non-blocking `writeSome` with an all-or-reset rule (control packets ≤256 B, so atomic-send is realistic). A "no blocking in the hot path" audit must sweep every syscall the path can reach, connect, DNS, read, *and* write, not just the loudest one; the re-review exists to catch exactly the partial fix that reads as complete. diff --git a/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md b/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md index ecfee2c2..10a2c1f9 100644 --- a/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md +++ b/docs/history/plans/Plan-20260706 - Home Assistant MQTT Discovery.md @@ -23,7 +23,7 @@ Feature branch `ha-mqtt-discovery` (already created; carries the earlier backlog ## Decisions locked (PO) - **JSON schema** discovery, not default schema. -- **`unique_id` = `<last6-MAC>`** (the stable id per [decisions.md § MQTT identity](../decisions.md)), `name` = `SystemModule::deviceName()`. Never the editable name as identity. +- **`unique_id` = `<last6-MAC>`** (the stable id per [ADR-0010](../../adr/0010-integration-identity-stable-hardware-id.md)), `name` = `SystemModule::deviceName()`. Never the editable name as identity. - Gated on a new **`haDiscovery`** bool control (default on where MQTT ships); toggling re-announces / retracts. - Existing mqttthing topics unchanged; Discovery is additive. diff --git a/docs/history/plans/README.md b/docs/history/plans/README.md new file mode 100644 index 00000000..2c5ad14a --- /dev/null +++ b/docs/history/plans/README.md @@ -0,0 +1,18 @@ +# Plans + +Approved feature plans, one per feature, the **design-intent** record: what we set out to build and why, written in `/plan` mode before implementation. This is one of the four homes for accumulated knowledge, and each holds a different genre: + +- **plans/** (here) the forward-looking intent, what a feature meant to do. +- **[../lessons.md](../lessons.md)** the debugging lessons, a bug, its cause, the fix. +- **[../../adr/](../../adr/README.md)** the architectural decisions, why approach A over B. +- **[CLAUDE.md](../../../CLAUDE.md) / [coding-standards.md](../../coding-standards.md)** the durable rules a lesson hardened into. + +## Naming + +`Plan-YYYYMMDD - <title>.md`, ISO-8601 date first so the directory sorts chronologically. The outcome rides in the filename as a trailing parenthetical: `… (shipped).md` once the design lands, `… (attempted, abandoned).md` if it was tried and dropped. An unmarked plan is still in flight. + +## Rules + +**Kept, not pruned.** Plans are the permanent design-intent record, they are not deleted when a lesson is absorbed (that is the lessons/ADR rule, not this one). The one exception: a multi-phase effort's per-phase plans may be consolidated into a single `… (shipped).md` once the whole effort lands, provided the consolidated record preserves each phase's intent and outcome. + +**Agents write, don't auto-read.** An agent writes a plan here when creating one, but does not read the existing plan files for context unless the product owner points to one (the same rule as the rest of `history/`). This is a product-owner reference archive. diff --git a/docs/moonmodules/core/services.md b/docs/moonmodules/core/services.md index 521d5c7d..77f8d388 100644 --- a/docs/moonmodules/core/services.md +++ b/docs/moonmodules/core/services.md @@ -8,7 +8,7 @@ The user-facing core services — the machinery that runs a show, each configura The device's identity and vitals — name (behind mDNS `<name>.local`, the SoftAP SSID, the DHCP hostname), uptime, heap, and per-module footprint reporting. Hosts the Audio and I2C-scan peripherals. -<img src="../../../assets/core/SystemModule.png" width="300" alt="System module controls"> +<img src="../../assets/core/SystemModule.png" width="300" alt="System module controls"> - `deviceName` — the device identity behind mDNS `<name>.local`, the SoftAP SSID, and the DHCP hostname. - `deviceModel` — the board model (drives the installer catalog entry). @@ -24,7 +24,7 @@ Detail: [technical](moxygen/SystemModule.md) WiFi / Ethernet connectivity, static-IP configuration, RSSI and TX-power reporting. Brings the device onto the LAN before the HTTP and WebSocket servers start. -<img src="../../../assets/core/NetworkModule.png" width="300" alt="Network module controls"> +<img src="../../assets/core/NetworkModule.png" width="300" alt="Network module controls"> - `mode` — WiFi / Ethernet / off. - `ssid` / `password` — WiFi credentials. @@ -43,7 +43,7 @@ Detail: [technical](moxygen/NetworkModule.md) Serial/BLE Improv Wi-Fi provisioning — the web installer hands credentials to a fresh device over this protocol during the flash-and-connect flow. -<img src="../../../assets/core/ImprovProvisioningModule.png" width="300" alt="Improv provisioning module controls"> +<img src="../../assets/core/ImprovProvisioningModule.png" width="300" alt="Improv provisioning module controls"> - `provision_status` — read-only provisioning state. @@ -55,7 +55,7 @@ Detail: [technical](moxygen/ImprovProvisioningModule.md) Discovers and lists other projectMM devices on the LAN (the `devices` List control), each row expanding to a detail panel; persists the last-known list across reboot. -<img src="../../../assets/core/DevicesModule.png" width="300" alt="Devices module — discovered LAN devices"> +<img src="../../assets/core/DevicesModule.png" width="300" alt="Devices module — discovered LAN devices"> - `devices` — a List control of discovered devices; each row expands to a detail panel. Persistable. @@ -69,7 +69,7 @@ Detail: [technical](moxygen/DevicesModule.md) Bridges the light to an MQTT broker so a home-automation hub (Homebridge) can control it — a transport over the shared `Scheduler::setControl` apply-core, not new control logic. Our own dependency-free MQTT 3.1.1 client; disabled until a broker is set. Topics, colour-wheel mapping, and the Homebridge config: ⌄ details. -<img src="../../../assets/core/MqttModule.png" width="300" alt="MQTT module controls"> +<img src="../../assets/core/MqttModule.png" width="300" alt="MQTT module controls"> - `broker` — the broker hostname (e.g. `homeassistant.lan`) or IP. A hostname is resolved via DNS. - `port` — broker port (default 1883). @@ -87,7 +87,7 @@ Detail: [technical](moxygen/MqttModule.md) Over-the-air firmware flashing — the one operation that swaps the binary and needs a power cycle (every *config* change applies live; a firmware OTA does not). -<img src="../../../assets/core/FirmwareUpdateModule.png" width="300" alt="Firmware update module controls"> +<img src="../../assets/core/FirmwareUpdateModule.png" width="300" alt="Firmware update module controls"> - `firmware` — the OTA image to flash. - read-only — `version`, `build`, `firmwarePartition`, `update_pct` (progress). @@ -102,7 +102,7 @@ Detail: [technical](moxygen/FirmwareUpdateModule.md) A boot-wired system tool (distinct from Filesystem, the persistence *engine*): browse and manage the device filesystem from a dedicated panel — a lazy expand/collapse folder tree (VS Code / Explorer shape) plus an inline text editor. Browsing is UI-side over `/api/dir` + `/api/file`, so the module itself stays minimal. Tree/toolbar/editor behaviour: ⌄ details. -<img src="../../../assets/core/FileManagerModule.png" width="300" alt="File Manager panel — folder tree + toolbar"> +<img src="../../assets/core/FileManagerModule.png" width="300" alt="File Manager panel — folder tree + toolbar"> - `file browser` — the panel itself: an expand/collapse folder tree with a toolbar (＋folder / ＋file / delete / refresh / upload) and an inline text editor. The module's main surface (⌄ details for the interactions). - `show hidden` — reveal dot-prefixed files/folders (e.g. `.config`); forwarded to `/api/dir` as its `hidden` filter. @@ -117,7 +117,7 @@ Detail: [technical](moxygen/FileManagerModule.md) A System peripheral (added by the user, not auto-wired): an I²S microphone (or line-in ADC) feeding the FFT that audio-reactive effects consume via `AudioModule::latestFrame()`. It also syncs audio over UDP, WLED-compatible: broadcast the local analysis for the WLED ecosystem, or receive a peer's audio to drive effects with no local mic. Idle until real GPIOs are entered. -<img src="../../../assets/core/AudioModule.png" width="300" alt="Audio module controls"> +<img src="../../assets/core/AudioModule.png" width="300" alt="Audio module controls"> - `wsPin` / `sdPin` / `sckPin` — the I²S GPIOs (unset until entered). - `mclkPin` — master-clock GPIO for a line-in ADC that needs one (e.g. the PCM1808); leave unset for a plain mic. @@ -137,7 +137,7 @@ Detail: [technical](moxygen/AudioModule.md) A System peripheral that probes the I²C bus (default GPIO21/22) on a button press and reports the addresses found. -<img src="../../../assets/core/I2cScanModule.png" width="300" alt="I2C scan module controls"> +<img src="../../assets/core/I2cScanModule.png" width="300" alt="I2C scan module controls"> - `sda` / `scl` — the bus GPIOs (default GPIO21/22). - `scan` — a button; press to probe the bus now. @@ -151,7 +151,7 @@ Detail: [technical](moxygen/I2cScanModule.md) A System peripheral (added per board): an IR remote receiver that drives other modules' controls through the shared `Scheduler::setControl` primitive. It **learns** any remote (NEC-over-RMT): pick an action in `learn`, press a button to bind its code. What each action does + the status-line messages: ⌄ details. -<img src="../../../assets/core/IrModule.png" width="300" alt="IR module controls"> +<img src="../../assets/core/IrModule.png" width="300" alt="IR module controls"> - `pin` — the IR receiver GPIO (unset until entered; on the SE16 it shares GPIO 5 with the Ethernet MISO via the board switch, on the LightCrafter it is its own GPIO 4 alongside Ethernet). - `learn` — pick an action to bind (`on/off` / brightness up / brightness down / palette next / palette prev); the next received code binds to it, then learning disarms. The first option, `off`, is the disarmed state (bind nothing), not a light action. diff --git a/docs/moonmodules/light/drivers.md b/docs/moonmodules/light/drivers.md index 1d776c41..4115449c 100644 --- a/docs/moonmodules/light/drivers.md +++ b/docs/moonmodules/light/drivers.md @@ -14,7 +14,7 @@ A driver reads its window of the [Drivers](moxygen/Drivers.md) container's share Addressable WS2812B-class LEDs over a wire, one GPIO per strand. Three peripherals do this — pick by chip: **RMT** (single/few strands, any ESP32), **LCD_CAM** (8 parallel strands, S3), **Parlio** (1–8 parallel strands, P4). Same controls, same wire contract; they differ only in how many strands clock out at once and on which chip. -<img src="../../../assets/light/drivers/RmtLedDriver.png" width="300" alt="LED output driver controls"> +<img src="../../assets/light/drivers/RmtLedDriver.png" width="300" alt="LED output driver controls"> - `pins` — data GPIO list, e.g. `18,17,16` (one strand each). Empty idles until set; changing it re-inits live. - `ledsPerPin` — lights per pin, matched by position; empty or short = even split of the remainder. @@ -33,7 +33,7 @@ Detail: [RMT](moxygen/RmtLedDriver.md) · [LCD](moxygen/LcdLedDriver.md) · [Par ### Network Send 💫 · UDP -<img src="../../../assets/light/drivers/NetworkSendDriver.png" width="300" alt="NetworkSend controls"> +<img src="../../assets/light/drivers/NetworkSendDriver.png" width="300" alt="NetworkSend controls"> Streams the buffer over UDP as **Art-Net**, **E1.31 / sACN**, or **DDP** — one burst per frame, compatible with Falcon/Advatek controllers, xLights, and LedFx. @@ -54,7 +54,7 @@ Detail: [technical](moxygen/NetworkSendDriver.md) ### Hue 💫 · bridge -<img src="../../../assets/light/drivers/HueDriver.png" width="300" alt="A HueDriver in the UI"> +<img src="../../assets/light/drivers/HueDriver.png" width="300" alt="A HueDriver in the UI"> Drives **Philips Hue bulbs as pixels**: each colour bulb in the driver's window becomes one pixel, pushed to the bridge over its HTTP API. Paced to the bridge's ~10 cmd/s limit — smooth ambient colour, not strobing. @@ -75,7 +75,7 @@ Detail: [technical](moxygen/HueDriver.md) ### Preview 💫 · web UI -<img src="../../../assets/light/drivers/PreviewDriver.png" width="300" alt="PreviewDriver controls"> +<img src="../../assets/light/drivers/PreviewDriver.png" width="300" alt="PreviewDriver controls"> Streams a true-shape 3D preview to the web UI over WebSocket as a **point list** — only the real lights at their real positions, so a sphere/ring/arbitrary map shows in its true shape. The one boot-wired driver. diff --git a/docs/moonmodules/light/effects.md b/docs/moonmodules/light/effects.md index c7688228..28a53ef8 100644 --- a/docs/moonmodules/light/effects.md +++ b/docs/moonmodules/light/effects.md @@ -6,7 +6,7 @@ Every effect, one block each: its preview, what it does, and what each control m **Migrating an effect — behaviour is the spec.** A ported effect must reproduce the original's **exact** visual behaviour: end users have relied on these for years, so a port that looks different is a regression, not an improvement. Don't get creative with defaults, oscillator math, colour mapping, or geometry, and don't silently drop a parameter that *is* the mechanism (the PaintBrush straight-vs-curved-lines bug was a dropped partial-line `length`; Game of Life was wrong the first time by not porting the real algorithm). Study the source for the algorithm, defaults, and visual result; pin it with unit + scenario tests; then write our **own** implementation against `EffectBase`/our primitives — carry the behaviour forward, don't trace or copy the structure (see [*Industry standards, our own code*](../../../CLAUDE.md#principles)). Credit the origin as prior art in the block below. -> Some WLED-origin effects show a preview gif from [WLED-Utils](https://github.com/scottrbailey/WLED-Utils) by scottrbailey (the canonical WLED effect gif set, cross-linked with credit); these show WLED's rendering. Effects with a local `../../../assets/…` gif show our own output. +> Some WLED-origin effects show a preview gif from [WLED-Utils](https://github.com/scottrbailey/WLED-Utils) by scottrbailey (the canonical WLED effect gif set, cross-linked with credit); these show WLED's rendering. Effects with a local `../../assets/…` gif show our own output. ## MoonLight effects @@ -66,7 +66,7 @@ Detail: [technical](moxygen/FreqSawsEffect.md) ### LavaLamp 💫🦅 · 2D -<img src="../../../assets/light/effects/LavaLampEffect.gif" width="300" alt="LavaLamp effect preview"> +<img src="../../assets/light/effects/LavaLampEffect.gif" width="300" alt="LavaLamp effect preview"> Three slow blobs through a black→red→orange→yellow→white ramp — atmospheric lava look. @@ -84,7 +84,7 @@ Detail: [technical](moxygen/LavaLampEffect.md) ### Lines 💫 · — -<img src="../../../assets/light/effects/LinesEffect.gif" width="300" alt="Lines effect preview"> +<img src="../../assets/light/effects/LinesEffect.gif" width="300" alt="Lines effect preview"> Sweeps axis-aligned planes in sync; red/green/blue name the X/Y/Z axis — a preview-orientation test pattern. @@ -99,7 +99,7 @@ Detail: [technical](moxygen/LinesEffect.md) ### Metaballs 💫🦅 · 2D -<img src="../../../assets/light/effects/MetaballsEffect.gif" width="300" alt="Metaballs effect preview"> +<img src="../../assets/light/effects/MetaballsEffect.gif" width="300" alt="Metaballs effect preview"> `count` blobs orbit via integer sin/cos; metaball field per pixel — bright HSV merge/split. @@ -118,7 +118,7 @@ Detail: [technical](moxygen/MetaballsEffect.md) ### Particles 💫🦅 · 2D -<img src="../../../assets/light/effects/ParticlesEffect.gif" width="300" alt="Particles effect preview"> +<img src="../../assets/light/effects/ParticlesEffect.gif" width="300" alt="Particles effect preview"> A swarm of drifting particles with persistent fading trails. @@ -137,7 +137,7 @@ Detail: [technical](moxygen/ParticlesEffect.md) ### Plasma 💫🦅 · 2D/3D -<img src="../../../assets/light/effects/PlasmaEffect.gif" width="300" alt="Plasma effect preview"> +<img src="../../assets/light/effects/PlasmaEffect.gif" width="300" alt="Plasma effect preview"> Summed sine waves on orthogonal + diagonal axes; large rolling blobs (3D on volumetric layouts). @@ -170,7 +170,7 @@ Detail: [technical](moxygen/PraxisEffect.md) ### Rainbow 💫 · 2D -<img src="../../../assets/light/effects/RainbowEffect.gif" width="300" alt="Rainbow effect preview"> +<img src="../../assets/light/effects/RainbowEffect.gif" width="300" alt="Rainbow effect preview"> Diagonal animated rainbow — always-visible default/test effect. @@ -200,7 +200,7 @@ Detail: [technical](moxygen/RandomEffect.md) ### Rings 💫🦅 · 2D -<img src="../../../assets/light/effects/RingsEffect.gif" width="300" alt="Rings effect preview"> +<img src="../../assets/light/effects/RingsEffect.gif" width="300" alt="Rings effect preview"> Expanding concentric rings from random centres, additive overlap (calm defaults). @@ -219,7 +219,7 @@ Detail: [technical](moxygen/RingsEffect.md) ### Ripples 💫🟦🦅 · 3D -<img src="../../../assets/light/effects/RipplesEffect.gif" width="300" alt="Ripples effect preview"> +<img src="../../assets/light/effects/RipplesEffect.gif" width="300" alt="Ripples effect preview"> Distance-from-centre sets a per-column wave phase; the lit surface ripples like water. @@ -285,7 +285,7 @@ Detail: [technical](moxygen/SphereMoveEffect.md) ### Spiral 💫🦅 · 2D -<img src="../../../assets/light/effects/SpiralEffect.gif" width="300" alt="Spiral effect preview"> +<img src="../../assets/light/effects/SpiralEffect.gif" width="300" alt="Spiral effect preview"> Rotating spiral from angle + distance (`atan2_8`/`dist8`). @@ -320,7 +320,7 @@ Detail: [technical](moxygen/StarFieldEffect.md) ### StarSky 💫 · 3D -<img src="../../../assets/light/effects/StarSkyEffect.gif" width="300" alt="StarSky effect preview"> +<img src="../../assets/light/effects/StarSkyEffect.gif" width="300" alt="StarSky effect preview"> Twinkling stars at random light positions, each fading in and out independently over a dark background. @@ -581,7 +581,7 @@ Detail: [technical](moxygen/WaveEffect.md) ### Fire ⚡️🦅 · 2D -<img src="../../../assets/light/effects/FireEffect.gif" width="300" alt="Fire effect preview"> +<img src="../../assets/light/effects/FireEffect.gif" width="300" alt="Fire effect preview"> Fire2012-style heat field — sparks at the base rise and cool through the active palette (heat = palette index, cold at the low end, hottest at the high end); spark count scales with width. @@ -600,7 +600,7 @@ Detail: [technical](moxygen/FireEffect.md) ### Noise ⚡️ · 2D/3D -<img src="../../../assets/light/effects/NoiseEffect.gif" width="300" alt="Noise effect preview"> +<img src="../../assets/light/effects/NoiseEffect.gif" width="300" alt="Noise effect preview"> Smooth animated value noise; true 3D field on volumetric layouts. diff --git a/docs/moonmodules/light/layouts.md b/docs/moonmodules/light/layouts.md index 0ef16619..181a8b9f 100644 --- a/docs/moonmodules/light/layouts.md +++ b/docs/moonmodules/light/layouts.md @@ -10,7 +10,7 @@ Every layout, one block each: what it does and what each control means — toget A pair of concentric-ring "headlight" clusters (nested rings of 1/8/12/16/24 LEDs) positioned to mimic a car's front lights — a fixed arrangement composed from [Ring](#ring) geometry. -<img src="../../../assets/light/layouts/CarLightsLayout.gif" width="300" alt="Car Lights layout preview"> +<img src="../../assets/light/layouts/CarLightsLayout.gif" width="300" alt="Car Lights layout preview"> - `scale` — overall size scale (1–10). @@ -153,7 +153,7 @@ Detail: [technical](moxygen/SpiralLayout.md) Maps a set of decorative "gourd" objects (a specific bar installation), each rendered at one of three granularities — one light per gourd, per side, or per LED. -<img src="../../../assets/light/layouts/TorontoBarGourdsLayout.gif" width="300" alt="Toronto Bar Gourds layout preview"> +<img src="../../assets/light/layouts/TorontoBarGourdsLayout.gif" width="300" alt="Toronto Bar Gourds layout preview"> - `granularity` — `One Gourd One Light`, `One Side One Light`, or `One LED One Light`. - `nrOfLightsPerGourd` — LEDs per gourd in the coarsest mode (1–128). diff --git a/docs/moonmodules/light/modifiers.md b/docs/moonmodules/light/modifiers.md index ea858b67..692863f8 100644 --- a/docs/moonmodules/light/modifiers.md +++ b/docs/moonmodules/light/modifiers.md @@ -20,7 +20,7 @@ Detail: [technical](moxygen/BlockModifier.md) ### Checkerboard 💫 · static -<img src="../../../assets/light/modifiers/CheckerboardModifier.gif" width="300" alt="Checkerboard modifier preview"> +<img src="../../assets/light/modifiers/CheckerboardModifier.gif" width="300" alt="Checkerboard modifier preview"> Masks the layer in a checkerboard: "off" squares are dropped, "on" squares pass through unchanged. @@ -63,7 +63,7 @@ Detail: [technical](moxygen/MirrorModifier.md) ### Multiply 💫 · static -<img src="../../../assets/light/modifiers/MultiplyModifier.gif" width="300" alt="Multiply modifier preview"> +<img src="../../assets/light/modifiers/MultiplyModifier.gif" width="300" alt="Multiply modifier preview"> Tiles the logical image across the box `multiply` times per axis, optionally mirroring alternate tiles (a pure mirror is `multiply = 2, mirror = true`). diff --git a/docs/moonmodules/light/supporting.md b/docs/moonmodules/light/supporting.md index 5cac1bb0..89efdddd 100644 --- a/docs/moonmodules/light/supporting.md +++ b/docs/moonmodules/light/supporting.md @@ -8,7 +8,7 @@ The light-domain machinery the catalog modules (effects, modifiers, layouts, dri One rendering layer — an effect writes into its buffer, modifiers transform the coordinate mapping, and the layer composites onto the shared output. The unit the render loop iterates. -<img src="../../../assets/light/Layer.png" width="300" alt="Layer container with a child effect"> +<img src="../../assets/light/Layer.png" width="300" alt="Layer container with a child effect"> - `blendMode` — how this layer composites onto the ones below (overwrite / alpha / additive). @@ -22,7 +22,7 @@ Detail: [technical](moxygen/Layer.md) The container of layers — composites them (blend mode + opacity per layer) into the final light buffer. -<img src="../../../assets/light/Layers.png" width="300" alt="Layers container"> +<img src="../../assets/light/Layers.png" width="300" alt="Layers container"> Detail: [technical](moxygen/Layers.md) @@ -34,7 +34,7 @@ Detail: [technical](moxygen/Layers.md) The container of layout modules — walks each layout's coordinates to build the physical light set the mapping consumes. -<img src="../../../assets/light/Layouts.png" width="300" alt="Layouts container"> +<img src="../../assets/light/Layouts.png" width="300" alt="Layouts container"> Detail: [technical](moxygen/Layouts.md) @@ -46,7 +46,7 @@ Detail: [technical](moxygen/Layouts.md) The container of driver modules — owns the shared driver buffer and the per-light output correction every driver applies before sending. -<img src="../../../assets/light/Drivers.png" width="300" alt="Drivers container with the on/off + brightness controls"> +<img src="../../assets/light/Drivers.png" width="300" alt="Drivers container with the on/off + brightness controls"> - `on` — master power (default on). Turning it off scales the whole output to black while preserving `brightness`, so on restores the exact level. The single power control every consumer drives (the UI toggle, IR's on/off, the WLED app / Home Assistant, MQTT/Homebridge) through `Scheduler::setControl`. - `brightness` — global output brightness (0–255). diff --git a/docs/usecases/led-signal-integrity.md b/docs/usecases/led-signal-integrity.md index 55b9a96e..0f230f71 100644 --- a/docs/usecases/led-signal-integrity.md +++ b/docs/usecases/led-signal-integrity.md @@ -2,7 +2,7 @@ Random wrong colours on LEDs the effect leaves black — most often a few stray pixels flickering — is, on a 3.3 V ESP32 driving WS2812 **directly**, almost always a **data-line signal-integrity** problem, not a firmware bug. WS2812 wants a logic-high near 0.7 × VDD (≈ 3.5 V on a 5 V strip), but the ESP32 drives only 3.3 V, so individual bits sit at the margin and noise tips them. -Confirm the firmware is innocent **before** reaching for the soldering iron. These checks were the actual diagnosis path on the bench (recorded in [decisions.md](../history/decisions.md)): +Confirm the firmware is innocent **before** reaching for the soldering iron. These checks were the actual diagnosis path on the bench (recorded in [lessons.md](../history/lessons.md)): 1. **Is the data clean?** The preview/source buffer is the logical RGB the effect produced — if it shows no stray colour, the effect is innocent (the corruption is downstream of the buffer). 2. **Is the firmware/peripheral clean?** Run the [`loopbackFrame` self-test](../moonmodules/light/drivers.md#led-output-drivers) through a short jumper on the data pin. A `PASS` means the RMT encode + transmit emit bit-perfect WS2812 — the GPIO is fine. diff --git a/moondeck/diag/preview_health.py b/moondeck/diag/preview_health.py index d204f6d5..b67c85e1 100644 --- a/moondeck/diag/preview_health.py +++ b/moondeck/diag/preview_health.py @@ -123,7 +123,7 @@ def close(self): def measure(host, seconds, grid): # --grid resizes the device's Grid for the run; snapshot the original so we restore it afterwards - # (the probe stays non-destructive, like the live scenarios — decisions.md). The restore is in the + # (the probe stays non-destructive, like the live scenarios — lessons.md). The restore is in the # finally at the end of the function. saved_grid = _read_grid(host) if grid else {} if grid: diff --git a/src/core/DevicePlugin.h b/src/core/DevicePlugin.h index fec7b389..b4e1f5de 100644 --- a/src/core/DevicePlugin.h +++ b/src/core/DevicePlugin.h @@ -17,7 +17,7 @@ // Discovery is PASSIVE UDP: a plugin declares the broadcast port it listens on and // classifies a received datagram into a device. This replaces the former mDNS *query* // path, which destabilised our own mDNS advertise (a PTR query for a service we also host -// exhausts the IDF mDNS pool — see docs/history/decisions.md). mDNS is +// exhausts the IDF mDNS pool — see docs/adr/0006-device-discovery-udp-mdns-advertise-only.md). mDNS is // now advertise-only (so the WLED app + Home Assistant find us); discovery never queries. // // The seam covers the discovery half, with two concrete plugins (projectMM and WLED) that diff --git a/src/core/DevicesModule.h b/src/core/DevicesModule.h index 300c7910..1877842d 100644 --- a/src/core/DevicesModule.h +++ b/src/core/DevicesModule.h @@ -37,7 +37,7 @@ namespace mm { /// 65506 lists it too (discovery-only: a receiving WLED shows us in its instances /// list, it does not sync to it). This replaces the former mDNS *query* path, which /// destabilised our own mDNS advertise (a PTR query for a service we also host -/// exhausts the IDF mDNS pool — see docs/history/decisions.md). mDNS is +/// exhausts the IDF mDNS pool — see docs/adr/0006-device-discovery-udp-mdns-advertise-only.md). mDNS is /// advertise-ONLY (announcing `_http._tcp`+`mm=1` and `_wled._tcp`+`mac=` so the /// WLED native app + Home Assistant, which only browse mDNS, discover us); discovery /// never queries. diff --git a/src/core/MqttModule.cpp b/src/core/MqttModule.cpp index 103922aa..3475feff 100644 --- a/src/core/MqttModule.cpp +++ b/src/core/MqttModule.cpp @@ -2,6 +2,7 @@ #include "core/Scheduler.h" // setControl — the shared apply-core #include "core/JsonUtil.h" // json::hasKey/parseBool/parseInt/parseString — the inbound ha/set parse +#include "core/JsonSink.h" // jsonEscape — escape the editable deviceName into the discovery JSON // (same flat helpers HttpServerModule::applyWledState uses; no arena) #include "light/Palette.h" // Palettes::nearestForHue — a pure hue/sat→index CONVERSION with no // light state or objects, the one narrow reach this core module makes @@ -80,18 +81,20 @@ void MqttModule::freeDiscoveryBuffers() { } void MqttModule::publishDiscovery(bool announce) { - // Retract (OFF): free the buffers unconditionally — freeing local memory needs no socket, so this - // runs even while disconnected/reconnecting (else a discovery-off toggle mid-reconnect would strand - // the buffers until teardown, breaking "no memory when discovery is off"). When we happen to be - // connected we ALSO send the empty retained payload so HA removes the entity; when not, the buffers - // still free and the entity clears on the broker's own retained-message expiry / next connect. + // Retract (OFF): send an empty retained payload to the config topic so HA removes the entity, then + // free the buffers. The retract frames into a small LOCAL buffer (a tombstone is topic + empty + // payload, well under kSendBufLen) — NOT the on-use discoveryBuf_, which may already be freed, and + // which we must not allocate on the OFF path (the "no memory when discovery is off" rule). When + // connected the tombstone goes out immediately; when disconnected we can't send, so it is deferred + // to the next CONNACK (which retracts when haDiscovery_ is false) — the broker keeps the last + // retained config until then. Freeing always runs, connected or not. if (!announce) { - if (state_ == Conn::Connected && discoveryBuf_) { + if (state_ == Conn::Connected) { char topic[96]; buildDiscoveryTopic(topic, sizeof(topic)); - const size_t n = buildMqttPublish(topic, nullptr, 0, discoveryBuf_, kDiscoveryBufLen, - /*retain=*/true); - if (n != 0) sendPacket(discoveryBuf_, n); + uint8_t buf[kSendBufLen]; + const size_t n = buildMqttPublish(topic, nullptr, 0, buf, sizeof(buf), /*retain=*/true); + if (n == 0 || !sendPacket(buf, n)) resetConnection("error: discovery retract failed"); } freeDiscoveryBuffers(); return; @@ -112,6 +115,10 @@ void MqttModule::publishDiscovery(bool announce) { std::snprintf(id, sizeof(id), "%s_%02x%02x%02x", prefixRoot_, mac[3], mac[4], mac[5]); const char* dn = systemModule_ ? systemModule_->deviceName() : nullptr; if (!dn || !dn[0]) dn = id; + // The deviceName is user-editable: a quote/backslash in it would produce invalid JSON. Escape it + // once (both name fields use it) with the shared jsonEscape — worst case doubles the 32-char name. + char dnEsc[72]; + jsonEscape(dn, dnEsc, sizeof(dnEsc)); char cmd[80], stat[80], avty[80]; buildTopic(cmd, sizeof(cmd), "ha/set"); @@ -124,14 +131,16 @@ void MqttModule::publishDiscovery(bool announce) { "{\"schema\":\"json\",\"name\":\"%s\",\"uniq_id\":\"%s\",\"cmd_t\":\"%s\"," "\"stat_t\":\"%s\",\"avty_t\":\"%s\",\"brightness\":true," "\"dev\":{\"ids\":[\"%s\"],\"name\":\"%s\",\"mf\":\"MoonModules\",\"mdl\":\"projectMM\"}}", - dn, id, cmd, stat, avty, id, dn); + dnEsc, id, cmd, stat, avty, id, dnEsc); if (pn <= 0 || static_cast<size_t>(pn) >= kDiscoveryPayloadLen) return; // truncated → don't send a broken config const size_t n = buildMqttPublish(topic, reinterpret_cast<const uint8_t*>(discoveryPayload_), static_cast<size_t>(pn), discoveryBuf_, kDiscoveryBufLen, /*retain=*/true); if (n == 0) { setStatusLine("error: discovery config too large"); return; } - sendPacket(discoveryBuf_, n); + // Same "reset on a failed send" contract as the ping / subscribe / state paths: a partial write + // means a wedged socket, so drop the connection rather than leave a truncated frame on the stream. + if (!sendPacket(discoveryBuf_, n)) resetConnection("error: discovery publish failed"); } // SUBSCRIBE to <prefix>/ha/set — the HA-native JSON command topic. Called at CONNACK (with the @@ -143,7 +152,7 @@ void MqttModule::subscribeHaSet() { buildTopic(topic, sizeof(topic), "ha/set"); uint8_t buf[kSendBufLen]; const size_t n = buildMqttSubscribe(nextPacketId_++, topic, buf, sizeof(buf)); - if (n != 0) sendPacket(buf, n); + if (n == 0 || !sendPacket(buf, n)) resetConnection("error: ha subscribe failed"); } void MqttModule::setup() { @@ -399,9 +408,13 @@ void MqttModule::handleInboundByte(uint8_t byte) { uint8_t sb[kSendBufLen]; const size_t sn = buildMqttPublish(st, reinterpret_cast<const uint8_t*>("online"), 6, sb, sizeof(sb), /*retain=*/true); - if (sn != 0) sendPacket(sb, sn); + if (sn == 0 || !sendPacket(sb, sn)) { resetConnection("error: availability publish failed"); return; } } + // On connect: announce + subscribe when discovery is on; when it's OFF, retract instead — a + // config retained from a previous session (discovery was on, then turned off while offline) + // would otherwise keep HA's entity alive across this reconnect. if (haDiscovery_) { publishDiscovery(true); subscribeHaSet(); } + else { publishDiscovery(false); } publishState(true); // publish initial state so mqttthing + HA show it } else if (type == static_cast<uint8_t>(MqttPacketType::Publish)) { const char* topic = nullptr; const uint8_t* payload = nullptr; size_t plLen = 0; @@ -431,8 +444,10 @@ void MqttModule::routePublish(const char* topic, const uint8_t* payload, size_t if (json::hasKey(body, "state")) { char st[8] = ""; json::parseString(body, "state", st, sizeof(st)); - const bool on = (std::strcmp(st, "ON") == 0); - setControlValue("on", on ? "{\"value\":true}" : "{\"value\":false}"); + // HA sends exactly "ON"/"OFF"; act only on those. A malformed or truncated value is + // ignored (not silently treated as OFF), so a bad payload never turns the light off. + if (std::strcmp(st, "ON") == 0) setControlValue("on", "{\"value\":true}"); + else if (std::strcmp(st, "OFF") == 0) setControlValue("on", "{\"value\":false}"); } if (json::hasKey(body, "brightness")) { int bri = json::parseInt(body, "brightness"); @@ -442,7 +457,8 @@ void MqttModule::routePublish(const char* topic, const uint8_t* payload, size_t std::snprintf(json, sizeof(json), "{\"value\":%d}", bri); setControlValue("brightness", json); } - // Future: an "effect" key → setControlValue("palette"/"effect", …) with effect_list in the config. + // The JSON schema carries every field in one message, so a new control is another key here + // (e.g. an "effect" key routing to setControlValue) rather than another topic. return; } diff --git a/src/core/MqttModule.h b/src/core/MqttModule.h index 9ae36c0e..d4f88f84 100644 --- a/src/core/MqttModule.h +++ b/src/core/MqttModule.h @@ -42,9 +42,9 @@ namespace mm { /// `<prefix>/status` → retained "online"; the CONNECT **Last-Will** publishes "offline" here /// on an ungraceful drop, so HA greys the entity out (`avty_t`) /// Toggling `haDiscovery` re-announces / retracts live (an empty retained config removes the entity), -/// no reconnect. `uniq_id` is the MAC-stable `projectMM_<mac6>`, never the editable name. JSON schema -/// (not the default schema) so future controls add a key — HA's native `effect`/`effect_list` maps a -/// preset/effect picker with no new topic. +/// no reconnect. `uniq_id` is the MAC-stable `projectMM_<mac6>`, never the editable name. The schema is +/// JSON (not the default schema): a control maps to a config key rather than a topic, so HA's native +/// `effect`/`effect_list` renders a picker on the one command topic — the reason JSON is chosen here. /// /// **Lifecycle** (all on loop1s(), off the render hot path — MQTT is slow control): connect lazily /// once `networkReady() && enabled`, CONNECT → CONNACK → SUBSCRIBE to the `set` topics, PINGREQ every @@ -80,6 +80,10 @@ class MqttModule : public MoonModule { void enableSendCaptureForTest(uint8_t* buf, size_t cap); size_t sentCaptureLenForTest() const { return sendCaptureLen_; } + /// The heap footprint dynamicBytes() reports while HA discovery is announcing — the sum of the two + /// discovery scratch regions. Exposed so a test asserts against this instead of a magic literal. + static constexpr size_t kDiscoveryDynamicBytes = 320 + 448; + private: // Connection state machine — advanced by loop1s(). ConnectingTcp = a non-blocking TCP connect is // in flight (polled, never blocks the tick); Connecting = TCP up, CONNECT sent, awaiting CONNACK. @@ -152,6 +156,8 @@ class MqttModule : public MoonModule { // the framed packet into discoveryBuf_. static constexpr size_t kDiscoveryPayloadLen = 320; static constexpr size_t kDiscoveryBufLen = 448; + static_assert(kDiscoveryDynamicBytes == kDiscoveryPayloadLen + kDiscoveryBufLen, + "public test constant must track the actual buffer sizes"); char* discoveryPayload_ = nullptr; uint8_t* discoveryBuf_ = nullptr; bool ensureDiscoveryBuffers(); // lazily alloc both; false on OOM. Sets dynamicBytes. diff --git a/src/light/drivers/HueDriver.h b/src/light/drivers/HueDriver.h index 7ffc2201..bbceb89b 100644 --- a/src/light/drivers/HueDriver.h +++ b/src/light/drivers/HueDriver.h @@ -102,7 +102,7 @@ class HueDriver : public DriverBase { /// Runs every render tick, but does at most ONE bounded PUT and only when the rate-limit /// interval has elapsed (a millis() gate, NOT work-every-tick) — otherwise a synchronous HTTP /// round-trip would stall the single-thread render loop (the "never block the loop" rule, - /// decisions.md). One PUT every kPutIntervalMs, round-robined across the lights; pairing + the + /// lessons.md). One PUT every kPutIntervalMs, round-robined across the lights; pairing + the /// bridge announce ride the slow 1 Hz tick. void loop() override { if (pairTicksLeft_ > 0) return; // pairing owns the bridge during its window diff --git a/src/light/drivers/LcdLedDriver.h b/src/light/drivers/LcdLedDriver.h index a8c9a9ac..5247eaea 100644 --- a/src/light/drivers/LcdLedDriver.h +++ b/src/light/drivers/LcdLedDriver.h @@ -33,7 +33,7 @@ class LcdLedDriver : public ParallelLedDriver<LcdLedDriver> { // runs to whatever GPIOs the user wired), so a hard-coded default would be a // guess that could drive a pin the user committed elsewhere — empty until set, // the driver idles meanwhile (the "default only when it cannot do harm" rule; - // see decisions.md). The ESP32-S3 N16R8 Dev bench wiring is pins "1,2,4,5,6,7,8,9", + // see lessons.md). The ESP32-S3 N16R8 Dev bench wiring is pins "1,2,4,5,6,7,8,9", // loopbackRxPin 12 (kept clear of the octal-PSRAM pins 26-37, USB 19/20, and // strapping pins) — set those again to reproduce the bench. (Base declares // pins="" and loopbackRxPin=0, so the empty default needs no code here.) diff --git a/src/light/drivers/ParlioLedDriver.h b/src/light/drivers/ParlioLedDriver.h index 3b77de40..8235761f 100644 --- a/src/light/drivers/ParlioLedDriver.h +++ b/src/light/drivers/ParlioLedDriver.h @@ -25,7 +25,7 @@ class ParlioLedDriver : public ParallelLedDriver<ParlioLedDriver> { // needed (the base default-initialises them). Pins/loopback are unset because the // strand is user-soldered: a hard-coded pin would guess the user's wiring and // could drive a pin committed elsewhere ("default only when it cannot do harm", - // see decisions.md). The P4-NANO bench used pins "20,21,22,23,24,25,26,27", + // see lessons.md). The P4-NANO bench used pins "20,21,22,23,24,25,26,27", // loopbackRxPin 33 (clear of the NANO's strapping 34-38, Ethernet RMII // 28-31/49-52, C6 SDIO 14-19/54, I2C 7-8 — clear GPIOs are 20-27, 32-33, 39-48). diff --git a/src/light/drivers/RmtLedDriver.h b/src/light/drivers/RmtLedDriver.h index 1876c764..05504ec3 100644 --- a/src/light/drivers/RmtLedDriver.h +++ b/src/light/drivers/RmtLedDriver.h @@ -52,7 +52,7 @@ class RmtLedDriver : public DriverBase { /// field and the driver idles. 24 bytes fit kMaxPins 2-digit GPIOs plus separators. Defaults /// to UNSET: the strand is user-soldered to whatever GPIO the user wired, so a hard-coded pin /// would be a guess that could drive a pin committed elsewhere — empty until set, idle - /// meanwhile (the "default only when it cannot do harm" rule; see decisions.md). Bench pin "18". + /// meanwhile (the "default only when it cannot do harm" rule; see lessons.md). Bench pin "18". char pins[24] = ""; /// Comma-separated lights-per-pin ("100,100,50"), matched to `pins` by position — each pin diff --git a/src/platform/esp32/platform_esp32.cpp b/src/platform/esp32/platform_esp32.cpp index d088f39e..6f479913 100644 --- a/src/platform/esp32/platform_esp32.cpp +++ b/src/platform/esp32/platform_esp32.cpp @@ -1165,7 +1165,7 @@ void mdnsShutdown() { // a projectMM device broadcasts and listens for the 44-byte presence packet on UDP 65506. // Keeping discovery off mDNS also keeps the advertise stable, because a PTR query for a // service this device -// also hosts destabilises our own advertise — see docs/history/decisions.md. +// also hosts destabilises our own advertise — see docs/history/lessons.md. // Outbound HTTP request (plain HTTP, LAN, no TLS) — see platform.h. A bounded blocking lwIP // socket call; the caller (HueDriver) runs it off the render path on loop1s. Mirrors the diff --git a/src/platform/esp32/platform_esp32_lcd.cpp b/src/platform/esp32/platform_esp32_lcd.cpp index 86587463..6b470e43 100644 --- a/src/platform/esp32/platform_esp32_lcd.cpp +++ b/src/platform/esp32/platform_esp32_lcd.cpp @@ -16,7 +16,7 @@ // it too; the driver never calls in thanks to platform::lcdLanes == 0). Gate on // SOC_LCDCAM_I80_LCD_SUPPORTED, NOT SOC_LCD_I80_SUPPORTED: the classic ESP32 sets // the latter for its unrelated I2S-LCD peripheral, which wired this driver onto a -// chip with no LCD_CAM and hung its boot (see platform_config.h + decisions.md). +// chip with no LCD_CAM and hung its boot (see platform_config.h + lessons.md). #include "platform/platform.h" diff --git a/test/python/test_mkdocs_slug.py b/test/python/test_mkdocs_slug.py index ea3257df..22b0faab 100644 --- a/test/python/test_mkdocs_slug.py +++ b/test/python/test_mkdocs_slug.py @@ -46,3 +46,41 @@ def test_slug_resolves_to_the_real_rendered_heading_id(): for heading in ["LED output — details", "GEQ · 3D — details"]: html = markdown.markdown(f"## {heading}", extensions=["toc"]) assert f'id="{_slug(heading)}"' in html, heading + + +# --- Catalog-page card images resolve on disk --- +# The catalog pages hand-author each card's preview as a raw `<img src="../../assets/…">`. +# MkDocs' --strict validates markdown `![]()` links but NOT raw-HTML `<img src>`, and the +# catalog hook moves the tag into a table cell without touching its src — so a wrong `../` +# depth ships a SILENT 404 on the live site (the exact bug this test pins: the pages sit one +# level shallower than the generated moxygen pages, so they need `../../assets`, not the +# moxygen pages' `../../../assets`). This test resolves every catalog `<img src>` against its +# page's own directory and fails on any that doesn't exist on disk. +import re # noqa: E402 + +_DOCS = ROOT / "docs" +_IMG_SRC_RE = re.compile(r'<img[^>]+src="([^"]+)"') + + +def _catalog_pages(): + return sorted( + (_DOCS / "moonmodules" / domain / f"{group}.md") + for domain, groups in { + "core": ("services", "supporting", "ui"), + "light": ("effects", "modifiers", "layouts", "drivers", "supporting"), + }.items() + for group in groups + if (_DOCS / "moonmodules" / domain / f"{group}.md").exists() + ) + + +def test_catalog_card_images_resolve_on_disk(): + missing = [] + for page in _catalog_pages(): + for src in _IMG_SRC_RE.findall(page.read_text(encoding="utf-8")): + if src.startswith(("http://", "https://", "data:", "/")): + continue + target = (page.parent / src.split("#")[0]).resolve() + if not target.exists(): + missing.append(f"{page.relative_to(ROOT)} -> {src}") + assert not missing, "catalog card image(s) 404 on the site:\n " + "\n ".join(missing) diff --git a/test/unit/core/unit_MqttModule.cpp b/test/unit/core/unit_MqttModule.cpp index 3cab8b81..13b0257e 100644 --- a/test/unit/core/unit_MqttModule.cpp +++ b/test/unit/core/unit_MqttModule.cpp @@ -77,6 +77,35 @@ struct Rig { } }; +// Walk the concatenated MQTT packet stream the capture holds and return the fixed-header first byte +// (type nibble + flags) of the first PUBLISH whose topic equals `wantTopic`, or -1 if none. Lets a +// test assert the RETAIN bit (bit 0, §3.3.1.3) rather than only string-matching the payload — a +// regression dropping retain=true flips this bit but leaves every substring intact. +int publishFlagsForTopic(const uint8_t* buf, size_t len, const char* wantTopic) { + size_t i = 0; + while (i < len) { + const uint8_t first = buf[i]; + // Remaining Length: a 1–4 byte varint (§2.2.3). + size_t j = i + 1, mult = 1, remLen = 0; + for (int b = 0; b < 4 && j < len; b++, j++) { + remLen += (buf[j] & 0x7F) * mult; + if (!(buf[j] & 0x80)) { j++; break; } + mult *= 128; + } + const size_t body = j; // first byte after the fixed header + if ((first & 0xF0) == 0x30 && body + 2 <= len) { // PUBLISH + const size_t topicLen = (size_t(buf[body]) << 8) | buf[body + 1]; + if (body + 2 + topicLen <= len && + std::strncmp(reinterpret_cast<const char*>(buf + body + 2), wantTopic, topicLen) == 0 && + std::strlen(wantTopic) == topicLen) { + return first; + } + } + i = body + remLen; + } + return -1; +} + } // namespace TEST_CASE("MqttModule: on/set drives Drivers.on") { @@ -170,7 +199,7 @@ TEST_CASE("MqttModule: CONNACK publishes a retained HA discovery config") { r.mqtt->feedForTest(connack, sizeof(connack)); const size_t len = r.mqtt->sentCaptureLenForTest(); REQUIRE(len > 0); - // The captured stream must contain the discovery topic, the retain bit, and the key config fields. + // The captured stream must contain the discovery topic + the key config fields. std::string sent(reinterpret_cast<const char*>(cap), len); CHECK(sent.find("homeassistant/light/projectMM_efcafe/config") != std::string::npos); CHECK(sent.find("\"schema\":\"json\"") != std::string::npos); @@ -179,6 +208,14 @@ TEST_CASE("MqttModule: CONNACK publishes a retained HA discovery config") { CHECK(sent.find("projectMM/efcafe/ha/state") != std::string::npos); // stat_t CHECK(sent.find("projectMM/efcafe/status") != std::string::npos); // avty_t CHECK(sent.find("online") != std::string::npos); // the retained availability publish + // The discovery config AND the availability publish must carry the RETAIN bit (bit 0 of the PUBLISH + // fixed header) — a late-joining HA reads the retained config/state, so dropping retain breaks it. + const int cfgFlags = publishFlagsForTopic(cap, len, "homeassistant/light/projectMM_efcafe/config"); + REQUIRE(cfgFlags >= 0); + CHECK((cfgFlags & 0x01) == 0x01); // discovery config retained + const int avtyFlags = publishFlagsForTopic(cap, len, "projectMM/efcafe/status"); + REQUIRE(avtyFlags >= 0); + CHECK((avtyFlags & 0x01) == 0x01); // availability "online" retained } // Regression (found live on P4/S31 hardware): turning haDiscovery OFF must free the discovery buffers @@ -193,11 +230,11 @@ TEST_CASE("MqttModule: retract frees the discovery buffers even while disconnect // CONNACK-accept → Connected → announce allocates the discovery buffers (448 + 320 = 768). const uint8_t connack[] = {0x20, 0x02, 0x00, 0x00}; r.mqtt->feedForTest(connack, sizeof(connack)); - CHECK(r.mqtt->dynamicBytes() == 768); + CHECK(r.mqtt->dynamicBytes() == MqttModule::kDiscoveryDynamicBytes); // A broker change re-homes the socket → resetConnection drops state to Idle, buffers still held // (a reconnect must not churn the heap). Now we're "allocated but not Connected". Scheduler::instance()->setControl("Mqtt", "broker", "{\"value\":\"10.0.0.9\"}"); - CHECK(r.mqtt->dynamicBytes() == 768); // reset kept the buffers (correct) + CHECK(r.mqtt->dynamicBytes() == MqttModule::kDiscoveryDynamicBytes); // reset kept the buffers (correct) // Toggle discovery OFF while disconnected — must free despite no live socket. Scheduler::instance()->setControl("Mqtt", "haDiscovery", "{\"value\":false}"); CHECK(r.mqtt->dynamicBytes() == 0); // the fix: retract freed even while not Connected From 7c6f1a05346a175d7691da3cda2c866347581166 Mon Sep 17 00:00:00 2001 From: ewowi <ewowi@icloud.com> Date: Tue, 7 Jul 2026 10:18:54 +0200 Subject: [PATCH 3/3] Pre-merge: CodeRabbit present-tense fixes, carry-forward lessons, MoonDeck DX MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Processes the second CodeRabbit round (present-tense in non-exempt files) and the pre-merge gates for PR #39: two hard-won lessons carried forward, and two MoonDeck developer-experience fixes surfaced while testing the branch. Core: - ParlioLedDriver / platform_esp32_lcd: reword two bench comments to present tense (the P4-NANO bench pin map, and the LCD_CAM SOC-macro guard) so they read as current configuration notes, not history. - platform_esp32: retarget the mDNS-pool comment from lessons.md to ADR-0006 (the device-discovery decision that lesson became), matching its DevicePlugin/DevicesModule siblings. Scripts / MoonDeck: - moondeck.py: free port 8420 from any prior instance on startup, so re-running MoonDeck replaces the old server instead of failing to bind; also picks up deviceModels.json edits (the catalog loads at import). MOONDECK_NO_KILL=1 opts out. Targets the port owner, not a name match, so it can never self-kill. - moondeck.py + app.js: the discover response now reports which subnet was scanned + how many devices answered, so a wrong/unset-network scan (finds nothing on an empty subnet) reads differently from "no devices online" — the discover log names the subnet and, on zero hits, hints to check the machine's subnet / VPN. Transient fields, not persisted. Docs / CI: - CLAUDE.md: add docs/adr/ to the present-tense exemption (and the principles-audit sweep) alongside history/backlog/lessons.md — an ADR's Context is the situation that forced the decision, a before/after contrast by nature, the same reason lessons.md is exempt. This closes the policy gap the ADR directory introduced. - led-signal-integrity.md: reword one sentence to present tense ("these checks are the bench diagnosis path"). - lessons.md: carry forward two branch lessons — a retract/cleanup path gated on "connected" strands the resource when disconnected (the HA discovery bug, found live on P4/S31), and raw-HTML <img src> is not validated by MkDocs --strict so a wrong ../ depth ships a silent 404. Reviews: - 🐇 present-tense in led-signal-integrity.md / ParlioLedDriver.h / platform_esp32_lcd.cpp (3 findings): fixed, reworded present-tense. - 🐇 present-tense in ADRs 0001/0003/0004 (3 findings): accepted, not changed — an ADR's Context section is a point-in-time record whose before-state contrast is the decision's justification (Nygard format); root cause fixed instead by exempting docs/adr/ in CLAUDE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- CLAUDE.md | 4 +- docs/history/lessons.md | 8 ++++ docs/usecases/led-signal-integrity.md | 2 +- moondeck/moondeck.py | 51 +++++++++++++++++++++++ moondeck/moondeck_ui/app.js | 9 ++++ src/light/drivers/ParlioLedDriver.h | 2 +- src/platform/esp32/platform_esp32.cpp | 2 +- src/platform/esp32/platform_esp32_lcd.cpp | 4 +- 8 files changed, 75 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index acf2799a..96854362 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,7 +22,7 @@ See `docs/architecture.md` for system design. This file contains only rules and - **Robust to any input.** A running device tolerates any sequence of UI actions or API calls: add, delete, replace, or reconfigure any module in any order, at any grid size, and it keeps running. Degraded or idle is acceptable; crashed is not. This robustness is a defining strongpoint of projectMM, and it's guarded by the test framework, not by hope: a discovered crash drives a new test that pins the fix (see the Hard Rule). Out of scope: power loss, malformed OTA, brown-out, and other physical/electrical faults the firmware can't intercept; this principle is about what the software accepts as input. - **No reboot to apply a configuration change.** Every setting takes effect live, on the next render tick — change a pin map, a strand length, an output protocol, a mic pin or rate, anything, on a running device and it just works. There is no init-once-at-boot step, and no *config* change requires a restart, which sets projectMM apart from most LED-controller firmware (where a pin or protocol change means a reboot). Like robustness, this is a defining strongpoint, and it falls out of the architecture for free rather than being hand-built per module: any control whose change reshapes derived state routes through the generic `onBuildState()` rebuild sweep, so drivers, the audio peripheral, effects, layouts, modifiers and network I/O all inherit it. When adding a feature, don't reach for a reboot/restart to apply config; make the change live. Full mechanism + rationale: [architecture.md § Live reconfiguration](docs/architecture.md#live-reconfiguration-every-change-applies-without-a-reboot). The one exception is what you'd expect: a *firmware* OTA flash swaps the binary and needs the usual power cycle — that's not a configuration change, and (like power loss and brown-out) it's the same physical-fault boundary the robustness principle draws. - **Domain-neutral core.** Separate core infrastructure from the light domain as much as practical. When mixing is necessary, use domain-neutral naming so the code stays open to future separation. -- **Present tense only.** Code, comments, and documentation describe the system as it is now. No changelogs, no roadmaps. History lives in git commits. This bans not just future-tense ("will be", "planned") but **absence-narration**: phrases like "no longer", "anymore", "formerly", "used to", "X was removed", or "there's no longer a Y" describe a *change from a past state* a present-tense reader never saw — state what *is*, not what stopped being. (The test: "there is no MCLK pin" is a present-tense *property* — keep it; "there's no SET_DEVICE_MODEL RPC anymore" narrates a removal — cut it, just describe the path that exists.) Exceptions: `docs/backlog/` (forward-looking) and `docs/history/` (backward-looking) — and `lessons.md` lessons, which legitimately contrast before/after because the contrast *is* the lesson. +- **Present tense only.** Code, comments, and documentation describe the system as it is now. No changelogs, no roadmaps. History lives in git commits. This bans not just future-tense ("will be", "planned") but **absence-narration**: phrases like "no longer", "anymore", "formerly", "used to", "X was removed", or "there's no longer a Y" describe a *change from a past state* a present-tense reader never saw — state what *is*, not what stopped being. (The test: "there is no MCLK pin" is a present-tense *property* — keep it; "there's no SET_DEVICE_MODEL RPC anymore" narrates a removal — cut it, just describe the path that exists.) Exceptions: `docs/backlog/` (forward-looking) and `docs/history/` (backward-looking) — and `docs/adr/` + `lessons.md`, which legitimately contrast before/after because the contrast *is* the record (an ADR's Context is the situation that forced the decision; a lesson's before-state is what makes it a lesson). ## Hard Rules @@ -179,7 +179,7 @@ The "end users will use this" moment. Per-release criteria are defined by the pr 5. **Changelog / release notes**: drafted in the GitHub release body. Skip only for unreleased pre-1.0 tags. 6. **Cross-platform smoke**: run scenarios on every supported platform (today: PC + ESP32; later: + Teensy, RPi), if the release claims new platform support or the version bumps a major or minor. -7. **Principles audit**: sweep `docs/` (except `docs/backlog/` and `docs/history/`) and `src/` for forward-looking language ("roadmap", "will be", "planned", "in the future", "currently lacks", `TODO`, `FIXME`) and other violations of § Principles. Acceptable hits carry a one-line justification; the rest get rewritten present-tense or moved to `docs/backlog/` / `docs/history/`. The reviewer agent can run this end-to-end. Skip only for releases where the diff against the previous tag is doc-empty. +7. **Principles audit**: sweep `docs/` (except `docs/backlog/`, `docs/history/`, and `docs/adr/`) and `src/` for forward-looking language ("roadmap", "will be", "planned", "in the future", "currently lacks", `TODO`, `FIXME`) and other violations of § Principles. Acceptable hits carry a one-line justification; the rest get rewritten present-tense or moved to `docs/backlog/` / `docs/history/`. The reviewer agent can run this end-to-end. Skip only for releases where the diff against the previous tag is doc-empty. What the agent reads: - Always: `CLAUDE.md`, `architecture.md` diff --git a/docs/history/lessons.md b/docs/history/lessons.md index b7af8ef9..e66fa643 100644 --- a/docs/history/lessons.md +++ b/docs/history/lessons.md @@ -234,3 +234,11 @@ After changing the `ethType` firmware default to None and reflashing an already- ## Fixing "blocking in the tick" at the connect misses the same violation at the write The MQTT client's first version blocked the render loop twice, a synchronous `connect()` (with DNS) *and* an unbounded blocking `write()` retry, both inside `loop1s` (which runs in `Scheduler::tick`). The first review caught the connect (reworked to non-blocking `connectStart`/`connectPoll`); the re-review caught that `conn_.write()` was still there: a zero-window broker (accepts TCP, stops reading) fills the send buffer and the next `write` never returns, freezing the render loop permanently, and the keepalive timeout can't even fire because control never comes back. Fix: sends go through a non-blocking `writeSome` with an all-or-reset rule (control packets ≤256 B, so atomic-send is realistic). A "no blocking in the hot path" audit must sweep every syscall the path can reach, connect, DNS, read, *and* write, not just the loudest one; the re-review exists to catch exactly the partial fix that reads as complete. + +## A retract/cleanup path gated on "connected" strands the resource when disconnected + +HA MQTT discovery's toggle-off publishes an empty retained config (so HA removes the entity) then frees the 768 B discovery buffers. The first cut guarded the whole path on `state_ == Conn::Connected`, so toggling discovery off *while the socket was mid-reconnect* returned before freeing, stranding the buffers until teardown, and never sent the tombstone, so the broker kept the stale retained config and HA kept the entity. Found live on P4/S31 hardware; every unit test had driven retract from the connected state, so the disconnected path had zero coverage. Fix: the retract frees unconditionally (freeing local memory needs no socket) and CONNACK retracts when discovery is off (so a reconnect clears a config left retained from a previous session); only the empty-retained PUBLISH itself needs a live link. General shape: a cleanup/retract path that also releases a resource must not gate the *release* on connection state, only the *network send*; and a path only ever exercised in one state (connected) needs an explicit test in the other. (`unit_MqttModule` "retract frees even while disconnected" pins it.) + +## Raw-HTML `<img src>` is not validated by MkDocs `--strict`, so a wrong `../` depth ships a silent 404 + +The catalog/summary pages hand-author each card image as a raw `<img src="../../assets/...">`, and the build hook moves the tag into a table cell without touching its src. MkDocs `--strict` validates markdown `![]()` links but NOT raw-HTML `<img src>`, so 34 card images with a wrong `../` depth (the catalog pages sit one level shallower than the generated moxygen pages, so they need `../../assets`, not the moxygen pages' `../../../assets`) resolved to a nonexistent repo-root `/assets/` and 404'd on the live site while every gate passed green. The desktop build, ctest, and `--strict` all miss it; only loading the deployed page shows it. Fix + guard: a `test/python` check resolves every catalog `<img src>` on disk (`test_mkdocs_slug.py::test_catalog_card_images_resolve_on_disk`). General: a generated-site check that only validates the syntaxes its linter knows leaves the others as silent-404 territory, verify the actual rendered output, or gate the un-validated syntax explicitly. diff --git a/docs/usecases/led-signal-integrity.md b/docs/usecases/led-signal-integrity.md index 0f230f71..7b0000c8 100644 --- a/docs/usecases/led-signal-integrity.md +++ b/docs/usecases/led-signal-integrity.md @@ -2,7 +2,7 @@ Random wrong colours on LEDs the effect leaves black — most often a few stray pixels flickering — is, on a 3.3 V ESP32 driving WS2812 **directly**, almost always a **data-line signal-integrity** problem, not a firmware bug. WS2812 wants a logic-high near 0.7 × VDD (≈ 3.5 V on a 5 V strip), but the ESP32 drives only 3.3 V, so individual bits sit at the margin and noise tips them. -Confirm the firmware is innocent **before** reaching for the soldering iron. These checks were the actual diagnosis path on the bench (recorded in [lessons.md](../history/lessons.md)): +Confirm the firmware is innocent **before** reaching for the soldering iron. These checks are the bench diagnosis path (recorded in [lessons.md](../history/lessons.md)): 1. **Is the data clean?** The preview/source buffer is the logical RGB the effect produced — if it shows no stray colour, the effect is innocent (the corruption is downstream of the buffer). 2. **Is the firmware/peripheral clean?** Run the [`loopbackFrame` self-test](../moonmodules/light/drivers.md#led-output-drivers) through a short jumper on the data pin. A `PASS` means the RMT encode + transmit emit bit-perfect WS2812 — the GPIO is fine. diff --git a/moondeck/moondeck.py b/moondeck/moondeck.py index 9fbb1105..b089f16d 100644 --- a/moondeck/moondeck.py +++ b/moondeck/moondeck.py @@ -761,6 +761,44 @@ def _kill_process_by_name(name: str): subprocess.run(["pkill", "-f", name], capture_output=True) +def _free_port(port: int): + """SIGTERM whatever already holds `port`, so a re-run of MoonDeck replaces the + prior instance instead of failing to bind ("Address already in use"). Targets + the port's owner (not a name match), so it can only hit a stale server, never + the current process (we haven't bound yet). Cross-platform: `lsof` on + macOS/Linux, `netstat` on Windows; a no-op if the port is free or the lookup + tool is absent.""" + pids: set[int] = set() + try: + if _IS_WIN: + out = subprocess.run(["netstat", "-ano", "-p", "TCP"], + capture_output=True, text=True).stdout + for line in out.splitlines(): + parts = line.split() + if len(parts) >= 5 and parts[1].endswith(f":{port}") and "LISTEN" in line: + with suppress(ValueError): + pids.add(int(parts[-1])) + else: + # -t: pids only, -sTCP:LISTEN: only the listener (not clients connected to it). + out = subprocess.run(["lsof", "-t", f"-iTCP:{port}", "-sTCP:LISTEN"], + capture_output=True, text=True).stdout + for tok in out.split(): + with suppress(ValueError): + pids.add(int(tok)) + except (OSError, FileNotFoundError): + return # no lookup tool → let the bind fail with its normal error + for pid in pids: + if pid == os.getpid(): + continue # never ourselves + with suppress(OSError, ProcessLookupError): + if _IS_WIN: + subprocess.run(["taskkill", "/F", "/PID", str(pid)], capture_output=True) + else: + os.kill(pid, signal.SIGTERM) + if pids: + print(f"Freed port {port} from a prior instance (pid {', '.join(map(str, pids))}).") + + def kill_script(script_id: str): with _lock: proc = _running.pop(script_id, None) @@ -1173,6 +1211,12 @@ def _merge_discover(state): # Fire pushes outside the lock — the state write has already # landed; pushes are best-effort device-side mirroring. _push_devices_in_parallel(pushes) + # Report WHICH subnet was scanned + how many answered, so a "found + # nothing" is distinguishable from "scanned the wrong network" (the + # scan uses the machine's auto-detected /24, not the active network's + # subnet). Transient, not persisted (the save already happened above). + result = {**result, "_scanned_subnet": scanned_subnet, + "_found_count": len(devices)} self._send_json(result) elif self.path == "/api/refresh": @@ -2140,6 +2184,13 @@ def _serve_static(self): # --------------------------------------------------------------------------- def main(): + # A prior MoonDeck already bound to PORT would make the fresh bind fail + # ("Address already in use"); replace it so a re-run just works. Also picks up + # any deviceModels.json / firmwares.json edits, since the catalog is loaded at + # module import (see _load_device_models). Set MOONDECK_NO_KILL=1 to opt out and + # run a second instance on a different PORT instead. + if not os.environ.get("MOONDECK_NO_KILL"): + _free_port(PORT) # ThreadingHTTPServer binds to "" → all interfaces, so MoonDeck is reachable # from other devices on the LAN, not just this machine. server = http.server.ThreadingHTTPServer(("", PORT), MoonDeckHandler) diff --git a/moondeck/moondeck_ui/app.js b/moondeck/moondeck_ui/app.js index 2e99464a..18a0f276 100644 --- a/moondeck/moondeck_ui/app.js +++ b/moondeck/moondeck_ui/app.js @@ -713,6 +713,15 @@ document.getElementById("discover-btn").addEventListener("click", async () => { // Client-side merging used to live here; moving it server-side keeps the // network-membership rule in one place. state = await resp.json(); + // Report which subnet was actually scanned + how many answered, so a + // wrong/unset-network scan (finds nothing on a subnet with no devices) + // reads differently from "no devices online". The scan uses the machine's + // auto-detected /24 — see the server's _scanned_subnet. + const scanned = state._scanned_subnet; + const found = state._found_count ?? 0; + if (scanned) appendLog(`Scanned ${scanned}.1-254 — ${found} device(s) answered\n`); + if (scanned && found === 0) + appendLog(` (nothing on ${scanned}.x — check this machine is on the devices' subnet, no VPN)\n`); renderNetworkBar(); renderDevices(); refreshPorts(); diff --git a/src/light/drivers/ParlioLedDriver.h b/src/light/drivers/ParlioLedDriver.h index 8235761f..540cbcd4 100644 --- a/src/light/drivers/ParlioLedDriver.h +++ b/src/light/drivers/ParlioLedDriver.h @@ -25,7 +25,7 @@ class ParlioLedDriver : public ParallelLedDriver<ParlioLedDriver> { // needed (the base default-initialises them). Pins/loopback are unset because the // strand is user-soldered: a hard-coded pin would guess the user's wiring and // could drive a pin committed elsewhere ("default only when it cannot do harm", - // see lessons.md). The P4-NANO bench used pins "20,21,22,23,24,25,26,27", + // see lessons.md). The P4-NANO bench uses pins "20,21,22,23,24,25,26,27", // loopbackRxPin 33 (clear of the NANO's strapping 34-38, Ethernet RMII // 28-31/49-52, C6 SDIO 14-19/54, I2C 7-8 — clear GPIOs are 20-27, 32-33, 39-48). diff --git a/src/platform/esp32/platform_esp32.cpp b/src/platform/esp32/platform_esp32.cpp index 6f479913..5f306e82 100644 --- a/src/platform/esp32/platform_esp32.cpp +++ b/src/platform/esp32/platform_esp32.cpp @@ -1165,7 +1165,7 @@ void mdnsShutdown() { // a projectMM device broadcasts and listens for the 44-byte presence packet on UDP 65506. // Keeping discovery off mDNS also keeps the advertise stable, because a PTR query for a // service this device -// also hosts destabilises our own advertise — see docs/history/lessons.md. +// also hosts destabilises our own advertise — see docs/adr/0006-device-discovery-udp-mdns-advertise-only.md. // Outbound HTTP request (plain HTTP, LAN, no TLS) — see platform.h. A bounded blocking lwIP // socket call; the caller (HueDriver) runs it off the render path on loop1s. Mirrors the diff --git a/src/platform/esp32/platform_esp32_lcd.cpp b/src/platform/esp32/platform_esp32_lcd.cpp index 6b470e43..3f877f82 100644 --- a/src/platform/esp32/platform_esp32_lcd.cpp +++ b/src/platform/esp32/platform_esp32_lcd.cpp @@ -15,8 +15,8 @@ // SOC_LCDCAM_I80_LCD_SUPPORTED with inert stubs otherwise (classic ESP32 builds // it too; the driver never calls in thanks to platform::lcdLanes == 0). Gate on // SOC_LCDCAM_I80_LCD_SUPPORTED, NOT SOC_LCD_I80_SUPPORTED: the classic ESP32 sets -// the latter for its unrelated I2S-LCD peripheral, which wired this driver onto a -// chip with no LCD_CAM and hung its boot (see platform_config.h + lessons.md). +// the latter for its unrelated I2S-LCD peripheral, so keying on it wires this driver +// onto a chip with no LCD_CAM and hangs its boot (see platform_config.h + lessons.md). #include "platform/platform.h"