Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
314 lines (259 loc) · 12.4 KB

ARCHITECTURE.md

File metadata and controls

314 lines (259 loc) · 12.4 KB
Raw

SCIoT architecture and data flow

This document describes the current Python client and edge-server architecture. It is intentionally descriptive rather than aspirational: experimental areas and known ownership issues are called out explicitly.

System boundary and deployment topologies

SCIoT splits TinyML inference between an IoT/client device and a nearby edge server. The Python repository currently contains:

  • a static-image HTTP client in src/client/python/http_client.py;
  • camera/benchmark client variants under src/client/python/;
  • an edge server entry point in src/server/edge/run_edge.py;
  • HTTP, WebSocket, and MQTT server-side transports under src/server/communication/;
  • model loading, suffix inference, split selection, telemetry, and validation support under src/server/.

Supported topologies:

  1. Single static-image HTTP client talking to one edge server.
  2. Multiple devices talking to one edge server over HTTP.
  3. Experimental WebSocket and MQTT transports sharing the same request handler.
  4. Local-only client fallback when the server cannot be reached or registration is rejected.

HTTP is the primary supported transport. WebSocket and MQTT are repaired and covered by focused tests, but should still be treated as experimental for production deployments until they are validated with the target device firmware and broker configuration.

Component boundaries

flowchart LR
    subgraph Client["Client process"]
        CfgC["Client YAML + CLI/env overrides"]
        Client["Static HTTP client"]
        TFLiteC["Client TFLite interpreter cache"]
        Lifecycle["Client lifecycle and fallback"]
    end

    subgraph Transport["Edge transport layer"]
        Coordinator["Transport coordinator"]
        HTTP["HTTP FastAPI/Uvicorn"]
        WS["WebSocket FastAPI/Uvicorn"]
        MQTT["MQTT Paho client"]
    end

    subgraph Core["Edge core"]
        Handler["RequestHandler"]
        State["Device profiles and model map"]
        Edge["Edge suffix inference"]
        Manager["ModelManager interpreter cache"]
        Algo["OffloadingAlgo"]
        Variance["VarianceDetector"]
    end

    subgraph Files["Files and telemetry"]
        Config["Server YAML + CLI/env overrides"]
        Models["Keras/TFLite model artifacts"]
        Data["Layer sizes and valid split points"]
        Logs["JSON/CSV/profiler outputs"]
    end

    CfgC --> Client
    Client --> TFLiteC
    Client --> Lifecycle
    Client -- "registration, input, inference result, split query" --> HTTP
    Coordinator --> HTTP
    Coordinator --> WS
    Coordinator --> MQTT
    Config --> Coordinator
    HTTP --> Handler
    WS --> Handler
    MQTT --> Handler
    Handler --> State
    Handler --> Edge
    Handler --> Algo
    Handler --> Variance
    Edge --> Manager
    Manager --> Models
    Algo --> Data
    Handler --> Logs
Loading

Responsibilities

Component Owns Does not own
Static HTTP client registration, local TFLite prefix inference, binary payload creation, server reconnect/local-only fallback server model selection, edge suffix inference
Transport coordinator starting/stopping enabled transports, signal propagation, per-transport failure state request semantics
HTTP/WebSocket/MQTT transports protocol-specific routing, request/response serialization, structured transport errors split algorithm state
RequestHandler registration, model hash matching, per-device timing state, inference-result handling, next split decision socket lifecycle
Edge / ModelManager edge-side suffix inference and interpreter caching transport routing
OffloadingAlgo cost evaluation and best split selection from timings, layer sizes, and network speed measurement collection
Telemetry/profilers JSON/CSV/profiler output and dashboard inputs correctness-critical control flow

Startup and process topology

The server starts one Python process. run_edge.py validates configuration, initializes configured model metadata, builds the model-hash registry, creates a shared RequestHandler, and hands enabled transports to the transport coordinator.

Each enabled transport runs in a managed daemon thread. HTTP and WebSocket each own a Uvicorn server. MQTT owns a Paho client loop. SIGINT/SIGTERM is handled by the coordinator, which asks every transport to stop and joins their threads.

Configuration details are documented in docs/CONFIGURATION.md; transport protocol details are documented in docs/TRANSPORTS.md.

Model artifact discovery and model-hash matching

Server models are configured under model in src/server/settings.yaml. Each model entry includes:

  • model_dir;
  • input_height;
  • input_width;
  • last_offloading_layer.

RequestHandler.build_model_registry() hashes the server-side Keras model file for every configured model. ModelFiles.get_model_h5_path(model_dir) first checks the nested layout:

src/server/models/test/<model_dir>/<model_dir>.h5

and falls back to the older flat layout:

src/server/models/test/<model_dir>.h5

The static client computes an MD5 hash from its configured TFLite submodels and sends that hash during registration. Registration succeeds only if the hash is present in the server registry. On success, the server stores the device-to-model mapping in RequestHandler.device_model_map and initializes a per-device profile in RequestHandler.device_profiles.

Runtime state ownership and lifetime

State/cache Owner Lifetime Notes
Client TFLite interpreters static client _interpreter_cache client process keyed by layer index; loaded lazily/preloaded at startup
Device profiles RequestHandler.device_profiles server process class-level dictionary keyed by device_id
Device-to-model map RequestHandler.device_model_map server process class-level dictionary keyed by device_id
Model hash registry RequestHandler.model_registry server process built during server startup
Layer-size cache RequestHandler._layer_sizes_cache server process keyed by model directory
Edge TFLite interpreters ModelManager._interpreter_cache server process / manager instance keyed by model key and layer; per-interpreter lock guards invocation
Valid offloading points OffloadingAlgo._valid_points_cache server process keyed by model directory
Current HTTP split decision HttpServer.best_offloading_layer_map HTTP transport lifetime keyed by device_id
WebSocket/MQTT best layer transport instance transport lifetime currently less complete than HTTP for multi-device state
Background writers request-handler module queue/thread server process writes images, debug data, telemetry CSV rows, and profiler JSON

Important caveat: several server state containers are class-level dictionaries. They are shared across enabled transports and request-handler instances. This is simple and fast, but it makes explicit thread-safety and lifecycle cleanup important future work.

Registration sequence

sequenceDiagram
    participant Client
    participant Transport as HTTP/WebSocket/MQTT
    participant Handler as RequestHandler
    participant Registry as model_registry
    participant State as device_profiles

    Client->>Transport: registration(device_id, model_hash)
    Transport->>Handler: handle_registration()
    Handler->>Registry: lookup model_hash
    alt known model hash
        Handler->>State: create/update device profile
        Handler-->>Transport: accepted device_id
        Transport-->>Client: registration success
    else unknown model hash
        Handler-->>Transport: None
        Transport-->>Client: rejected/not available
    end
Loading

Split inference sequence

sequenceDiagram
    participant Client
    participant HTTP as HTTP transport
    participant Handler as RequestHandler
    participant Edge
    participant Manager as ModelManager
    participant Algo as OffloadingAlgo

    Client->>HTTP: GET offloading_layer?device_id=...
    HTTP-->>Client: current split or safe local default
    Client->>Client: run local prefix layers
    Client->>HTTP: POST binary inference result
    HTTP->>Handler: decode and process in worker thread
    Handler->>Handler: update device timing profile
    alt split before final layer
        Handler->>Edge: run suffix inference
        Edge->>Manager: invoke cached TFLite layer interpreters
        Manager-->>Edge: prediction + edge layer timings
        Edge-->>Handler: prediction + edge layer timings
    else local-only/full result
        Handler->>Handler: use client output as prediction
    end
    Handler->>Algo: choose next split from timings, layer sizes, speed
    Algo-->>Handler: next offloading_layer_index
    Handler-->>HTTP: next split, device_id, prediction
    HTTP-->>Client: prediction response
Loading

Local-only fallback and reconnect

sequenceDiagram
    participant Client
    participant Server

    Client->>Server: registration
    alt server unavailable or model rejected
        Server--xClient: connection error / 404
        Client->>Client: mark server unavailable
        Client->>Client: run all layers locally
    else registered
        Server-->>Client: success
        Client->>Server: normal split workflow
    end
    loop reconnect interval
        Client->>Server: registration retry
        alt registration succeeds
            Server-->>Client: success
            Client->>Client: resume offloading
        else still unavailable
            Client->>Client: keep local-only mode
        end
    end
Loading

Split selection inputs and output

The split algorithm consumes:

  • network speed estimated from payload size and latency;
  • smoothed per-layer device inference times from client payloads;
  • smoothed per-layer edge inference times from suffix inference;
  • per-layer output sizes from data/models/layer_sizes_<model_dir>.json;
  • optional valid split points from data/models/valid_offloading_points_<model_dir>.json.

OffloadingAlgo.static_offloading() evaluates:

  • mixed device/edge execution;
  • device-only execution;
  • edge-only cost, currently not considered as the returned split in the same way as mixed/device-only paths.

The output is the next offloading_layer_index. -1 is used elsewhere as a forced local-only signal. The HTTP transport stores the next split per device and returns it on the next offloading_layer request.

Telemetry and background output

The server collects:

  • decoded inference payload metadata;
  • device and edge timing profiles;
  • average payload speed and latency;
  • prediction path timing phases through AdvancedProfiler;
  • periodic macro JSON and optional cProfile output;
  • evaluation, decision, and simulation CSV rows.

Background file writes are queued through the request-handler module-level I/O queue to keep hot request paths from blocking on disk.

Dashboard/statistics consumers read files under data/models/ and server output paths. This telemetry is useful for inspection, but several schemas are still implicit and should be consolidated before being treated as stable public data.

Known experimental areas and architectural decisions

  • HTTP is the primary supported transport. WebSocket and MQTT share repaired transport helpers but remain experimental for production.
  • The binary inference-result protocol is bounded and version-aware. See docs/TRANSPORTS.md.
  • Configuration is validated at runtime boundaries. See docs/CONFIGURATION.md.
  • TensorFlow/Keras loading policy is documented in docs/TENSORFLOW_KERAS_POLICY.md.
  • Per-device state is shared in class-level dictionaries. This makes concurrent transport support possible but leaves explicit locking/state isolation as future work.
  • /api/split_inference still exists as a separate HTTP path with model-specific logic. It should eventually delegate to the same suffix-inference service used by the primary inference-result endpoint.
  • Model artifact layout supports both nested and older flat H5 locations. TFLite submodels are discovered through configured model directories and layer IDs.

Documentation update checklist

When future stories change architecture, update this document if they modify:

  • runtime process/thread topology;
  • supported transports or protocol fields;
  • registration or model-hash matching;
  • device state ownership or cache lifetime;
  • model artifact layout or discovery;
  • offloading algorithm inputs/outputs;
  • telemetry file schemas or dashboard consumers;
  • fallback/reconnect behavior.