diff --git a/.claude/skills/multi-server-devnet/SKILL.md b/.claude/skills/multi-server-devnet/SKILL.md new file mode 100644 index 00000000..4246dcbb --- /dev/null +++ b/.claude/skills/multi-server-devnet/SKILL.md @@ -0,0 +1,519 @@ +--- +name: multi-server-devnet +description: This skill should be used for managing long-lived lean-consensus devnets that run as detached docker containers on remote hosts — the usual setup is one INDEPENDENT single-host devnet per server, federated into one Grafana, but it also covers a single chain whose node indices are SPLIT across hosts. Works for any server names and any number of servers/nodes. Trigger when the user asks to "start/reset the devnet(s)", "start a devnet on ", "reset the cross-server devnet", "restart the devnet on both servers", "restart the devnet", "rolling restart", "recover stalled nodes", "the devnet stopped finalizing", "investigate the finality stall", "convert nodes to zeam/ream/qlean/grandine/gean/lantern", "multi-client devnet", "add a client to the devnet", "canary a new client/build", "scale the devnet / add a node", "set up grafana/prometheus for the devnets", "pull the latest image on the servers", or "add swap / memory limits to the servers". Distinct from the local single-host `devnet-runner` skill. +version: 0.3.0 +--- + +# Multi-Server Devnet + +Operate lean-consensus devnets that run as detached containers +(`docker run -d --restart unless-stopped --network host`) on remote hosts, +surviving SSH disconnects and reboots. This is NOT the local `devnet-runner` +(spin-node.sh) workflow — these are hand-managed detached containers. + +## Topology model: one independent devnet per server + +The current setup runs **one self-contained devnet per server** — each its own +chain (own genesis, own GENESIS_TIME, own fork-choice, own finality), with no +cross-server peering. They are kept apart by pointing **every ENR at +`127.0.0.1`**, so a node only ever discovers peers on its own host even though +all devnets share the dummy fork digest. One central Grafana/Prometheus +federates all of them, labelled per devnet. + +**Variant: one chain spanning several hosts.** A deployment may instead run a +single chain whose node indices are split across hosts (host A owns `0..k`, host +B owns `k+1..N-1`). Everything below still applies with three changes, and +`start-range.sh` / `check-range.sh` / `teardown.sh` exist for exactly this shape: + +- **ENRs carry each host's real (e.g. tailnet) IP**, not `127.0.0.1`, or the + halves never discover each other. Put the per-host ip in the source + `validator-config.yaml`'s `enrFields.ip`; `make-genesis.sh` pins `127.0.0.1` + and so is NOT usable as-is — generate from a hand-built validator-config + instead (keep it: it also carries the node privkeys, so reusing it verbatim + preserves every node's identity across a reset). +- **Aggregators are no longer "node k → subnet k".** Each host owns a slice of + the subnets (host B's node `36` may aggregate subnet `4`), so the mapping is + passed explicitly rather than derived from the index. +- **`ceil(2/3·NODES)` spans both hosts**, so losing one host halts the whole + chain rather than half of it, and a restart on either half checkpoint-syncs + from the other. + +Nothing about the servers is hardcoded. Establish these from the operator: + +- `SERVERS` — the SSH targets (any count, any names), one devnet each. +- `SSH_USER` — login user. `docker` is invoked with `sudo`. +- Per-devnet `NODES` (validators on that server) and `SUBNETS` + (`ATTESTATION_COMMITTEE_COUNT`); these can differ between servers. +- A central host for Grafana + the federating Prometheus (often one of the + servers); each server's per-host Prometheus remote-writes to it. + +Put those values in `scripts/devnet.env` (copy `scripts/devnet.env.example`; +gitignored) instead of retyping them: the operator-side scripts source it via +`scripts/devnet-env.sh`, and an env var exported in the shell still wins over the +file. It is the one place a deployment's hosts, urls, and Grafana ids live. +Per-devnet `NODES`/`SUBNETS` are recorded there too, but as the operator's +inventory — the scripts take them as positional args, and the authority on a +running devnet is always its own `genesis/config.yaml` +(`ATTESTATION_COMMITTEE_COUNT`), which `start-devnet.sh` and `convert.sh` check +against. + +Per devnet, node `n` (0 ≤ n < NODES) on its host: + +- ports: gossip `9000+n`, api `5052+n`, metrics `9200+n` (n is the only index — + no cross-host BASE offset; each devnet starts numbering at 0). +- **Three-layer naming:** + - **identity** `node_n` — node-id, `node_n.key`, data-dir, genesis validator + name; client-agnostic, all genesis artifacts key off it. + - **container/display** `_n` (e.g. `zeam_8`, `ethlambda_0`) — the docker + `--name`, and hence cAdvisor's `name` label and promtail's `node` label, so + Grafana shows the client at a glance. `client_type` is derived from this prefix. + - **index** `n` — the ports above. +- attests on subnet `n % SUBNETS`. +- genesis dir `/opt/lean-quickstart/genesis` → `/config`, data + `/opt/lean-quickstart/data/node_n` → `/data`. +- management scripts address a node by index: they find its container via the + `_n$` name suffix (client prefix may differ), so conversions never rename data + or node identity. + +**Aggregators:** one per subnet — nodes `0 .. SUBNETS-1`, where node `k` +aggregates subnet `k` (`--aggregate-subnet-ids k`). leanVM aggregation is +CPU-heavy, so keep one aggregator per subnet, no more. + +**Finality threshold is per devnet:** `ceil(2/3 · NODES)` aligned votes on that +server. Every restart/conversion decision hinges on staying above it for that +devnet only — one server's chain stalling never affects another's. + +## Golden rules (each learned from an outage) + +1. **Always pass a checkpoint-sync URL on restart:** + `--checkpoint-sync-url http://127.0.0.1:/lean/v0/states/finalized`. + Current images then prefer resuming from the on-disk DB when it's fresh + (head within `MAX_RESUMABLE_DB_STATE_AGE` slots of wall clock, + `bin/ethlambda/src/main.rs`) and fall back to checkpoint sync when stale — + but only if a URL is provided; without one the node re-inits from genesis. + On older images keep-DB resume panics the consensus actor (`"safe target + exists"`: process stays up, consensus stops), so wipe-data + checkpoint sync + remains the safe default and is what `cs-restart.sh` does. +2. **Wait 60s between stopping and starting a running node.** Skipping the + gossipsub backoff leaves the node out of the attestation meshes — it attests + but aggregators never hear it, so the vote count stays below threshold and + finality stalls. Restart one node at a time per devnet; servers in parallel. +3. **One GENESIS_TIME per devnet, kept stable across that devnet's restarts.** + Each server's chain has its OWN GENESIS_TIME (they deliberately differ). + Within a devnet, a drifted value forks a node onto a private clock/chain and + breaks checkpoint sync (`genesis time mismatch`). Regenerating genesis = a new + chain; restarting nodes must reuse the on-disk value. +4. **Keep the memory guards.** A swapfile per host + per-container memory limits. + Default every node to `--memory 8g --memory-swap 16g --memory-reservation 2g` + (raise further for a client that legitimately needs it). A client leak (qlean + leaks ~1.8 GiB/h) must OOM one container, not starve the whole host. With + several devnets-per-fleet this matters more, not less — but here each host runs + only its own devnet. +5. **A node must come back with the role it had.** A restart that drops + `--is-aggregator` leaves its subnet with no aggregator: attestations still + verify and the logs look healthy, but nobody stores the gossip signatures, so + blocks are built with `attestation_count=0` and finality dies quietly. + `cs-restart.sh` preserves the role; `convert.sh` warns when a conversion would + remove one. After any restart, confirm one `AGG=yes` per subnet with + `host-check.sh` (or `lean_is_aggregator`). Same for `--log-opt`: every + `docker run` must cap the json log, or one verbose canary fills the disk. + +## Editing genesis / mass restarts needs care + +Regenerating genesis or mass `rm -rf` of data dirs is destructive and may be +permission-gated — state the exact scope and let the user confirm. Never invent +a `GENESIS_TIME`; derive it (`now + offset`) at generation or read the live +chain's value. Save crash logs before `docker rm` when debugging a crash +(`sudo sh -c "docker logs X > /opt/lean-quickstart/crash-logs/X.log"` — the +redirect must run under sudo). + +## Workflows + +Examples assume `SSH_USER` is set and you iterate over `SERVERS`. Per server you +pass its own `NODES`/`SUBNETS`. + +### Pull the latest images on all servers +```bash +for h in $SERVERS; do ssh "$SSH_USER@$h" 'sudo docker pull ghcr.io/lambdaclass/ethlambda:devnet5'; done +``` +`devnet5` is the current devnet tag; it tracks the chain's leanVM/proof format, +so it (and the other clients' tags in `references/clients.md`) move together on a +bump. If a fresh build is pending, watch it: `gh run watch --repo --exit-status`. + +### Start a fresh devnet on a server (genesis is per-devnet) +Genesis is generated **per devnet on the operator machine** (each needs a +distinct genesis root + GENESIS_TIME), then shipped to the host. Then the host +launches its nodes. + +**Order matters: GENESIS_TIME starts ticking mid-generation.** `generate-genesis.sh` +stamps `GENESIS_TIME = now + offset` *early* (before its per-validator loop), so every +step after that burns the countdown. Do the slow, chain-age-dependent teardown FIRST, +then generate, and the offset only has to cover ship + launch + mesh. + +```bash +# 0. host-side FIRST (pre-stamp: this is the step that scales with chain age -- +# wiping 54 GB of RocksDB is not something to do against a running clock). +ssh "$SSH_USER@$h" 'N=$(sudo docker ps -aq --filter "name=_[0-9]*$") + for c in $N; do sudo docker update --restart=no $c; done # else it respawns + [ -n "$N" ] && sudo docker rm -f $N + sudo rm -rf /opt/lean-quickstart/data/node_*' + +# 1. operator-side: build this devnet's genesis (127.0.0.1 ENRs, subnet=i%SUBNETS, +# aggregators 0..SUBNETS-1). KEYS_DIR = canonical hash-sig keys; SRC_VCONFIG = a +# validator-config holding privkeys for nodes 0..NODES-1. +bash scripts/make-genesis.sh OUTDIR NODES SUBNETS KEYS_DIR SRC_VCONFIG [GT_OFFSET] +# More than one validator per node: set VALIDATORS_PER_NODE=V (KEYS_DIR must then +# hold NODES*V key pairs). make-genesis.sh runs subnet-align-validators.py after +# generate-genesis.sh so each node's validators all sit in ONE subnet -- see that +# script for why the default contiguous assignment spreads them over every subnet. +# Needs NODES % SUBNETS == 0. Aggregators stay one-per-subnet (node k -> subnet k). +# GT_OFFSET defaults to 240s. Measured 2026-07-28, 32 nodes, one host: generation +# after the stamp 13s (128 val) / 28s (256 val), ship+launch ~105s, mesh ~30s +# => ~165s. Do NOT pad this to 600-900s "to be safe": that is 7-12 min of dead +# wall clock per restart and it bought nothing. Raise it only for many hosts or +# a slow link. + +# 2. ship genesis dir to the host's /opt/lean-quickstart/genesis (tar over ssh): +(cd OUTDIR && tar cf - config.yaml genesis.json genesis.ssz nodes.yaml validators.yaml \ + annotated_validators.yaml validator-config.yaml *.key) | \ + ssh "$SSH_USER@$h" 'sudo find /opt/lean-quickstart/genesis -maxdepth 1 \( -name "node_*.key" -o -name "ethlambda_*.key" \) -delete; sudo tar xf - -C /opt/lean-quickstart/genesis' +# hash-sig keys: sync the per-validator *.ssz once (servers usually hold only a +# subset). Spot-check with md5sum; Tailscale policy often blocks server↔server +# ssh, so push Mac→host. + +# 3. host-side: launch all nodes (+ canaries). Args: NODES SUBNETS [N:client ...] +scp scripts/start-devnet.sh scripts/host-check.sh "$SSH_USER@$h:/tmp/" +ssh "$SSH_USER@$h" "bash /tmp/start-devnet.sh NODES SUBNETS 28:zeam 29:ream 30:qlean 31:lantern" +``` +`start-devnet.sh` refuses to launch when its `SUBNETS` disagrees with the genesis +on disk, when node data dirs are non-empty (step 0 skipped ⇒ nodes resume an old +DB against a new genesis and fork off; `ALLOW_STALE_DATA=1` overrides), or when a +canary spec names an unknown client — all three otherwise cost the whole devnet. + +Verify ~2 min after genesis, host-side first, then across devnets: +```bash +ssh "$SSH_USER@$h" "bash /tmp/host-check.sh NODES" # every node up + following (exit≠0 = attention) +bash scripts/sweep.sh # head/justified/finalized per devnet +``` +A young devnet can sit at `finalized=0` while justified jumps +(square/pronic-distance slots) — that's the bootstrap regime, not a stall; leave +it alone (see `references/operations.md`). + +### Reset a chain that SPANS hosts (one genesis, node indices split) +Same order as above, but each host owns a slice and the aggregator→subnet map is +explicit. Reuse the existing source `validator-config.yaml` verbatim (real +per-host ENR ips + node privkeys) and re-stamp GENESIS_TIME with lean-quickstart's +`generate-genesis.sh --mode ansible --offset N` — keygen is +skipped when the keys are already there, so only the small artifacts change. + +```bash +# 0. stage the new genesis alongside the LIVE one (non-destructive: nothing stops yet). +# Do the big key sync here, while the old chain still runs. +(cd BUILD_DIR && COPYFILE_DISABLE=1 tar cf - .) | \ + ssh "$SSH_USER@$h" 'sudo rm -rf /opt/lean-quickstart/genesis-new + sudo mkdir -p /opt/lean-quickstart/genesis-new + sudo tar xf - -C /opt/lean-quickstart/genesis-new' + +# 1. PRE-FLIGHT the image against that staged genesis BEFORE anything is wiped -- +# a pubkey-size mismatch (52B leanSig vs 32B leanVM-main) fails at genesis parse, +# upstream of every sync path, and no --checkpoint-sync-url can save it. +# --network none + a throwaway data dir cannot touch the live devnet. +# Use `docker run -d --name X` + explicit `docker rm -f`: `timeout` kills the +# docker CLIENT, not the container, so --rm never fires and it leaks. +ssh "$SSH_USER@$h" 'sudo docker run -d --name gval --network none \ + -v /opt/lean-quickstart/genesis-new:/config:ro -v /tmp/gval-data:/data IMAGE \ + --genesis /config/config.yaml ... --node-id node_0' # want: "Loaded genesis + # configuration validator_count=N" -> "Loaded validator key pairs" -> "Initialized store" + +# 2. teardown per host (containers, archive genesis, wipe data) -- run BEFORE the +# GENESIS_TIME stamp. Archives genesis to genesis.bak- so the retired +# chain's hash-sig keys survive, and promotes genesis-new into place. +ssh "$SSH_USER@$h" 'bash /tmp/teardown.sh' + +# 3. re-stamp GENESIS_TIME, reship ONLY the regenerated small files, launch each slice +ssh "$SSH_USER@$hA" "ETH_IMG=IMAGE bash /tmp/start-range.sh 0 31 8 0:0 1:1 2:2 3:3" +ssh "$SSH_USER@$hB" "ETH_IMG=IMAGE bash /tmp/start-range.sh 32 63 8 36:4 37:5 38:6 39:7" +ssh "$SSH_USER@$h" 'bash /tmp/check-range.sh START END' # range-aware host-check +``` +Measured on a 64-node/2-host chain: teardown incl. a 184 GB RocksDB wipe **7s per +host** (`rm -rf` of RocksDB is far cheaper than the genesis-countdown budget +assumes), reship + 32 `docker run` ~25s per host, mesh at full peer count *before* +genesis — so `--offset 300` left minutes of slack. Starting host A before host B +fills A's log with WARN `Failed to negotiate transport protocol(s)` for B's +not-yet-up nodes; benign dial races, peers converge. Grep `ERROR` by level, since +`-i error` also matches those warnings. + +### Set up / refresh observability +Per host: a Prometheus that scrapes this devnet's nodes with a `network=` +label + `client_type`, remote-writing to the central Prometheus. One central +Prometheus + Grafana federates all devnets; the dashboard filters on `network`. +The host's **node_exporter** is a systemd service (NOT a container) on +`:9122`, installed from `github.com/lambdaclass/monitoring-stack` +(`make inventory TARGET=...` + `make node_exporter`); the per-host prometheus +scrapes it at `:9122`. Install ONLY node_exporter from that repo — its +grafana/prometheus/loki/etc. would collide with this docker stack. cadvisor stays +a container (`172.17.0.1:9098`); its per-container metrics are consumed by the +**Devnet Resources** dashboard (`scripts/resources-dashboard.json`, per-node CPU + +memory). Details + the Node Exporter Full dashboard in `references/operations.md`. +```bash +# operator-side: emit this devnet's prometheus.yml (network label + per-node client_type) +bash scripts/prometheus-config.sh NETWORK NODES HOST_IP CENTRAL_WRITE_URL [N:client ...] > /tmp/prometheus.yml +cat /tmp/prometheus.yml | ssh "$SSH_USER@$h" 'sudo tee /opt/lean-quickstart/observability/prometheus.yml >/dev/null && sudo docker restart prometheus' +# host-side: create the scrapers on a NEW host (existing ones are left alone +# unless RECREATE=1 — a config change only needs the restart above) +scp scripts/start-observability.sh "$SSH_USER@$h:/tmp/"; ssh "$SSH_USER@$h" 'bash /tmp/start-observability.sh' +``` +Relabel gotcha: that file is a single-file bind-mount — edit by full overwrite +(`tee`) + `docker restart prometheus`, never `sed -i`+HUP (inode trap, see +`references/operations.md`). The central Prometheus needs +`--web.enable-remote-write-receiver`; Grafana groups by **job name**, so a +converted node must be relabelled (both `job_name:` and `client_type:`). +`scripts/devnet-overview-dashboard.json` is a ready Grafana dashboard (head / justified +/ finalized per devnet, one series per `network`) — copy it to the central +Grafana's **dashboards dir** and it auto-loads. `scripts/client-dashboard.json` is +the main per-node dashboard for the same dir (see the inventory below). Every +dashboard picks its datasource through a **template variable** (`ds_prom`/`ds_loki`; +`datasource` on the client dashboard) rather than a pinned uid, so they drop into +any Grafana unedited — no deployment-specific datasource ids in the JSON. + +**That dir is NOT `/dashboards`** — the provisioning tree holds +only the provider yaml (`dashboards.yml`, which declares +`options.path: /var/lib/grafana/dashboards`); the JSONs live in whatever host dir +is bind-mounted there, a sibling of the provisioning tree in this deployment +(`GRAFANA_DASHBOARDS_DIR` in `devnet.env`). Confirm before copying, since dropping +a dashboard in the provisioning dir loads nothing and looks like a silent no-op: +```bash +ssh "$METRICS_HOST" 'sudo docker inspect '"$GRAFANA_CONTAINER"' \ + --format "{{range .Mounts}}{{.Source}} -> {{.Destination}} (ro={{not .RW}}){{println}}{{end}}"' +# ship it (back up first; the provisioner sweeps every ~30s) +ssh "$METRICS_HOST" "cp -p $GRAFANA_DASHBOARDS_DIR/client-dashboard.json{,.bak-\$(date +%Y%m%d-%H%M%S)}" +cat scripts/client-dashboard.json | ssh "$METRICS_HOST" "cat > $GRAFANA_DASHBOARDS_DIR/client-dashboard.json" +``` +Verify it landed with +`curl -s $GRAFANA_BASE_URL/api/dashboards/uid/ | jq '.dashboard.version'` (the +version bumps on each sweep that reads a changed file). + +**Ship a dashboard under the same FILENAME the skill uses, and keep exactly one +`.json` per uid in that dir.** Grafana keys a provisioned dashboard on its `uid`, +not its filename, so a copy left behind under an old name is invisible in the UI +yet still claims the uid: adding a second file next to it makes two provider files +fight over one uid, which Grafana resolves by logging an error and dropping one. +The failure looks like "my update did not land". This bit us once already: the +finality dashboard was `scripts/finality-dashboard.json` here but had been +deployed as `devnet-overview-dashboard.json`, so copying it under the skill's name +would have collided rather than updated it. The two names are now reconciled on +`devnet-overview-dashboard.json` (the deployed one, since the server is what +operators actually look at). To retire a stale copy, rename it to +`.json.bak-` in the same step that writes the new file — the provider +only reads `*.json`, which is also why the `.bak-*` backups already in that dir +are inert. Audit with `md5sum` of every server `.json` against `scripts/*.json`; +the names AND the digests should match, with `node-exporter-full.json` the one +expected exception (see the note under the inventory: it is owned upstream). + +Filenames are for humans; only the `uid` inside the JSON is load-bearing. Do not +"fix" a filename to look more like its uid — `devnet-overview-dashboard.json` +carries uid `devnet-finality-overview` and that mismatch is harmless, whereas +renaming a live file costs a provisioner re-add (the dashboard's `version` resets +to 1, same uid, so links and alerts survive). + +Gotcha: that dir is +bind-mounted **read-only** into the container, so Grafana UI edits are reverted on +the next provisioner sweep (~30s) despite `allowUiUpdates: true`. Edit the JSON and +re-copy; treat the repo copy as the source of truth and keep it in sync, since +nothing pulls server-side edits back. + +**Logs (Loki + promtail).** Mirrors the metrics path: a per-host **promtail** +ships node-container stdout/stderr to the **central Loki** (7d retention; +provisioned Grafana datasource). It discovers containers through the **Docker +daemon API**, so it is immune to json-file rotation and safe to deploy on a live +devnet — it only mounts the socket. Indexed labels match prometheus (`network`, +`node` = `_N`, `client_type`, `instance`, `stream`, `level`); +high-cardinality fields (`slot`, `validator`, `block_root`, …) ride as Loki +**structured metadata** instead, so they stay filterable (`| slot="123"`) without +exploding stream cardinality. Query in Grafana Explore +(`{network=""} | level="ERROR"`) or use the provisioned **Devnet Logs** +dashboard (`scripts/logs-dashboard.json`). +```bash +# operator-side: emit this host's promtail.yml (network label + push to central Loki) +bash scripts/promtail-config.sh NETWORK HOST_IP LOKI_PUSH_URL [N:client ...] \ + | ssh "$SSH_USER@$h" 'sudo tee /opt/lean-quickstart/observability/promtail.yml >/dev/null' +# host-side: launch (or restart after a config change) the shipper +scp scripts/start-promtail.sh "$SSH_USER@$h:/tmp/"; ssh "$SSH_USER@$h" 'bash /tmp/start-promtail.sh' +``` +`LOKI_PUSH_URL` = `http://:3100/loki/api/v1/push`. Keep the canary +`N:client` specs in sync with `prometheus-config.sh` so logs and metrics agree on +`client_type`. Same single-file bind-mount inode trap as prometheus.yml: overwrite +with `tee` then `sudo docker restart promtail`. + +**One trap on first launch:** promtail replays each container's whole retained log +history (docker_sd has no "since"), which Loki rejects as too-old and which burns +2+ cores per host while it reads. `promtail-config.sh` bakes in the `drop +older_than: 1h` guard, but on a host with large logs you also want to seed +`positions.yaml` to "now" once — the pipeline stages, the exact seed command, and +why the stage order matters are in `references/operations.md`. + +**Finality alert (Slack).** `scripts/grafana-finality-alert.yaml.template` provisions +a per-devnet "lost finality" rule into the central Grafana: fires when +`max(head) − max(finalized) > 512` for a `network` (network maxima, so one stuck +canary can't false-fire), 5m `for`, 15m anti-flap, at most one notify per 12h. +Deploy it with `scripts/deploy-finality-alert.sh`, which takes every +deployment-specific value from `devnet.env` (`METRICS_HOST`, `GRAFANA_PROV_DIR`, +`GRAFANA_CONTAINER`, `GRAFANA_BASE_URL`, `PROM_DS_UID`) — nothing host-specific +lives in the template: +```bash +bash scripts/deploy-finality-alert.sh # values from devnet.env +DRY_RUN=1 bash scripts/deploy-finality-alert.sh # render + validate, ship nothing +PROM_DS_UID=other bash scripts/deploy-finality-alert.sh # one-off override +``` +It refuses to ship if any `__PLACEHOLDER__` survived rendering (Grafana would +silently ignore a malformed provisioning file). +`WEBHOOK_FILE` (default `./webhook.txt`) holds the Slack webhook — a **secret**; the +repo `.gitignore` covers `webhook.txt` and the rendered alert file, and the script +renders to a `mktemp` it deletes on exit. Unlike the dashboards, a provisioned alert +rule needs a concrete datasource uid, which is why `PROM_DS_UID` is explicit. + +### Recover crashed / frozen / split nodes (no chain reset) +Use `scripts/cs-restart.sh` on the affected server — checkpoint-sync restart with +the 60s backoff, per-container crash-log preservation, and **aggregator-role +preservation** built in: +```bash +ssh "$SSH_USER@$h" "bash /tmp/cs-restart.sh ..." +ssh "$SSH_USER@$h" "bash /tmp/host-check.sh " # verify after +``` +`CS_PORT` = a healthy same-devnet node's api (an aggregator stays up — use it); +comma-separate a second one and ethlambda falls over to it if the first fails. +It reads the topology flags (`--is-aggregator`, `--aggregate-subnet-ids`, +`--attestation-committee-count`) off the container it replaces, so restarting an +aggregator no longer demotes it — the failure that looks like a healthy devnet +which silently stops finalizing. If the container is already gone there is nothing +to read: pass `SUBNETS=` so the rule (nodes `0..SUBNETS-1`) can be re-derived. +Restart aggregators last, sourcing checkpoint from a healthy non-aggregator. +`AGG=` overrides the preserved role when you want to *change* it, and `IMAGE=` +canaries a build on the node (aggregators included): + +| | | +|---|---| +| `AGG` unset | preserve the role the container had (default) | +| `AGG=auto SUBNETS=n` | apply the topology rule — repairs a node that lost its role | +| `AGG=` | promote this node to aggregate subnet `` | +| `AGG=off` | demote deliberately | + +For many nodes, launch detached and poll `/tmp/cs-restart.log` (detached-SSH and +pkill-bracket gotchas in `references/operations.md`). + +### "Is this node working?" — the all-is-good checklist +A tool to answer, at any time, whether a running node is doing its job — a spot +check on a node, a sanity pass after a config change or conversion, or one input +to a stall investigation. `references/node-health.md` is a per-node checklist +(follows the chain, emits valid+useful attestations, builds non-empty blocks, +aggregates+publishes if an aggregator, on-time duties, peers+mesh, votes landing +in blocks) with a concrete Prometheus query and log grep for each item, plus a +separate devnet-level "finality advances" section that is explicitly *not* a +per-node fault (so a healthy node isn't blamed for a chain-wide stall). For duty +*timing* detail use the `devnet-profiling` skill. + +### "A devnet stopped finalizing" — investigate +Do NOT immediately restart. Diagnose that devnet in order (full method in +`references/operations.md`): is it the young-devnet bootstrap regime (justified +still advancing, finalized=0)? → `host-check.sh` for the per-node picture (who is +down, restarting, or lagging past the sync gate) → vote count vs +`ceil(2/3·NODES)` → which validator cohort is missing on the aggregator → are +missing nodes silent or voting STALE targets → host memory/CPU via the central +Prometheus (reachable even when that server's SSH is down) → confirm no block +rejections (rules out a canary interop fork). Common causes: a host OOM freezing +fork choice; nodes restarted without the 60s backoff; **an aggregator that came +back without its role** (check `lean_is_aggregator`/the `AGG` column — one per +subnet must be `yes`); a frozen canary thinning votes; aggregator backlog. Fix the +cause, then `cs-restart.sh` the affected nodes; deep in the spiral, a fresh genesis +is faster. + +### Convert nodes to other clients (zeam/ream/qlean/grandine/gean/lantern) +Node identity stays `node_n`; the container is renamed to `_n` to reflect +the new client (so Grafana/logs show it). Use `scripts/convert.sh` (per-client CLI +shapes + memory limits + 60s backoff; committee count read from the genesis on +disk, `ACC=` only to override): +```bash +ssh "$SSH_USER@$h" "bash /tmp/convert.sh ..." +``` +It validates every spec before removing anything (a typo used to be discovered +after the node's container and data were already gone) and warns when the node it +is replacing was an aggregator and the new spec has no `:agg` — that silently +leaves a subnet with no aggregator. +**Canary one node per client type first** and confirm the vote count holds before +scaling — a broken client converted en masse can drop the ethlambda count below +`ceil(2/3·NODES)` and kill that devnet's finality. Keep aggregators on ethlambda +unless deliberately moving them. After converting, relabel Prometheus (see +observability above). Per-client images, flags, and interop status are in +`references/clients.md`. **Re-canary after any leanVM/image bump.** + +### Add swap / memory limits +Swap (persistent): `fallocate -l 16G /swapfile && chmod 600 && mkswap && swapon` ++ an `/etc/fstab` line, per host. Live container limits without recreating: +`sudo docker update --memory 8g --memory-swap 16g --memory-reservation 2g `. + +## Resources + +**Scripts.** "operator" runs on your machine, "host" is scp'd and run on a server. + +| Script | Where | Args / env | What | +|---|---|---|---| +| `make-genesis.sh` | operator | `OUTDIR NODES SUBNETS KEYS_DIR SRC_VCONFIG [GT_OFFSET]`; `VALIDATORS_PER_NODE` | Build one devnet's genesis dir: 127.0.0.1 ENRs, per-subnet aggregators, hardlinked keys, runs lean-quickstart `generate-genesis.sh` + subnet alignment | +| `merge-keyshards.py` | operator | `OUTDIR SHARD_DIR...` | Merge hash-sig key shards into one 0..N-1 dir + manifest (hash-sig-cli has no `--start-index`, so parallel keygen collides). Refuses to mix key formats, so a stale 52-byte `Dim46` shard can't poison a 32-byte `Dim42` genesis | +| `subnet-align-validators.py` | operator | `GENESIS_DIR NODES SUBNETS VPN` | Rewrite `annotated_validators.yaml` so each node's validators sit in ONE subnet. Called by `make-genesis.sh`; identity at `VPN=1` | +| `start-devnet.sh` | host | `NODES SUBNETS [N:client ...]`; `ETH_IMG`, `MEM`, `LOGOPT`, `EXTRA_ETH_FLAGS`, `ALLOW_STALE_DATA` | Launch a fresh devnet (all-ethlambda + canaries, per-subnet aggregators). Preflight: genesis/SUBNETS match, empty data dirs, known clients | +| `host-check.sh` | host | `[NODES]`; `LAG_THRESHOLD` | Per-node table (container status, restarts, head/justified/finalized, sync, aggregator, peers) read from `127.0.0.1` only — works when the central prom doesn't. Exit ≠0 = needs attention | +| `start-range.sh` | host | `START END SUBNETS [N:subnet ...]`; `ETH_IMG`, `MEM`, `LOGOPT`, `EXTRA_ETH_FLAGS`, `ALLOW_STALE_DATA` | Launch one host's SLICE of a chain that spans hosts (`start-devnet.sh` only does `0..NODES-1` with aggregator = index). Aggregator→subnet map is explicit; preflight also logs the genesis pubkey size so a 52B/32B image mismatch is caught before launch | +| `check-range.sh` | host | `START END` | Range-aware `host-check.sh` for a host owning `START..END` (status, head/justified/finalized, peers, aggregator, restarts). Exit ≠0 = needs attention | +| `teardown.sh` | host | `KEEP_GENESIS`, `STAGED` | Retire this host's slice in wipe-before-stamp order: clear `--restart` policy then remove containers, archive genesis to `genesis.bak-` (**keeps the retired chain's hash-sig keys**), promote a staged genesis, wipe `data/node_*` | +| `cs-restart.sh` | host | `CS_PORT[,PORT2] node...`; `IMAGE`, `AGG`, `SUBNETS` | Checkpoint-sync restart: 60s backoff, crash logs, aggregator role **preserved** by default (`AGG=auto\|\|off` to set it, `IMAGE=` to canary a build) | +| `convert.sh` | host | `CS_PORT N:client[:agg]...`; `ACC` (default: from genesis) | Convert nodes to other clients. Validates specs first; warns on aggregator loss | +| `start-observability.sh` | host | `RECREATE`, `PROM_*`, `CADVISOR_*` | Create this host's prometheus + cadvisor containers (node_exporter is systemd, not here) | +| `start-promtail.sh` | host | `PROMTAIL_IMG` | (Re)launch the log shipper. Safe on a live devnet (not a gossip peer) | +| `prometheus-config.sh` | operator | `NETWORK NODES HOST_IP CENTRAL_WRITE_URL [N:client ...]` | Emit a per-devnet prometheus.yml (`network` label + per-node `client_type`) | +| `promtail-config.sh` | operator | `NETWORK HOST_IP LOKI_PUSH_URL [N:client ...]` | Emit a per-host promtail.yml (docker_sd → central Loki, labels mirror prometheus, backlog guard) | +| `sweep.sh` | operator | `CENTRAL_PROM_URL` | Cross-devnet audit: head/justified/finalized + client mix from the central Prometheus | +| `deploy-finality-alert.sh` | operator | `[WEBHOOK_FILE]`; `METRICS_HOST`, `GRAFANA_*`, `PROM_DS_UID`; `DRY_RUN` | Render + ship the "lost finality" Slack alert to the central Grafana | +| `devnet-env.sh` / `devnet.env.example` | operator | `$DEVNET_ENV`, `./devnet.env`, script dir | Load this deployment's hosts/urls/Grafana ids as defaults; exported vars win. Copy the example to `devnet.env` (gitignored) once | + +**Grafana dashboards** (copy into the central Grafana's dashboards dir — +`GRAFANA_DASHBOARDS_DIR`, *not* the provisioning tree; they auto-load in ~30s and +pick their datasource through a template variable, so no editing): + +| File | uid | Content | +|---|---|---| +| `client-dashboard.json` | `lean-ethereum-clients-dashboard` | The main per-node dashboard: Overview (start time, validators, committees, head/justified/finalized, slot + finality-delay graphs) plus 14 collapsed sections (config, sync, peers, req/resp + mesh, gossip, gossip arrival timing, PQ signatures, aggregation coverage, block production, proposal internals, fork choice, attestations, state transition, storage + tick health). Filters `network`/`job`/`instance` — `instance` is the **host**, not the node. The config section's **Image build** table answers "which commit is this node running" for **every** client, from the image's OCI `revision`/`ref.name` labels re-exported by cAdvisor as `container_label_*` (see `references/operations.md`) — ethlambda's own `lean_node_info{version}` already embeds its short SHA, partner clients' usually do not | +| `devnet-overview-dashboard.json` | `devnet-finality-overview` | head / justified / finalized per devnet, one series per `network` | +| `resources-dashboard.json` | `devnet-resources` | Per-node CPU cores + memory working set (+ limit + %-of-limit for OOM watch), OOM kills, restart events, current uptime, and the same **Image build** commit-per-node table as the client dashboard — all from cAdvisor, so read a fresh uptime next to an unchanged commit as "came back on the same build" | +| `logs-dashboard.json` | `devnet-logs` | Loki logs panel + log-volume-by-level, filtered `network`/`node`/`stream`/`search` | +| `grafana-finality-alert.yaml.template` | — | Unified-alerting provisioning for the per-devnet "lost finality" rule + Slack contact point; placeholders filled by the deploy script | + +**One dashboard in that dir is NOT ours: `node-exporter-full.json` (uid `rYdddlPWk`).** +It is host-level CPU/RAM/disk/net for the systemd node_exporter on `:9122`, +the counterpart to the container-level resources dashboard, and it comes from the +same repo the node_exporter itself does — `lambdaclass/monitoring-stack`, as +`ansible/files/infra-dashboard.json` (its `ansible/grafana.yml` installs it). That +repo in turn tracks the community **Node Exporter Full** dashboard, `gnetId: 1860`, +maintained at `github.com/rfmoz/grafana-dashboards`. Deliberately **not** vendored +here: it is ~460 KB of generated JSON we do not maintain, it would go stale against +its source, and this skill has no business owning it. Refresh it from the source +instead, which needs no edit (its `ds_prometheus` variable already ships with an +empty `current`, so it falls back to the default datasource): +```bash +gh repo clone lambdaclass/monitoring-stack /tmp/monitoring-stack -- --depth 1 +cat /tmp/monitoring-stack/ansible/files/infra-dashboard.json \ + | ssh "$METRICS_HOST" "sudo tee $GRAFANA_DASHBOARDS_DIR/node-exporter-full.json >/dev/null" +``` +So it is the one file the `md5sum` audit below must skip: every OTHER server +`.json` has a `scripts/` counterpart, this one has an upstream instead. + +**References.** `node-health.md` — per-node "all is good" checklist (chain follow, +valid+useful attestations, non-empty proposals, aggregate+publish, on-time duties, +peers+mesh, votes-land-in-blocks) with a Prometheus query + log grep per item, and +why devnet-health is judged separately. `operations.md` — topology model, the +failure mode behind each golden rule, bootstrap-vs-stall diagnosis, detached-SSH / +pkill gotchas, prometheus + Loki wiring and their traps, host recovery. +`clients.md` — per-client images, CLI shapes, conversion principle, interop status. + +Topology specifics and chain history for the operator's particular deployment +live in their project memory notes; verify versions/interop against the live +chain — status is version-specific. diff --git a/.claude/skills/multi-server-devnet/references/clients.md b/.claude/skills/multi-server-devnet/references/clients.md new file mode 100644 index 00000000..d69cf792 --- /dev/null +++ b/.claude/skills/multi-server-devnet/references/clients.md @@ -0,0 +1,158 @@ +# Multi-Client Conversion Reference + +How to run zeam / ream / qlean / gean / lantern / grandine validators in place of +ethlambda on a devnet, and current interop status. `scripts/convert.sh` implements +all of this; this file explains the why and the gotchas. + +## Core principle: keep the node identity, rename only the container + +All genesis artifacts are keyed by **node identity** `node_N` +(`annotated_validators.yaml`, `validator-config.yaml`, `node_N.key`, +`hash-sig-keys/validator_N_*`, data dir `data/node_N`). So any client can take over +a validator slot **without touching identity or data layout**: keep `--node-id +node_N` and the key `/config/node_N.key`. + +What *does* change is the **container/display name**, which becomes `_N` +(e.g. `zeam_8`) so cAdvisor's `name` label and promtail's `node` label — and hence +Grafana — show which client is running. Scripts locate a node's existing container +by the `_N$` suffix, so the prefix can differ from run to run. + +Conversion = swap image + CLI shape + wipe data + checkpoint sync. Never keep-DB +across a client switch (wrong-format data) — wipe and checkpoint-sync from a +healthy **same-devnet** node. + +## Vote math constraint + +Finality threshold is `ceil(2/3 · NODES)` aligned votes on that devnet. Keep +aggregators on ethlambda and keep the ethlambda node count above the threshold, so +finality survives any single non-eth cohort failing. Do NOT convert so many nodes +that ethlambda drops below the threshold unless every other client is proven to +reliably contribute justification votes. Convert a **canary** of each client type +first and confirm `vote_count` rises before mass conversion. + +## Images (devnet5 — current) + +| Client | Image | Notes | +|--------|-------|-------| +| ethlambda | `ghcr.io/lambdaclass/ethlambda:devnet5` | | +| zeam | `blockblaz/zeam:devnet5` | needs `--security-opt seccomp=unconfined` | +| ream | `ghcr.io/reamlabs/ream:latest-devnet5` | | +| qlean | `qdrvm/qlean-mini:devnet-5-amd64` | arch-specific tag (`-amd64`/`-arm64`) | +| gean | `ghcr.io/geanlabs/gean:devnet5` | | +| lantern | `bitminemavan/lantern:devnet5` | `v0.0.X` == devnetX; see mirror note | +| grandine | `sifrai/lean:devnet-5` | | + +**`bitminemavan/lantern` is lantern's current repo** — it is what upstream +lean-quickstart's ansible pulls and where new builds land (including +`devnet5-leanvm-main` for chains on leanVM `main`). `piertwo/lantern` is the older +repo: same tag names, but it stopped tracking (its `devnet5` predates +bitminemavan's). Use `bitminemavan`; the scripts default to it. + +devnet4 images (older): `blockblaz/zeam:devnet4`, +`ghcr.io/reamlabs/ream:latest-devnet4`, `qdrvm/qlean-mini:devnet-4-amd64`, +`sifrai/lean:devnet-4`, `bitminemavan/lantern:v0.0.4`. + +Image tag must match the chain's leanVM/proof format. After a leanVM bump, other +clients must ship a matching build or signature aggregation deserialization fails. +**Re-canary every client after any bump.** + +## CLI shapes (per node) + +Placeholders below: `N` = node index, `G`/`A`/`M` = gossip/api/metrics ports +(`9000+N`/`5052+N`/`9200+N`), `ACC` = the devnet's `SUBNETS` +(`ATTESTATION_COMMITTEE_COUNT`), `URL` = `http://127.0.0.1:/lean/v0/states/finalized`. +Aggregator flags in `[...]` are added only for an aggregator node, and it covers a +**single** subnet: `--aggregate-subnet-ids $((N % ACC))`. + +zeam — `--validator-config genesis_bootnode` (NOT a file path; the file form fails +with `NotDir`). Reads the genesis dir via `--custom-genesis`. The log flags come +before the `node` subcommand; `--console_log_level info` is what puts zeam's own +duty publishes on stdout where the duty-timing profiler can read them (the file +sink stays at warn): +``` +--log_file_active_level warn --console_log_level info node \ + --custom-genesis /config --validator-config genesis_bootnode --data-dir /data \ + --node-id node_N --node-key /config/node_N.key --metrics-enable \ + --api-port A --metrics-port M --attestation-committee-count ACC \ + [--is-aggregator --aggregate-subnet-ids ] --checkpoint-sync-url URL --db-backend lmdb +``` + +ream — uses `annotated_validators.yaml`, `--socket-port` for QUIC: +``` +--data-dir /data lean_node --network /config/config.yaml \ + --validator-registry-path /config/annotated_validators.yaml --bootnodes /config/nodes.yaml \ + --node-id node_N --node-key /config/node_N.key --socket-port G \ + --metrics --metrics-address 0.0.0.0 --metrics-port M --http-address 0.0.0.0 --http-port A \ + --attestation-committee-count ACC [--is-aggregator --aggregate-subnet-ids ] --checkpoint-sync-url URL +``` + +qlean — `--genesis-dir`, multiaddr listen, NO `--aggregate-subnet-ids` (crashes +`unrecognised option`); it aggregates only auto-derived subnets: +``` +--genesis-dir /config --data-dir /data --node-id node_N --node-key /config/node_N.key \ + --listen-addr /ip4/0.0.0.0/udp/G/quic-v1 --metrics-host 0.0.0.0 --metrics-port M \ + --api-host 0.0.0.0 --api-port A --attestation-committee-count ACC [--is-aggregator] --checkpoint-sync-url URL -linfo +``` + +lantern — C client. Docker entrypoint runs the binary directly when argv[0] starts +with `--`. `--validator_config` is a DIR (holds both annotated_validators.yaml + +validator-config.yaml); needs `--genesis-state /config/genesis.ssz`; node key via +`--node-key-path`; multiaddr `--listen-address`. Reads the gossip fork digest from +genesis ENRs in nodes.yaml, falling back to `--devnet NAME` as the topic segment — +pass `--devnet 12345678` to match the cross-client dummy digest. Supports +`--checkpoint-sync-url` (URL must include the `/lean/v0/states/finalized` path) and +`--is-aggregator --aggregate-subnet-ids`: +``` +--data-dir /data --genesis-config /config/config.yaml --nodes-path /config/nodes.yaml \ + --genesis-state /config/genesis.ssz --validator_config /config \ + --node-id node_N --node-key-path /config/node_N.key \ + --listen-address /ip4/0.0.0.0/udp/G/quic-v1 --http-port A --metrics-port M \ + --hash-sig-key-dir /config/hash-sig-keys --attestation-committee-count ACC \ + --devnet 12345678 [--is-aggregator --aggregate-subnet-ids ] --checkpoint-sync-url URL +``` + +grandine — `--port` for QUIC, `--hash-sig-key-dir` (singular): +``` +--genesis /config/config.yaml --validator-registry-path /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml --node-id node_N --node-key /config/node_N.key \ + --port G --address 0.0.0.0 --http-address 0.0.0.0 --http-port A \ + --metrics --metrics-address 0.0.0.0 --metrics-port M --hash-sig-key-dir /config/hash-sig-keys \ + --attestation-committee-count ACC [--is-aggregator --aggregate-subnet-ids ] --checkpoint-sync-url URL +``` + +gean — Go client. `--custom-network-config-dir` is the genesis dir; gossip port flag +like ethlambda. **`--data-dir` is not optional in practice**: it defaults to a +container-relative `./data` (= `/app/data`), so without it the Pebble DB lives +inside the container instead of the `/data` mount — a host-side `rm -rf +data/node_*` then silently wipes nothing, and the DB dies with the container: +``` +--custom-network-config-dir /config --data-dir /data --gossipsub-port G --node-id node_N \ + --node-key /config/node_N.key --http-address 0.0.0.0 --api-port A --metrics-port M \ + --attestation-committee-count ACC [--is-aggregator --aggregate-subnet-ids ] [--checkpoint-sync-url URL] +``` + +## convert.sh usage + +``` +convert.sh ... spec = N:client[:agg] client in zeam|ream|qlean|gean|lantern|grandine +# e.g. convert.sh 5052 25:zeam 26:zeam 27:zeam +# ACC= only to override what the genesis says +``` +No BASE arg (each devnet numbers nodes from 0). `ACC` (the devnet's `SUBNETS` / +`ATTESTATION_COMMITTEE_COUNT`) is read from `genesis/config.yaml` on the host, so it +can't silently disagree with the chain — a wrong value puts the node's votes on a +subnet nobody aggregates. CS_PORT = a healthy same-devnet node's api port. + +Guards, both learned the hard way: + +- **Specs are validated before anything is removed.** A typo'd client name used to + be caught only after the container was gone, the data wiped and the 60s backoff + slept — leaving the node down. +- **Converting an aggregator away warns.** If the node currently runs with + `--is-aggregator` and the new spec has no `:agg`, that subnet is left with no + aggregator: votes verify, nothing is stored, `attestation_count` goes to 0 and + finality dies quietly. Either keep the role (`N:client:agg`) or move it to + another node first (`AGG= cs-restart.sh `). + +Keeps the 60s backoff, memory limits, and json-log caps. After converting, relabel +prometheus (see operations.md) and confirm with `host-check.sh`. diff --git a/.claude/skills/multi-server-devnet/references/node-health.md b/.claude/skills/multi-server-devnet/references/node-health.md new file mode 100644 index 00000000..7f51fdae --- /dev/null +++ b/.claude/skills/multi-server-devnet/references/node-health.md @@ -0,0 +1,351 @@ +# Is this node working? — the "all is good" checklist + +A tool for answering, any time you want to know, whether a running node is doing +its job: a spot check on one node, a sanity pass after a config change or +conversion, or one input into a stall investigation. Items 1–7 verify the node +itself. + +It also separates a **node** problem from a **devnet** problem, because the two +are not the same: a node can be perfectly healthy while the chain fails to +finalize (not enough of its peers are voting), and a node can be quietly broken +while the chain still finalizes without it. Judge the chain with the devnet-level +section at the end, never from a single node. + +Everything below is grounded in real metric names (`crates/**/metrics.rs`) and +log strings (`crates/blockchain/src`). Cross-checks: + +- **Duty *timing*** (where in the slot each duty actually lands, spill into the + next slot): use the `devnet-profiling` skill — this doc only flags whether + timing is in-budget, not the per-offset breakdown. +- **Log triage** (error classes, fork analysis): `devnet-log-review` skill. +- **Bootstrap-vs-stall** for the finality section: `references/operations.md`. + +## Conventions for the queries below + +- Metrics come from the central Prometheus, read either through the central + Grafana or directly via its API (`$CENTRAL_PROM_URL/api/v1/query`, the same base + `sweep.sh` takes). The Grafana URL, datasource uid, and whether anonymous read is + enabled are deployment-specific — get them from the operator's notes, don't assume. + Every series carries `network=""`, `client_type`, `job`, `instance`, + `type`. Queries here use `{network="$NET"}` for a whole devnet. +- **Scope to one node with `job`, not `instance`.** `job` is the per-node id + `_n` (e.g. `job="ethlambda_0"`, `job="zeam_8"`), matching the + container/display name. `instance` is the **host** — every node on a host shares + it — so it cannot isolate a node. Single-node filter: `{network="$NET", job="ethlambda_0"}`. +- The queries below were run against a live datasource; the sanity values quoted + are from a healthy 32-node devnet (scale the per-devnet counts to your `NODES`). +- Slot = 4 s, `INTERVALS_PER_SLOT` = 5, so one interval ≈ 800 ms. A validator + attests **once per slot** on subnet `n % SUBNETS`. Healthy per-validator + attestation rate ≈ 0.25/s. +- Log greps are written as match strings. Apply them to + `sudo docker logs ` or, via Loki in Grafana Explore, as + `{network="$NET", node="_n"} |= ""`. +- **No Prometheus?** `scripts/host-check.sh` on the host covers items 1, 4 and 6 + at a glance (head/justified/finalized, sync state, aggregator flag, peer count, + container status + restart count) straight off each node's `127.0.0.1` metrics + port. Use it when the central stack is unreachable, or as the first pass before + reaching for the queries below. + +--- + +## Checklist + +A healthy running node should, continuously: + +1. **Follow the chain** — import blocks as they arrive; head keeps climbing and + stays within a slot or two of wall-clock (not stuck in `syncing`). +2. **Emit a valid, *useful* attestation every slot** — not just a well-formed + vote, but one that can actually move justification (see item 2 for the + source/target rule). +3. **Build and publish a non-empty block in its proposer slots** — a block that + actually packs attestations, not an empty one. +4. **If it is an aggregator** — aggregate its subnet's gossip signatures into a + proof and publish the aggregate. +5. **Complete its duties on time** — build/attest/aggregate finish inside their + interval budget and don't spill into the next slot. +6. **Keep peers and gossip mesh healthy** — connected to the rest of the devnet + and subscribed with a populated mesh on the topics it uses. + +Plus one network-visible check that a single node only *partly* controls: + +7. **Its votes land in blocks** — blocks on the chain carry `attestation_count` + > 0, i.e. the votes it emits are being aggregated and packed by proposers. + +And one **devnet-wide** check that is **not** a node fault on its own: + +- **Finality actually advances** — `finalized`/`justified` climb across the + devnet, and `head − finalized` stays bounded. + +--- + +## 1. Follows the chain (imports blocks) + +**Means:** the node applies each new block, its head advances, and it is not +gated in `syncing`. Head advancing is the baseline "alive" signal. + +**Prometheus:** +```promql +# head is climbing (slots/sec; ~0.25 healthy) +deriv(lean_head_slot{network="$NET"}[5m]) > 0 +# head lag behind wall-clock; must stay small. The sync gate trips and +# suppresses duties once lag > SYNC_LAG_THRESHOLD (currently 4). +lean_current_slot{network="$NET"} - lean_head_slot{network="$NET"} +# sync state: 2=synced (healthy), 1=syncing, 0=idle +lean_node_sync_status{network="$NET", status="synced"} == 1 +# blocks are actually being processed (state transition runs) +rate(lean_state_transition_block_processing_time_seconds_count{network="$NET"}[5m]) > 0 +``` + +**Log:** `"Block imported successfully"` (one per imported block). + +**Red flags:** head flat while `lean_current_slot` climbs → `head_lag` grows past +4 → `lean_node_sync_status{status="syncing"}==1`, which **suppresses attest + +propose** (log `"Skipping attestations while syncing"`). Repeated +`"Block parent missing, storing as pending"` / `"Requested missing block from +network"` → it is missing ancestors and back-filling. + +--- + +## 2. Emits a valid, *useful* attestation every slot + +**Means — two levels:** + +- **Valid** (passes `validate_attestation_data` in `crates/blockchain/src/store.rs`): source → target + → head lie on one parent chain, `source.slot ≤ target.slot ≤ head.slot`, each + checkpoint's slot matches its block, and the vote isn't from the future. +- **Useful:** an attestation only *helps* if its **source matches the head + state's latest-justified checkpoint**, and its **target is a justifiable slot + strictly between the source and the head — not the head itself and not the + source.** ethlambda's `produce_attestation_data` (same file) enforces both: + - Source is read from the **head state's** `latest_justified`, which always + lies on the head's own chain — deliberately *not* the store's global + `latest_justified` (which is a highest-slot-wins max that can latch onto an + off-head sibling). Sourcing off-head makes every head-chain target fail + `is_valid_vote`, so the head can never re-justify and block production stalls + (leanSpec #1166 / #595). + - Target is `get_attestation_target`: walk back from the head + toward the safe target, then back to the nearest slot satisfying + `slot_is_justifiable_after(finalized)`, clamped to `≥ source`. That yields a + justifiable, on-chain slot newer than the source and at/behind the head. + +**Prometheus:** +```promql +# it is producing ~1 attestation/slot (count of production-timing observations) +rate(lean_attestations_production_time_seconds_count{network="$NET"}[5m]) # ≈0.25/validator +rate(lean_pq_sig_attestation_signing_time_seconds_count{network="$NET"}[5m]) # it signs its own vote +# nobody on the devnet is dropping votes as invalid (receiver side; should stay flat) +rate(lean_attestations_invalid_total{network="$NET"}[10m]) == 0 +``` + +**Log:** `"Published attestation"` (`%slot %validator_id`, once per slot per local +validator). + +**Red flags:** the clamp warning +`"Attestation target walked behind justified source, clamping to justified"` — +the vote stays valid but couldn't target a *new* justifiable slot that slot. +Receiver-side `UnknownSourceBlock` / `SourceNotAncestorOfTarget` / +`HeadOlderThanTarget` in *other* nodes' logs mean someone is emitting off-chain +(useless) votes. No `"Published attestation"` for a live validator → it is gated +(see item 1) or not assigned. + +--- + +## 3. Builds and publishes a non-empty block in its proposer slots + +**Means:** when it is the proposer, it builds a block that actually packs +aggregated attestation payloads (not empty) and publishes it aligned to the slot. + +**Prometheus:** +```promql +# builds are succeeding, none failing +rate(lean_block_building_success_total{network="$NET"}[15m]) > 0 +rate(lean_block_building_failures_total{network="$NET"}[15m]) == 0 +# blocks are non-empty: avg aggregated payloads per built block > 0 + rate(lean_block_aggregated_payloads_sum{network="$NET"}[15m]) +/ rate(lean_block_aggregated_payloads_count{network="$NET"}[15m]) +# distinct AttestationData packed per proposal (histogram; >0) +rate(lean_block_proposal_attestation_data_selected_sum{network="$NET"}[15m]) > 0 +``` + +**Log:** `"We are the proposer for this slot"` → `"Finished building block"` → +`"Published block"` (all `%slot %validator_id`). + +**Red flags:** `"We are the proposer"` with no following `"Published block"` (build +overran its budget — see item 5), or non-empty ratio near 0 while the devnet is +gossiping attestations → **no aggregator is storing signatures** (classic +`--is-aggregator`-missing symptom: attestations verify but `attestation_count` +stays 0). Empty blocks that aren't genesis/anchor are a real problem. + +--- + +## 4. If an aggregator: aggregate signatures and publish the aggregate + +**Only applies when `lean_is_aggregator == 1`.** Skip otherwise. + +**Means:** at interval 2 it aggregates the gossip signatures it collected on its +subnet into a proof and publishes the aggregate back onto gossip. + +**Prometheus:** +```promql +lean_is_aggregator{network="$NET"} == 1 +# aggregates are being produced, and are valid +rate(lean_pq_sig_aggregated_signatures_total{network="$NET"}[10m]) > 0 +rate(lean_pq_sig_aggregated_signatures_invalid_total{network="$NET"}[10m]) == 0 +# raw gossip signatures it collected to aggregate over — SCOPE to the aggregator +# job; non-aggregators read 0 BY DESIGN (they never store gossip sigs) +lean_gossip_signatures{network="$NET", job="ethlambda_0"} +# aggregation jobs are not being skipped/dropped (counter has a `reason` label) +rate(lean_aggregator_skipped_total{network="$NET"}[10m]) # ~0 +# validators the aggregate covers. High-cardinality (section × subnet), so ALWAYS +# pick a selector; section=combined,subnet=combined is the headline number +# (=NODES at full coverage, e.g. 32/32). It counts validators covered — it can't +# be mapped back to one validator's own vote. +lean_attestation_aggregate_coverage_validators{network="$NET", section="combined", subnet="combined"} +``` + +**Log:** `"Committee signature aggregated"` (`session_id, slot, raw_sigs, +participants, elapsed`). `"Failed to publish reaggregated attestation"` on the +publish path is a warn to watch. + +**Red flags:** `is_aggregator=1` but `lean_pq_sig_aggregated_signatures_total` +flat, or rising `lean_aggregator_skipped_total`, or the aggregator's +`lean_gossip_signatures` ≈ 0 (it hears no subnet signatures — often the +missing-60s-backoff mesh problem from golden rule #2). Coverage +(`section=combined`) stuck below `ceil(2/3·NODES)` starves finality. + +--- + +## 5. Completes its duties on time + +**Means:** each per-slot duty finishes inside its interval so it doesn't get +skipped or spill into the next slot. The tick scheduler drops an interval whose +work overran, so a slow build/aggregate simply doesn't publish. + +**Prometheus (histograms → use quantiles / `_count`):** +```promql +# time BETWEEN ticks; healthy p90 ≈ one interval (~0.8s) BY DESIGN (not a duty +# duration). Alarm only when it climbs WELL above ~0.8s → actor starved, ticks +# (hence duties) delayed. Live-healthy: ~0.80s. +histogram_quantile(0.9, sum by (le) (rate(lean_tick_interval_duration_seconds_bucket{network="$NET"}[10m]))) +# block build must finish in time to publish at interval 0 (keep p90 well under a +# slot). Live-healthy: ~0.98s. +histogram_quantile(0.9, sum by (le) (rate(lean_block_building_time_seconds_bucket{network="$NET"}[10m]))) +# aggregation must beat AGGREGATION_DEADLINE (800ms); early start opens at +# interval − EARLY_AGGREGATION_WINDOW (600ms) +histogram_quantile(0.9, sum by (le) (rate(lean_committee_signatures_aggregation_time_seconds_bucket{network="$NET"}[10m]))) +histogram_quantile(0.9, sum by (le) (rate(lean_pq_sig_aggregated_signatures_building_time_seconds_bucket{network="$NET"}[10m]))) +``` + +**Receiver side** — the same question asked of what *arrives* rather than what +this node produces. `lean_gossip_*_arrival_total` classifies each gossip message +by whether it landed inside the interval it was due in, so an on-time fraction +well under 1 says votes are arriving late rather than not at all: +```promql +# share of arrivals inside their due interval, per node (block|attestation|aggregation) +sum by (job) (rate(lean_gossip_attestation_arrival_total{network="$NET", position="inside"}[10m])) +/ sum by (job) (rate(lean_gossip_attestation_arrival_total{network="$NET"}[10m])) +# how late, in seconds (absolute distance from the due interval's start) +histogram_quantile(0.9, sum by (le, job) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network="$NET"}[10m]))) +``` +One node low while the fleet is fine is that node's own clock or CPU; everyone +dropping together is chain-wide production or propagation. Graphed in the client +dashboard's **Gossip Arrival Timing** row. Aggregates are anchored to the latest +aggregation-interval boundary, not their own data slot, so their tail reads as +aggregation cost (cross-check the two aggregation-time histograms above), and +`position="before"` never appears for them. + +**Log:** presence/absence timing — e.g. `"Finished building block"` arriving +after the next slot boundary. For exact slot-relative offsets and spill +detection, use the **`devnet-profiling`** skill (this item only says in/out of +budget). + +**Red flags:** tick-interval p90 *well above* ~0.8 s (actor starved, ticks +delayed — ~0.8 s itself is normal); build p90 climbing toward a slot (leanVM +proving under CPU contention); aggregation p90 near/over AGGREGATION_DEADLINE +(800 ms) → aggregates published late or dropped. Usually a **host CPU/contention** +symptom, not a logic bug. + +--- + +## 6. Peers and gossip mesh healthy + +**Means:** connected to the rest of its (single-host) devnet and subscribed with +a populated mesh on the topics it uses (block, aggregation, its attestation +subnet). + +**Prometheus:** +```promql +# connected peers (on an isolated single-host devnet ≈ NODES-1) +sum(lean_connected_peers{network="$NET"}) +# mesh peers per topic — should be non-zero on subscribed topics +lean_gossip_mesh_peers{network="$NET"} +# connection churn should be low/flat +rate(lean_peer_disconnection_events_total{network="$NET"}[10m]) +``` + +**Log:** peer connect/disconnect events (`%peer_id %direction peer_count ...`). + +**Red flags:** `connected_peers` well below `NODES-1`, an empty mesh on the +node's attestation subnet (its votes reach no aggregator — the golden-rule-#2 +backoff trap), or high disconnect churn. + +--- + +## 7. Its votes land in blocks (network-visible) + +**Means:** the votes this node emits are actually aggregated and packed by +proposers, so blocks on the chain carry `attestation_count` > 0. A node only +*partly* controls this — it emits the vote (item 2) and reaches the mesh (item +6), but landing also needs a working aggregator (item 4) and an honest proposer. + +**Prometheus:** +```promql +# payloads packed per block. Buckets are coarse, so the median reads as a +# fractional bucket estimate (e.g. 0.5) — prefer the avg from item 3, or the +# attestation_count log below, as the clean "non-empty" check. +histogram_quantile(0.5, sum by (le) (rate(lean_block_aggregated_payloads_bucket{network="$NET"}[15m]))) +rate(lean_block_proposal_aggregates_selected_sum{network="$NET"}[15m]) > 0 +# validators covered by the aggregates (section=combined,subnet=combined = +# headline count, =NODES at full coverage). Counts validators, not a single vote. +lean_attestation_aggregate_coverage_validators{network="$NET", section="combined", subnet="combined"} +``` + +**Log:** `"Block multi-message aggregate proof verified"` carries +`attestation_count` — grep it and confirm the field is > 0 on imported blocks: +```bash +sudo docker logs 2>&1 | grep "Block multi-message aggregate proof verified" +# each line shows attestation_count=; n should be > 0 +``` + +**Red flags:** `attestation_count=0` on non-genesis blocks devnet-wide → no +aggregator is storing signatures, or the mesh is broken. Coverage stuck below +`ceil(2/3·NODES)` → votes land but not enough of them to justify. + +--- + +## Devnet-level: finality actually advances — NOT a per-node fault + +This is a property of the **chain**, not of one node. Judge it separately, and +never conclude "node X is broken" from non-finality alone: X can be doing items +1–7 perfectly while the devnet stalls because *other* nodes aren't voting. + +**Prometheus (aggregate across the devnet with `max`):** +```promql +# justified and finalized both climbing +deriv(max(lean_latest_justified_slot{network="$NET"})[10m:]) > 0 +deriv(max(lean_latest_finalized_slot{network="$NET"})[10m:]) > 0 +rate(lean_finalizations_total{network="$NET"}[15m]) > 0 +# head−finalized gap stays bounded (the finality Slack alert fires > 512) +max(lean_head_slot{network="$NET"}) - max(lean_latest_finalized_slot{network="$NET"}) +``` + +**Reading it:** +- A **young devnet** can sit at `finalized=0` while `justified` jumps only on + square/pronic-distance slots — that's the bootstrap regime, **not** a stall + (see `references/operations.md`). +- `finalized` frozen with `head` climbing and the gap widening = a real stall. + Diagnose the **devnet** ("A devnet stopped finalizing" workflow in `SKILL.md`): + vote count vs `ceil(2/3·NODES)` → which validator cohort is missing on the + aggregator → are missing nodes silent or voting stale targets → host CPU/mem. +- Per-node items 1–7 tell you whether *this* node is one of the healthy voters + or one of the culprits. diff --git a/.claude/skills/multi-server-devnet/references/operations.md b/.claude/skills/multi-server-devnet/references/operations.md new file mode 100644 index 00000000..cefd3b14 --- /dev/null +++ b/.claude/skills/multi-server-devnet/references/operations.md @@ -0,0 +1,359 @@ +# Multi-Server Devnet Operations + +Hard-won operational detail for lean-consensus devnets that run as detached +containers on remote hosts — **one independent single-host devnet per server**. +SKILL.md has the workflows; this file has the depth and the failure modes. + +## Topology model + +- Operator supplies SSH targets via `SERVERS` (any count, any names) and + `SSH_USER`; `docker` runs under `sudo`. No host count, address, or spec is + assumed. +- **Each server runs its own complete devnet** — its own genesis, GENESIS_TIME, + fork choice, and finality. No cross-server peering: every ENR is pinned to + `127.0.0.1`, so discovery never leaves the host even though all devnets share + the dummy fork digest. Per-devnet validator count `NODES` and committee count + `SUBNETS` can differ between servers. +- Three-layer naming, `0 ≤ n < NODES`, numbered from 0 **per devnet** (no + cross-host BASE): **identity** `node_n` (node-id, `node_n.key`, data-dir, + genesis validator name — client-agnostic); **container/display** `_n` + (docker `--name` → cAdvisor `name` + promtail `node`; `client_type` derived from + this prefix); **index** `n` for ports gossip `9000+n`, api `5052+n`, metrics + `9200+n`. Scripts locate a node's container by the `_n$` name suffix, so the + client prefix can change without touching identity or data. +- One aggregator per subnet: nodes `0 .. SUBNETS-1`, node `k` aggregates subnet + `k` (leanVM CPU isolation — one per subnet, no more). Node `n` attests on + subnet `n % SUBNETS`. +- Containers: `docker run -d --restart unless-stopped --network host`. Genesis + mounted `/opt/lean-quickstart/genesis:/config`, data + `/opt/lean-quickstart/data/node_n:/data`. +- **Finality threshold is per devnet:** `ceil(2/3 · NODES)` for that server. One + devnet stalling never affects another. +- A central Grafana/Prometheus stack runs on one chosen host; each server's + per-host prometheus on `127.0.0.1:9090` remote-writes to it with a per-devnet + `network` label. The central prom API (its host address, a distinct port like + `:9099`) is the one cross-devnet metrics source that survives a server going + dark, and what `sweep.sh` and Grafana read. + +## Launching commands detached over SSH + +A plain `ssh host 'setsid bash script & disown'` often hangs because the SSH +channel stays open. Use: + +```bash +ssh "$SSH_USER@$host" "setsid bash /tmp/script.sh ARGS >/tmp/script.out 2>&1 ` and the rule (nodes `0..SUBNETS-1`, node `k` → subnet `k`) is +re-derived. To SET the role instead of preserving it, the same script takes +`AGG=auto` (re-apply the topology rule — the repair for a node that lost its role), +`AGG=` (promote), or `AGG=off` (deliberate demotion); `IMAGE=` swaps the image +in the same pass, so canarying a build on an aggregator keeps its role. + +There is deliberately only ONE restart path: with a separate script per role, the +role handling silently drifts out of whichever one gets used every day. + +Verify after any restart with `host-check.sh` — one `AGG=yes` per subnet. + +## Diagnosing a finalization stall + +A stall is rarely "a client crashed." Walk these in order (all scoped to the one +affected devnet): + +0. **Bootstrap, not stall?** If the devnet is young and justified is still + advancing (jumping square/pronic slots) with finalized=0, it's the bootstrap + regime — leave it (see the young-devnet section above). +0b. **Per-node picture, from the host itself.** `host-check.sh [NODES]` on the + affected server: which containers are down/restarting, which heads lag past the + sync-gate threshold (those nodes stop attesting and proposing), and whether one + `AGG=yes` still exists per subnet. It reads only `127.0.0.1`, so it works when + remote_write or the central stack is the thing that is broken. +1. **Vote math.** Threshold is `ceil(2/3 · NODES)` for that devnet. Pull recent + `Checkpoint justified ... vote_count=V threshold=T` from a stable node. If + `V < T`, not enough aligned votes. +2. **Who's missing.** On the aggregator, list distinct validators heard: + `sudo docker logs --tail 4000 ethlambda_ | grep -oE "Received attestation from gossip slot=[0-9]+ validator=[0-9]+" | grep -oE "validator=[0-9]+" | sort -u` + Compare to the full set. A missing contiguous cohort = a partitioned/overloaded + host or a mesh-rejoin failure. +3. **Stale votes vs silence.** A node can keep attesting a frozen head/target + (counts for nothing) without going silent. Group recent votes by + `head_root|target_slot|source_slot`; fragmentation across many targets = nodes + on divergent heads. `participants=1` aggregation sessions = aggregator isolated. +4. **Host resources.** Query the central prom (reachable even when a host's SSH + is down): + - MemAvailable history: `node_memory_MemAvailable_bytes{instance=""}` via `query_range`. + - Per-container memory: `topk(8, container_memory_working_set_bytes{instance="",name!=""})` (cadvisor). + A host starving on RAM/CPU freezes its nodes' fork choice → stale votes → stall. +5. **Not a fork.** Check the other clients aren't rejecting blocks + (`grep -ciE "invalid|reject"` in zeam/qlean/ream logs). Zero rejections rules + out interop divergence. + +## The non-finality spiral + +Once finality stops, the unfinalized tree grows on every node, per-slot +processing cost rises, nodes lag at different rates, heads scatter, votes +fragment further — self-reinforcing. Recovery requires getting a coherent +threshold-meeting set back to the tip fast (checkpoint sync), not waiting for +laggards to crawl through the swamp (forward sync through the unfinalized region +is slow, ~1.3 slots/s ≈ minutes per node). When deep in the spiral, a full reset +to fresh genesis is faster and cleaner than nursing it back. + +## Resource guards (added after a host OOM) + +Each host gets a 16 GiB swapfile (`/swapfile`, in `/etc/fstab`). Each container +defaults to `--memory 8g --memory-swap 16g --memory-reservation 2g` (raise further +for a client that legitimately needs it). Apply live with `docker update --memory +... `; bake into any `docker run`. This converts a host-killing leak into a +single-container OOM-kill. +leanVM working set is normally ~4-5 GiB/node; a node sustained above that is +leaking. + +## Prometheus / Grafana wiring + +Each server runs its own per-host prometheus (`--web.listen-address=:9090`, +network-host) scraping that devnet's nodes (`172.17.0.1:9200+n`) with a +**`network=` label** + per-node `client_type`, plus the host's +node_exporter (`:9122`) and cadvisor (`172.17.0.1:9098`). It +`remote_write`s to the **central +prometheus**, which must run with `--web.enable-remote-write-receiver` and is +bound to a routable address + distinct port (e.g. `:9099`). Grafana reads the +central prometheus; the dashboards have a `network` template variable to pick a +devnet (`scripts/devnet-overview-dashboard.json` = head/justified/finalized; +`scripts/resources-dashboard.json` = per-node CPU + memory from cAdvisor). The +cAdvisor scrape job already exists in each host's prometheus.yml (target +`172.17.0.1:9098`, per-node `client_type` via `metric_relabel` on the container +`name`); the resources dashboard just consumes it. NOTE per-container **network/fs** +cAdvisor metrics are NOT meaningful under `--network host` (all containers report +the host namespace) — use node_exporter for host-level net/disk; cAdvisor is +trustworthy for per-container **CPU + memory** only. GOTCHA: cAdvisor scrapes EVERY +container (nodes + prometheus/cadvisor/promtail/grafana + the root cgroup), so the +resources dashboard's `name` template var MUST use `allValue: ethlambda_.*` (not +`.*`) — otherwise "All" pulls in infra containers, and since unlimited containers +report the HOST's total RAM as their `container_spec_memory_limit_bytes` (and 0 for +some → `inf%`), the memory limit line jumps to host-RAM and the %-of-limit panel +breaks. (For a multi-client devnet use `[a-z]+_[0-9]+` so canary containers like +`zeam_8`/`grandine_7` are included too.) SECOND GOTCHA: that `allValue` is spliced +VERBATIM into PromQL as `name=~""`, so it must be a PromQL-safe regex — +double-quoted PromQL strings reject `\d`/`\w` escapes (`parse error: unknown escape +sequence U+0064 'd'`), which 400s EVERY panel whenever "All" nodes is selected (the +default). Use POSIX classes (`[0-9]` not `\d`); the variable's `regex:` filter field +is JS and CAN keep `\d`, since only `allValue` reaches the query. (The Loki logs +dashboard can safely use `.*` because promtail only ships +`ethlambda_*` containers in the first place.) `scripts/prometheus-config.sh` +emits the per-host file; `scripts/start-observability.sh` creates the per-host +`prometheus` + `cadvisor` containers (it refuses to touch existing ones without +`RECREATE=1`, since a live scraper's flags may not be the ones it would use). + +**cAdvisor also carries the image's OCI labels**, which is the only cross-client +way to see *which commit a node is actually running*: docker copies an image's +`LABEL`s onto every container created from it, and cAdvisor (default +`--store_container_labels=true`) re-exports them on every `container_*` series as +`container_label_`. So +`container_label_org_opencontainers_image_revision` = the full 40-char SHA and +`..._ref_name` = the branch/tag — for partner clients too, whose own +`lean_node_info{version=…}` is often just a release string (`lantern` reports +`v0.0.5`) while ethlambda's already embeds the short SHA. It reflects the image +the container was **created** with, so it stays honest after a `docker pull` that +was not followed by a recreate — unlike the mutable tag in `image`. The client +dashboard's "Image build" table consumes it; if `Commit` is empty for a client, +that client's Dockerfile simply does not set the label (ethlambda's does, fed by +`GIT_COMMIT`/`GIT_BRANCH` build args from `docker_publish.yaml`). Two matcher +gotchas when writing such a panel: cAdvisor's `job` is `cadvisor` and its +`instance` is the **host**, so neither the `$job` nor the `$instance` filter of +the node panels applies — match the node through cAdvisor's `name` instead +(`name=~"$job"`, since the container name IS the node's job label), and AND a +`name=~".+_[0-9]+"` guard onto it so "All" (`.*`) does not drag the infra +containers in, per the gotcha above. +The **central** stack is a separate one-time deployment: a prometheus started with +`--web.enable-remote-write-receiver` on a routable address + distinct port (e.g. +`:9099`), a Loki with `allow_structured_metadata: true` + schema v13/tsdb and +`retention_enabled: true` (it is false by default and `retention_period` alone is +silently ignored), and a Grafana with provisioning dirs for datasources, +dashboards, and alerting. Two distinct dirs matter here: the **provisioning** tree +(`GRAFANA_PROV_DIR`, mounted at `/etc/grafana/provisioning`) holds the datasource, +dashboard-provider and alerting yamls; the **dashboard JSONs** live in a separate +host dir (`GRAFANA_DASHBOARDS_DIR`) bind-mounted read-only at the provider's +`options.path` (`/var/lib/grafana/dashboards`). `deploy-finality-alert.sh` writes +to the first; dashboards go to the second. A JSON dropped in +`GRAFANA_PROV_DIR/dashboards` is silently ignored. + +**node_exporter is a systemd service, not a container.** It's installed from +`github.com/lambdaclass/monitoring-stack` (ansible: `make inventory TARGET=...` +then `make node_exporter`), binds `:9122`, and `Requires= +tailscaled.service`. The per-host prometheus scrapes it at `:9122` (the +host's tailscale IP — reachable from the bridge container, same path as the +remote_write target). A "Node Exporter Full" Grafana dashboard (uid `rYdddlPWk`) +sits in the central grafana dashboards dir as `node-exporter-full.json`. It is NOT +vendored into this skill: it belongs to the same monitoring-stack repo the +node_exporter comes from (`ansible/files/infra-dashboard.json`, installed by that +repo's `ansible/grafana.yml`), which itself tracks the community dashboard +`gnetId: 1860` from `github.com/rfmoz/grafana-dashboards`. Copy it from there when +it needs refreshing — the source needs no edit, since its `ds_prometheus` variable +ships with an empty `current` and so resolves to the default datasource. It is the +one dashboard in that dir with no `scripts/` counterpart. +The monitoring-stack can also deploy grafana/prometheus/loki/promtail/alloy/ +pyroscope, but those would collide with this docker-based stack — install ONLY +node_exporter from it. + +After a client **conversion**, relabel BOTH `job_name:` and `client_type:` in +that host's `prometheus.yml` (Grafana groups by job name; converted-but-not- +relabeled nodes still show as ethlambda) — regenerate with `prometheus-config.sh` +passing the canary `N:client` specs. + +**Reload gotcha (single-file bind-mount inode trap):** `prometheus.yml` is +bind-mounted as a SINGLE FILE. `sed -i` (and most editors) write a new inode and +rename over the old one, but the bind mount is pinned to the original inode — so +the container keeps reading the OLD content. `docker kill -s HUP prometheus` +reloads the stale inode, and even `promtool check config` validates the stale +file, masking it. **Do `sudo docker restart prometheus`** after editing (rebinds +to the current inode), OR overwrite in place preserving the inode (`... | sudo +tee file`, not `sed -i`/`mv`). Verify with `curl 127.0.0.1:9090/api/v1/targets` +(job/client_type per scrapeUrl). All clients export standardized `lean_*` +metrics so shared panels work; client-native metrics (e.g. `zeam_*`) need +separate panels. After a host reboot, the per-host `prometheus`/`promtail` +containers may need `docker start`. + +## Logs: Loki + promtail + +Log shipping mirrors the metrics path. A per-host **promtail** container +(`--network host --user root`, docker-socket mounted) discovers the devnet's node +containers via the **Docker daemon API** (`docker_sd_configs`) and ships their +stdout/stderr to the **central Loki** (`grafana/loki`, filesystem store, 7d +retention, published on `:3100`). Grafana reads it via a provisioned +datasource (`url: http://loki:3100`, same docker network as grafana). Reading via +the daemon API — not the json files — makes promtail immune to `--log-opt` +rotation: it consumes the live stream. `promtail-config.sh` emits the per-host +file; `start-promtail.sh` launches it. promtail is NOT a gossip peer, so deploying +it needs no 60s backoff and never touches the devnet — only the socket. + +**Pipeline (in `promtail-config.sh`), in order:** (1) `drop older_than 1h` (backlog +guard, below); (2) `replace` to strip ANSI colour escapes (`\x1b\[[0-9;]*m`) from +the client pretty-logger output — without this every line is full of escape codes +and Loki's level auto-detect fails; (3) `multiline` to merge multi-line events +(fork-choice tree dumps, panics) — `firstline` is a leading ISO timestamp, so any +line that does NOT start with one is folded into the previous entry. State is +per-stream (node label). Without it, every tree-dump line is a separate level-less +entry (a large `` series ≈ nodes × dumps/min × lines, plus clutter); with it +the dump is one entry that inherits its firstline's level. NOTE the order matters — +strip ANSI BEFORE multiline so `firstline` can anchor on the bare `^\d{4}-…` +timestamp (verified: putting it after, or matching ANSI-prefixed lines, was the +difference between dumps merging and not). (4) `regex`+`labels` to lift the tracing +level into a low-card **indexed label** `level` (also drives Grafana's level +colouring); (5) a `match |= "="` wrapping `logfmt`+`structured_metadata` that attaches +high-cardinality fields (`slot`, `validator`, `proposer`, `block_root`, target/ +source slot+root, `attestation_count`, `peer_id`) as **structured metadata** — +queryable (`{…} | slot="123"`) and visible in Grafana log-details, but NOT indexed +as stream labels, so stream cardinality stays bounded. Indexed labels stay low-card: +`network` (SAME value as prometheus), `node` (`_N`, e.g. `zeam_8`), `client_type`, +`instance`, `stream`, `level`. **Rule of thumb:** low-cardinality → label; high- +cardinality → structured metadata (or a query-time `unwrap` metric / promtail +`metrics` stage), NEVER a label. The `match |= "="` gate keeps `logfmt` off lines +with no fields (e.g. multi-line tree dumps) so the per-line CPU cost stays low. +Requires Loki `allow_structured_metadata: true` + schema v13/tsdb (both already set). + +**First-start backlog is the one trap.** docker_sd has no `since`, so with an empty +`positions.yaml` promtail replays each container's ENTIRE retained history. With +large json-file caps (e.g. the old 600m×10 = 6 GiB) that is days of logs: +- Loki rejects everything older than its 7d retention (`400 ... timestamp too old`) + and rate-limits the firehose (`429 ... ingestion rate limit`, default 4 MB/s). +- promtail burns 1.7–2.5 cores per host *just reading* the backlog — which + competes with latency-sensitive leanVM block-building and can disturb the devnet. + +`promtail-config.sh` bakes in a `drop older_than: 1h` pipeline stage so only recent +lines are *shipped* (no Loki flood, no too-old/429 spam), but promtail still *reads* +the whole backlog to advance its position. So on a host with large logs, **seed the +positions to "now" after first launch** to skip the read entirely: +```bash +POS=/opt/lean-quickstart/observability/promtail-data/positions.yaml +sudo docker stop promtail; N=$(date +%s) +sudo sed -i -E "s/: \"[0-9]+\"/: \"$((N-300))\"/" "$POS"; sudo docker start promtail +``` +The positions format is `cursor-: ""` — the value +is the last-read Docker log timestamp, so overwriting it with `now-300` makes +promtail resume ~5 min ago and tail forward. Positions live in the mounted +`promtail-data/` dir (NOT a single-file bind-mount, so `sed -i` is safe here), so the +seed is a one-time per-host fixup; subsequent restarts resume cleanly. CPU drops to +~0.1% once caught up; steady-state shipping of warn/info devnet logs is well under +the 4 MB/s Loki limit. Verify ingestion: +`curl -s :3100/loki/api/v1/label/network/values` (all devnets present) +and a `query_range` on `{network="..."}`. promtail's own `:9080/metrics` +(`promtail_sent_entries_total`, `promtail_request_duration_seconds_count` by +`status_code`, `promtail_dropped_entries_total`) is the definitive "is it shipping?" +check — a 204 status with rising sent-entries means logs are landing even if a +narrow query window shows none yet (verbose hosts lag a few seconds under load). + +Reducing the json-file caps (`start-devnet.sh` LOGOPT, now 200m×5 = 1 GiB) also +shrinks the first-start backlog, but the caps only change on container recreate — +they do NOT apply to already-running nodes, so a live fleet keeps its old (possibly +6 GiB) logs until the next cs-restart/convert/fresh-genesis. Until then, seed +positions rather than letting promtail chew through the old backlog. + +## Recovering an unreachable host + +If a host drops off the network (e.g. OOM starved sshd), it may only be +recoverable via a provider console reboot — there may be no alternate route. +After reboot its old containers auto-start (`--restart unless-stopped`) on the +stale chain. Since each server is an independent devnet, the other devnets are +unaffected and keep finalizing. For the recovered server, either `cs-restart.sh` +its nodes (if the chain is still healthy and you only need them to rejoin) or, if +the chain is dead, stop the stale containers (`docker rm -f`) and start fresh +(`make-genesis.sh` → ship → `start-devnet.sh`). The central +Grafana/Prometheus, if it lives on a different host, keeps showing every other +devnet throughout. diff --git a/.claude/skills/multi-server-devnet/scripts/check-range.sh b/.claude/skills/multi-server-devnet/scripts/check-range.sh new file mode 100755 index 00000000..814dc212 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/check-range.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# HOST-side: summarise this host's slice of a cross-server devnet. The skill's +# host-check.sh assumes nodes 0..N-1; a host owning 32-63 needs a range. +# Usage: check-range.sh START END +set -u +START=$1; END=$2 +printf "%-12s %-8s %6s %6s %6s %5s %4s %6s\n" NODE STATUS HEAD JUST FIN PEERS AGG RESTRT +bad=0 +for n in $(seq "$START" "$END"); do + c=$(sudo docker ps -a --format '{{.Names}}' | grep -E "_${n}$" | head -1) + [ -z "$c" ] && { printf "%-12s %-8s\n" "node_$n" MISSING; bad=1; continue; } + st=$(sudo docker inspect "$c" --format '{{.State.Status}}') + rs=$(sudo docker inspect "$c" --format '{{.RestartCount}}') + m=$(curl -s --max-time 3 "localhost:$((9200+n))/metrics" 2>/dev/null) + g(){ echo "$m" | awk -v k="$1" '$1==k{printf "%d", $2; exit}'; } + head=$(g lean_head_slot); just=$(g lean_latest_justified_slot); fin=$(g lean_latest_finalized_slot) + agg=$(g lean_is_aggregator); peers=$(echo "$m" | grep -c '^lean_connected_peers') + printf "%-12s %-8s %6s %6s %6s %5s %4s %6s\n" "node_$n" "$st" "${head:--}" "${just:--}" "${fin:--}" "$peers" "${agg:-0}" "$rs" + [ "$st" != running ] && bad=1 + [ -z "$head" ] && bad=1 +done +exit $bad diff --git a/.claude/skills/multi-server-devnet/scripts/client-dashboard.json b/.claude/skills/multi-server-devnet/scripts/client-dashboard.json new file mode 100644 index 00000000..d6a524c5 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/client-dashboard.json @@ -0,0 +1,11445 @@ +{ + "__inputs": [ + { + "name": "datasource", + "label": "prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "12.3.2" + }, + { + "type": "panel", + "id": "piechart", + "name": "Pie chart", + "version": "" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "stat", + "name": "Stat", + "version": "" + }, + { + "type": "panel", + "id": "table", + "name": "Table", + "version": "" + }, + { + "type": "panel", + "id": "text", + "name": "Text", + "version": "" + }, + { + "type": "panel", + "id": "timeseries", + "name": "Time series", + "version": "" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "links": [], + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 16, + "panels": [], + "title": "Overview", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Process start time of the longest-running node in scope (min across series), shown relative to now \u2014 i.e. how long this devnet has been running uninterrupted. A restart of any node does NOT move it; only the oldest node stopping does. Only ethlambda exports lean_node_start_time_seconds, so converted canaries never contribute.", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "super-light-green", + "mode": "fixed" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "dateTimeFromNow" + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 4, + "x": 0, + "y": 1 + }, + "id": 76, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": { + "valueSize": 20 + }, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "min by(network) (lean_node_start_time_seconds{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} * 1000)", + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "A" + } + ], + "title": "Oldest node start", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 4, + "x": 4, + "y": 1 + }, + "id": 103, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(network) (lean_validators_count{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Total number of validators", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 4, + "x": 8, + "y": 1 + }, + "id": 40, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_attestation_committee_count{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Number of attestation committees", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 4, + "x": 12, + "y": 1 + }, + "id": 87, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Head slot", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 4, + "x": 16, + "y": 1 + }, + "id": 86, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Latest justified slot", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 4, + "x": 20, + "y": 1 + }, + "id": 88, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Latest finalized slot", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 4 + }, + "id": 35, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Head slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 4 + }, + "id": 34, + "options": { + "legend": { + "calcs": [ + "max", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Latest justified slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 4 + }, + "id": 33, + "options": { + "legend": { + "calcs": [ + "max", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job, instance)(lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": " Latest finalized slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 8, + "x": 0, + "y": 11 + }, + "id": 30, + "options": { + "legend": { + "calcs": [ + "min", + "max", + "mean", + "last" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m]) - avg_over_time(lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m])", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Head - Finalized delay (slots)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 8, + "x": 8, + "y": 11 + }, + "id": 29, + "options": { + "legend": { + "calcs": [ + "min", + "max", + "mean", + "last" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m]) - avg_over_time(lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m])", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Head - Justified delay (slots)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 8, + "x": 16, + "y": 11 + }, + "id": 31, + "options": { + "legend": { + "calcs": [ + "min", + "max", + "mean", + "last" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m]) - avg_over_time(lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m])", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Justified - Finalized delay (slots)", + "type": "timeseries" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 21 + }, + "id": 138, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Number of validators attached to each node", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + }, + "fieldMinMax": false, + "mappings": [] + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "custom.hideFrom", + "value": { + "legend": false, + "tooltip": true, + "viz": true + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 30 + }, + "id": 51, + "options": { + "legend": { + "displayMode": "table", + "placement": "right", + "showLegend": true, + "values": [] + }, + "pieType": "donut", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "sort": "desc", + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job, instance) (lean_validators_count{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "interval": "", + "legendFormat": "{{job}}", + "range": false, + "refId": "B", + "useBackend": false + } + ], + "title": "Validators list", + "type": "piechart" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "super-light-green", + "mode": "fixed" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false, + "filterable": true + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Field" + }, + "properties": [ + { + "id": "displayName", + "value": "aggregator" + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 30 + }, + "id": 89, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [] + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network,job,instance) (lean_is_aggregator{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}) > 0", + "instant": true, + "legendFormat": "{{job}}", + "range": false, + "refId": "A" + } + ], + "title": "Aggregators", + "transformations": [ + { + "id": "reduce", + "options": { + "reducers": [ + "lastNotNull" + ] + } + }, + { + "id": "filterFieldsByName", + "options": { + "include": { + "names": [ + "Field" + ] + } + } + } + ], + "type": "table", + "description": "Every node reporting lean_is_aggregator > 0. The reduce transformation is load bearing: an instant query returns one frame per aggregator, and the table panel renders multi-frame input as a frame picker, which is why this used to show a single entry with a dropdown." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "name" + }, + "properties": [ + { + "id": "custom.width", + "value": 112 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "job" + }, + "properties": [ + { + "id": "custom.width", + "value": 116 + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 30 + }, + "id": 75, + "options": { + "cellHeight": "sm", + "showHeader": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "last_over_time(lean_node_info{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[1m])", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Node info", + "transformations": [ + { + "id": "labelsToFields", + "options": { + "keepLabels": [ + "name", + "version", + "job" + ] + } + }, + { + "id": "merge", + "options": {} + }, + { + "id": "filterFieldsByName", + "options": { + "include": { + "names": [ + "job", + "version" + ] + } + } + }, + { + "id": "organize", + "options": { + "excludeByName": {}, + "includeByName": {}, + "indexByName": {}, + "renameByName": { + "job": "Node", + "name": "Client", + "version": "Version" + } + } + } + ], + "type": "table" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time since each node's process started, one tile per node. Colour flags a recent (re)start: red < 15m, orange < 1h, yellow < 6h, dark green beyond. Thresholds compare the tile's own value, which is why the value is an age rather than a timestamp. Only ethlambda exports lean_node_start_time_seconds, so converted canaries are absent.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "orange", + "value": 900 + }, + { + "color": "yellow", + "value": 3600 + }, + { + "color": "dark-green", + "value": 21600 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 39 + }, + "id": 72, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": { + "titleSize": 11, + "valueSize": 16 + }, + "textMode": "value_and_name", + "wideLayout": false + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "time() - lean_node_start_time_seconds{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}", + "instant": true, + "legendFormat": "{{job}}", + "range": false, + "refId": "A" + } + ], + "title": "Time since node start", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Which attestation subnet each node aggregates on. Was a line chart, which interpolated between discrete subnet indices: a node going from 2 to 5 was drawn passing through 3.5, which never happens. The assignment is static per node, so the value is shown directly instead. Sorted by subnet, so uneven distribution across subnets is visible at a glance.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "continuous-GrYlRd" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "filterable": true, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byType", + "options": "number" + }, + "properties": [ + { + "id": "custom.cellOptions", + "value": { + "mode": "basic", + "type": "color-background" + } + }, + { + "id": "custom.width", + "value": 120 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 39 + }, + "id": 90, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [ + { + "desc": false, + "displayName": "subnet" + } + ] + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job, instance) (lean_attestation_committee_subnet{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "interval": "", + "legendFormat": "{{job}}", + "range": false, + "refId": "B", + "useBackend": false + } + ], + "title": "Attestation committee subnet", + "type": "table", + "transformations": [ + { + "id": "reduce", + "options": { + "reducers": [ + "lastNotNull" + ] + } + }, + { + "id": "organize", + "options": { + "excludeByName": {}, + "includeByName": {}, + "indexByName": {}, + "renameByName": { + "Field": "node", + "Last *": "subnet", + "Last": "subnet" + } + } + } + ] + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Which build each node is actually running, read from the OCI labels baked into its container image (`org.opencontainers.image.revision` / `.ref.name`) and exported by cAdvisor as `container_label_*`. This is the label of the image the container was CREATED with, so it stays correct after a `docker pull` that has not been followed by a recreate -- unlike the image tag, which is mutable.\n\nCoverage depends on the client's Dockerfile setting the labels; a client that does not leaves Commit empty. Filtered by $network and $job only: cAdvisor's `instance` is the HOST and its `job` is `cadvisor`, so neither the $job nor the $instance matcher of the other panels applies -- the node is matched through cAdvisor's `name` (the container name, same string as the node's `job`).", + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "filterable": true, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "noValue": "No cAdvisor data for this network -- is its cadvisor container scraped?" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Node" + }, + "properties": [ + { + "id": "custom.width", + "value": 130 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Client" + }, + "properties": [ + { + "id": "custom.width", + "value": 110 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Commit" + }, + "properties": [ + { + "id": "custom.width", + "value": 350 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Branch / tag" + }, + "properties": [ + { + "id": "custom.width", + "value": 230 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Image built" + }, + "properties": [ + { + "id": "custom.width", + "value": 230 + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 47 + }, + "id": 215, + "options": { + "cellHeight": "sm", + "showHeader": true, + "sortBy": [ + { + "desc": false, + "displayName": "Node" + } + ] + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "last_over_time(container_last_seen{network=~\"$network\", name=~\"$job\", name=~\".+_[0-9]+\"}[5m])", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Image build (commit per node)", + "transformations": [ + { + "id": "labelsToFields", + "options": { + "keepLabels": [ + "name", + "client_type", + "container_label_org_opencontainers_image_revision", + "container_label_org_opencontainers_image_ref_name", + "image", + "container_label_org_opencontainers_image_created" + ] + } + }, + { + "id": "merge", + "options": {} + }, + { + "id": "filterFieldsByName", + "options": { + "include": { + "names": [ + "name", + "client_type", + "container_label_org_opencontainers_image_revision", + "container_label_org_opencontainers_image_ref_name", + "image", + "container_label_org_opencontainers_image_created" + ] + } + } + }, + { + "id": "organize", + "options": { + "excludeByName": {}, + "includeByName": {}, + "indexByName": { + "name": 0, + "client_type": 1, + "container_label_org_opencontainers_image_revision": 2, + "container_label_org_opencontainers_image_ref_name": 3, + "image": 4, + "container_label_org_opencontainers_image_created": 5 + }, + "renameByName": { + "name": "Node", + "client_type": "Client", + "container_label_org_opencontainers_image_revision": "Commit", + "container_label_org_opencontainers_image_ref_name": "Branch / tag", + "image": "Image", + "container_label_org_opencontainers_image_created": "Image built" + } + } + } + ], + "type": "table" + } + ], + "title": "Devnet configuration", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 22 + }, + "id": 128, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Share of the fleet in each state, stacked to full height (percent stacking, as in node-exporter's CPU Basic). Panel height is independent of node count, so this stays readable at 64+ nodes. The bars are normalized, so the exact node counts are in the legend and the tooltip instead of on the axis: percent stacking only changes rendering, the field values are still raw counts. $network is single-select, so this is always one devnet. Each state has its own target with `or vector(0)`, because bare count(... == 1) returns no series when a state is empty and 'zero syncing' would silently vanish from the stack instead of reporting 0.", + "gridPos": { + "h": 12, + "w": 16, + "x": 0, + "y": 31 + }, + "id": 137, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "hidden", + "fillOpacity": 80, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineWidth": 1, + "scaleDistribution": { + "type": "linear" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "synced" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "syncing" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "idle" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + } + ] + }, + "options": { + "barRadius": 0, + "barWidth": 0.97, + "fullHighlight": false, + "groupWidth": 0.7, + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "orientation": "auto", + "showValue": "never", + "stacking": "percent", + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + }, + "xTickLabelRotation": 0, + "xTickLabelSpacing": 100 + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(lean_node_sync_status{status=\"synced\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) or vector(0)", + "legendFormat": "synced", + "range": true, + "instant": false, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(lean_node_sync_status{status=\"syncing\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) or vector(0)", + "legendFormat": "syncing", + "range": true, + "instant": false, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(lean_node_sync_status{status=\"idle\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) or vector(0)", + "legendFormat": "idle", + "range": true, + "instant": false, + "refId": "C" + } + ], + "title": "Node sync status (fleet counts)", + "type": "barchart", + "transformations": [ + { + "id": "joinByField", + "options": { + "byField": "Time", + "mode": "outer" + } + } + ] + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current state of each node. Click the state column header to sort worst-first, or use the column filter to isolate non-synced nodes. Current state only, no history: read it alongside the fleet counts to its left.", + "gridPos": { + "h": 12, + "w": 8, + "x": 16, + "y": 31 + }, + "id": 133, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "filterable": true, + "inspect": false + }, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "idle", + "color": "blue", + "index": 0 + }, + "1": { + "text": "syncing", + "color": "orange", + "index": 1 + }, + "2": { + "text": "synced", + "color": "green", + "index": 2 + } + } + }, + { + "type": "special", + "options": { + "match": "null", + "result": { + "text": "no data", + "color": "text", + "index": 3 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + }, + { + "color": "orange", + "value": 1 + }, + { + "color": "green", + "value": 2 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byType", + "options": "number" + }, + "properties": [ + { + "id": "custom.cellOptions", + "value": { + "mode": "basic", + "type": "color-background" + } + }, + { + "id": "custom.width", + "value": 160 + } + ] + } + ] + }, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [] + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (network, job, instance) (\n (lean_node_sync_status{status=\"synced\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) * 2\n or\n (lean_node_sync_status{status=\"syncing\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) * 1\n or\n (lean_node_sync_status{status=\"idle\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) * 0\n)", + "legendFormat": "{{job}}", + "range": true, + "instant": false, + "refId": "A" + } + ], + "title": "Node sync status (per node)", + "type": "table", + "transformations": [ + { + "id": "reduce", + "options": { + "reducers": [ + "lastNotNull" + ] + } + } + ] + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 43 + }, + "id": 66, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_current_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Current slot", + "type": "timeseries" + } + ], + "title": "Sync Status", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 23 + }, + "id": 53, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 32 + }, + "id": 54, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_connected_peers{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Connected peers per node", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 32 + }, + "id": 105, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": " sum by (network, job, instance, client) (lean_connected_peers{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", client!=\"\"})", + "instant": false, + "interval": "", + "legendFormat": "{{job}} - {{client}}", + "range": true, + "refId": "A" + } + ], + "title": "Connected peers per node (detailed)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 41 + }, + "id": 55, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_peer_connection_events_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Peer connection events (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 41 + }, + "id": 56, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_peer_disconnection_events_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Peer disconnection events (per minute)", + "type": "timeseries" + } + ], + "title": "Peers", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 24 + }, + "id": 121, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 33 + }, + "id": 122, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (lean_gossip_mesh_peers{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Gossipsub mesh slots per node (all topics)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 33 + }, + "id": 123, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (protocol, le) (rate(lean_reqresp_request_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])))", + "interval": "", + "legendFormat": "{{protocol}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (protocol, le) (rate(lean_reqresp_request_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])))", + "interval": "", + "legendFormat": "{{protocol}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Req/resp request size (p99, by protocol)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 33 + }, + "id": 124, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (protocol, le) (rate(lean_reqresp_response_chunk_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])))", + "interval": "", + "legendFormat": "{{protocol}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (protocol, le) (rate(lean_reqresp_response_chunk_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])))", + "interval": "", + "legendFormat": "{{protocol}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Req/resp response chunk size (p99, by protocol)", + "type": "timeseries" + } + ], + "title": "Req/Resp & Gossip Mesh", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 25 + }, + "id": 99, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 34 + }, + "id": 100, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_gossip_block_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_gossip_block_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Bytes size of a gossip block message", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 34 + }, + "id": 101, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_gossip_attestation_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_gossip_attestation_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Bytes size of a gossip attestation message", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 34 + }, + "id": 102, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_gossip_aggregation_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_gossip_aggregation_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Bytes size of a gossip aggregated attestation message", + "type": "timeseries" + } + ], + "title": "Gossip messages", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 26 + }, + "id": 199, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Absolute distance between arrival and the start of the interval the message was due in, per node. Blocks anchor to interval 0 of their own slot. Only gossip arrivals are sampled; blocks fetched over req/resp during sync are excluded. The dashed line is one interval (0.8s): above it the typical message misses the interval it was due in. Because the histogram observes the absolute delta, an arrival 300ms early shares a bucket with one 300ms late; the arrival position panel below is what separates them. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.8 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 27 + }, + "id": 200, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n sum by (le, job) (rate(lean_gossip_block_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n sum by (le) (rate(lean_gossip_block_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))\n)", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip block arrival delay (p50)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Absolute distance between arrival and the start of the interval the message was due in, per node. Attestations anchor to interval 1 (AttestationProduction) of their data slot. An attestation that lands in its own slot's aggregation interval missed the interval it was due in, so it reads `after`, not `inside`. The dashed line is one interval (0.8s): above it the typical message misses the interval it was due in. Because the histogram observes the absolute delta, an arrival 300ms early shares a bucket with one 300ms late; the arrival position panel below is what separates them. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.8 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 27 + }, + "id": 201, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n sum by (le, job) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n sum by (le) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))\n)", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip attestation arrival delay (p50)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Absolute distance between arrival and the start of the interval the message was due in, per node. Aggregates anchor to the most recent aggregation-interval boundary at or before arrival, not to their own data slot (a stale-group catch-up aggregate can carry a data slot several slots back). The delay is therefore bounded below one slot and `before` is unreachable. Includes an aggregator's own locally produced aggregates. The dashed line is one interval (0.8s): above it the typical message misses the interval it was due in. Because the histogram observes the absolute delta, an arrival 300ms early shares a bucket with one 300ms late; the arrival position panel below is what separates them. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.8 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 27 + }, + "id": 202, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n sum by (le, job) (rate(lean_gossip_aggregation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n sum by (le) (rate(lean_gossip_aggregation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))\n)", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip aggregate arrival delay (p50)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Absolute distance between arrival and the start of the interval the message was due in, per node. Blocks anchor to interval 0 of their own slot. Only gossip arrivals are sampled; blocks fetched over req/resp during sync are excluded. The dashed line is one interval (0.8s): above it the slowest 1% of messages miss the interval they were due in. Because the histogram observes the absolute delta, an arrival 300ms early shares a bucket with one 300ms late; the arrival position panel below is what separates them. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.8 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 35 + }, + "id": 212, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n sum by (le, job) (rate(lean_gossip_block_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n sum by (le) (rate(lean_gossip_block_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))\n)", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip block arrival delay (p99)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Absolute distance between arrival and the start of the interval the message was due in, per node. Attestations anchor to interval 1 (AttestationProduction) of their data slot. An attestation that lands in its own slot's aggregation interval missed the interval it was due in, so it reads `after`, not `inside`. The dashed line is one interval (0.8s): above it the slowest 1% of messages miss the interval they were due in. Because the histogram observes the absolute delta, an arrival 300ms early shares a bucket with one 300ms late; the arrival position panel below is what separates them. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.8 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 35 + }, + "id": 213, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n sum by (le, job) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n sum by (le) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))\n)", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip attestation arrival delay (p99)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Absolute distance between arrival and the start of the interval the message was due in, per node. Aggregates anchor to the most recent aggregation-interval boundary at or before arrival, not to their own data slot (a stale-group catch-up aggregate can carry a data slot several slots back). The delay is therefore bounded below one slot and `before` is unreachable. Includes an aggregator's own locally produced aggregates. The dashed line is one interval (0.8s): above it the slowest 1% of messages miss the interval they were due in. Because the histogram observes the absolute delta, an arrival 300ms early shares a bucket with one 300ms late; the arrival position panel below is what separates them. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.8 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 35 + }, + "id": 214, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n sum by (le, job) (rate(lean_gossip_aggregation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n sum by (le) (rate(lean_gossip_aggregation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))\n)", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip aggregate arrival delay (p99)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Distribution of arrival delay across the selected nodes: the in-Prometheus analogue of a per-message beeswarm, which exposes the bimodal profile that percentiles average away. Blocks anchor to interval 0 of their own slot. Only gossip arrivals are sampled; blocks fetched over req/resp during sync are excluded. A band at the bottom is messages that made their interval; a second band higher up is the ones that did not.", + "fieldConfig": { + "defaults": { + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 43 + }, + "id": 203, + "options": { + "calculate": false, + "cellGap": 1, + "color": { + "exponent": 0.5, + "fill": "dark-orange", + "mode": "scheme", + "reverse": false, + "scale": "exponential", + "scheme": "Spectral", + "steps": 64 + }, + "exemplars": { + "color": "rgba(255,0,255,0.7)" + }, + "filterValues": { + "le": 1e-09 + }, + "legend": { + "show": true + }, + "rowsFrame": { + "layout": "auto", + "value": "messages/s" + }, + "showValue": "never", + "tooltip": { + "mode": "single", + "showColorScale": false, + "yHistogram": false + }, + "yAxis": { + "axisPlacement": "left", + "reverse": false, + "unit": "s" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (le) (rate(lean_gossip_block_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{le}}", + "range": true, + "refId": "A", + "format": "heatmap" + } + ], + "title": "Gossip block arrival delay distribution", + "type": "heatmap" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Distribution of arrival delay across the selected nodes: the in-Prometheus analogue of a per-message beeswarm, which exposes the bimodal profile that percentiles average away. Attestations anchor to interval 1 (AttestationProduction) of their data slot. An attestation that lands in its own slot's aggregation interval missed the interval it was due in, so it reads `after`, not `inside`. A band at the bottom is messages that made their interval; a second band higher up is the ones that did not.", + "fieldConfig": { + "defaults": { + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 43 + }, + "id": 204, + "options": { + "calculate": false, + "cellGap": 1, + "color": { + "exponent": 0.5, + "fill": "dark-orange", + "mode": "scheme", + "reverse": false, + "scale": "exponential", + "scheme": "Spectral", + "steps": 64 + }, + "exemplars": { + "color": "rgba(255,0,255,0.7)" + }, + "filterValues": { + "le": 1e-09 + }, + "legend": { + "show": true + }, + "rowsFrame": { + "layout": "auto", + "value": "messages/s" + }, + "showValue": "never", + "tooltip": { + "mode": "single", + "showColorScale": false, + "yHistogram": false + }, + "yAxis": { + "axisPlacement": "left", + "reverse": false, + "unit": "s" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (le) (rate(lean_gossip_attestation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{le}}", + "range": true, + "refId": "A", + "format": "heatmap" + } + ], + "title": "Gossip attestation arrival delay distribution", + "type": "heatmap" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Distribution of arrival delay across the selected nodes: the in-Prometheus analogue of a per-message beeswarm, which exposes the bimodal profile that percentiles average away. Aggregates anchor to the most recent aggregation-interval boundary at or before arrival, not to their own data slot (a stale-group catch-up aggregate can carry a data slot several slots back). The delay is therefore bounded below one slot and `before` is unreachable. Includes an aggregator's own locally produced aggregates. A band at the bottom is messages that made their interval; a second band higher up is the ones that did not.", + "fieldConfig": { + "defaults": { + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 43 + }, + "id": 205, + "options": { + "calculate": false, + "cellGap": 1, + "color": { + "exponent": 0.5, + "fill": "dark-orange", + "mode": "scheme", + "reverse": false, + "scale": "exponential", + "scheme": "Spectral", + "steps": 64 + }, + "exemplars": { + "color": "rgba(255,0,255,0.7)" + }, + "filterValues": { + "le": 1e-09 + }, + "legend": { + "show": true + }, + "rowsFrame": { + "layout": "auto", + "value": "messages/s" + }, + "showValue": "never", + "tooltip": { + "mode": "single", + "showColorScale": false, + "yHistogram": false + }, + "yAxis": { + "axisPlacement": "left", + "reverse": false, + "unit": "s" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (le) (rate(lean_gossip_aggregation_arrival_delay_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{le}}", + "range": true, + "refId": "A", + "format": "heatmap" + } + ], + "title": "Gossip aggregate arrival delay distribution", + "type": "heatmap" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Arrival rate across the selected nodes, split by where the message landed relative to the interval it was due in. The only view that recovers the sign the delay histogram discards: rising `before` means clock skew, rising `after` means propagation or CPU. `inside` is half-open, so a delta of exactly one interval reads `after`. Blocks anchor to interval 0 of their own slot. Only gossip arrivals are sampled; blocks fetched over req/resp during sync are excluded.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "before" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "inside" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "after" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 51 + }, + "id": 206, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (position) (rate(lean_gossip_block_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{position}}", + "range": true, + "refId": "A" + } + ], + "title": "Gossip block arrival position", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Arrival rate across the selected nodes, split by where the message landed relative to the interval it was due in. The only view that recovers the sign the delay histogram discards: rising `before` means clock skew, rising `after` means propagation or CPU. `inside` is half-open, so a delta of exactly one interval reads `after`. Attestations anchor to interval 1 (AttestationProduction) of their data slot. An attestation that lands in its own slot's aggregation interval missed the interval it was due in, so it reads `after`, not `inside`.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "before" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "inside" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "after" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 51 + }, + "id": 207, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (position) (rate(lean_gossip_attestation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{position}}", + "range": true, + "refId": "A" + } + ], + "title": "Gossip attestation arrival position", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Arrival rate across the selected nodes, split by where the message landed relative to the interval it was due in. The only view that recovers the sign the delay histogram discards: rising `before` means clock skew, rising `after` means propagation or CPU. `inside` is half-open, so a delta of exactly one interval reads `after`. Aggregates anchor to the most recent aggregation-interval boundary at or before arrival, not to their own data slot (a stale-group catch-up aggregate can carry a data slot several slots back). The delay is therefore bounded below one slot and `before` is unreachable. Includes an aggregator's own locally produced aggregates.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "inside" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "after" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 51 + }, + "id": 208, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (position) (rate(lean_gossip_aggregation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{position}}", + "range": true, + "refId": "A" + } + ], + "title": "Gossip aggregate arrival position", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Share of arrivals that landed inside the interval they were due in, per node. Blocks anchor to interval 0 of their own slot. Only gossip arrivals are sampled; blocks fetched over req/resp during sync are excluded. A node well below its peers is late on its own (clock skew or a starved CPU); the whole fleet dropping together is a chain-wide production or propagation problem. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 0.9 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 59 + }, + "id": 209, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (job) (rate(lean_gossip_block_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", position=\"inside\"}[$__rate_interval]))\n/\nsum by (job) (rate(lean_gossip_block_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum (rate(lean_gossip_block_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", position=\"inside\"}[15m]))\n/ sum (rate(lean_gossip_block_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip block on-time fraction", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Share of arrivals that landed inside the interval they were due in, per node. Attestations anchor to interval 1 (AttestationProduction) of their data slot. An attestation that lands in its own slot's aggregation interval missed the interval it was due in, so it reads `after`, not `inside`. A node well below its peers is late on its own (clock skew or a starved CPU); the whole fleet dropping together is a chain-wide production or propagation problem. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 0.9 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 59 + }, + "id": 210, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (job) (rate(lean_gossip_attestation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", position=\"inside\"}[$__rate_interval]))\n/\nsum by (job) (rate(lean_gossip_attestation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum (rate(lean_gossip_attestation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", position=\"inside\"}[15m]))\n/ sum (rate(lean_gossip_attestation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip attestation on-time fraction", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Share of arrivals that landed inside the interval they were due in, per node. Aggregates anchor to the most recent aggregation-interval boundary at or before arrival, not to their own data slot (a stale-group catch-up aggregate can carry a data slot several slots back). The delay is therefore bounded below one slot and `before` is unreachable. Includes an aggregator's own locally produced aggregates. A node well below its peers is late on its own (clock skew or a starved CPU); the whole fleet dropping together is a chain-wide production or propagation problem. The thick `fleet` line is the same statistic computed over the whole fleet at once (no per-node split, 15m rate window), smoothed: a trend baseline for the jagged per-node lines.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 0.9 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "fleet" + }, + "properties": [ + { + "id": "custom.lineWidth", + "value": 3 + }, + { + "id": "custom.lineInterpolation", + "value": "smooth" + }, + { + "id": "custom.spanNulls", + "value": true + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.showPoints", + "value": "never" + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 59 + }, + "id": 211, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (job) (rate(lean_gossip_aggregation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", position=\"inside\"}[$__rate_interval]))\n/\nsum by (job) (rate(lean_gossip_aggregation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum (rate(lean_gossip_aggregation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", position=\"inside\"}[15m]))\n/ sum (rate(lean_gossip_aggregation_arrival_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[15m]))", + "interval": "", + "legendFormat": "fleet", + "range": true, + "refId": "Z" + } + ], + "title": "Gossip aggregate on-time fraction", + "type": "timeseries" + } + ], + "title": "Gossip Arrival Timing", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 27 + }, + "id": 45, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of individual attestation signatures", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 35 + }, + "id": 60, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_attestation_signatures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Attestation signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of valid individual attestation signatures", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 35 + }, + "id": 64, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_attestation_signatures_valid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Valid attestation signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of invalid individual attestation signatures", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 35 + }, + "id": 65, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_attestation_signatures_invalid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Invalid attestation signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 43 + }, + "id": 46, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_attestation_signing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.50, rate(lean_pq_sig_attestation_signing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to sign an attestation", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 43 + }, + "id": 47, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_attestation_verification_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.50, rate(lean_pq_sig_attestation_verification_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to verify an attestation signature", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 51 + }, + "id": 79, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_aggregated_signatures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Aggregated signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 51 + }, + "id": 61, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_attestations_in_aggregated_signatures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Attestations included in aggregated signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "ALWAYS ZERO as of this build: the counter is registered but never incremented (inc_pq_sig_aggregated_signatures_valid/_invalid have no callers in crates/blockchain). Flat zero here means 'not instrumented', NOT 'none observed'.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 59 + }, + "id": 77, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_aggregated_signatures_valid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Valid aggregated signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "ALWAYS ZERO as of this build: the counter is registered but never incremented (inc_pq_sig_aggregated_signatures_valid/_invalid have no callers in crates/blockchain). Flat zero here means 'not instrumented', NOT 'none observed'.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 59 + }, + "id": 78, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_pq_sig_aggregated_signatures_invalid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Invalid aggregated signatures (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 67 + }, + "id": 62, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_aggregated_signatures_building_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.50, rate(lean_pq_sig_aggregated_signatures_building_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to build an aggregated signature", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 67 + }, + "id": 63, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_aggregated_signatures_verification_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.50, rate(lean_pq_sig_aggregated_signatures_verification_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to verify an aggregated signature", + "type": "timeseries" + } + ], + "title": "PQ Signatures", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 28 + }, + "id": 106, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 36 + }, + "id": 107, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (network, job) (lean_attestation_aggregate_coverage_validators{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", section=\"combined\", subnet=\"combined\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Aggregate coverage \u2014 validators (combined)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 36 + }, + "id": 108, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (network, job) (lean_attestation_aggregate_coverage_subnets{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", section=\"combined\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Aggregate coverage \u2014 subnets covered", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 36 + }, + "id": 109, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (section) (lean_attestation_aggregate_coverage_validators{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", subnet=\"combined\"})", + "interval": "", + "legendFormat": "{{section}}", + "range": true, + "refId": "A" + } + ], + "title": "Aggregate coverage \u2014 validators by section", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 44 + }, + "id": 110, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (direction) (lean_attestation_aggregate_coverage_diff_validators{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{direction}}", + "range": true, + "refId": "A" + } + ], + "title": "Coverage diff \u2014 block_only vs timely_only", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 44 + }, + "id": 111, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (reason) (rate(lean_aggregator_skipped_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{reason}}", + "range": true, + "refId": "A" + } + ], + "title": "Aggregations skipped, by reason (fleet /s)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 44 + }, + "id": 112, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum (rate(lean_aggregation_early_starts_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "early starts /s", + "range": true, + "refId": "A" + } + ], + "title": "Early aggregation starts (fleet /s)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 52 + }, + "id": 113, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_aggregation_early_start_lead_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_aggregation_early_start_lead_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Early aggregation start lead (p99)", + "type": "timeseries" + } + ], + "title": "Aggregation Coverage", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 29 + }, + "id": 91, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 37 + }, + "id": 19, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_block_aggregated_payloads_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_block_aggregated_payloads_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Number of aggregated payloads", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 37 + }, + "id": 93, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_block_building_payload_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_block_building_payload_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to build aggregated payloads", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 45 + }, + "id": 80, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_block_building_success_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Successful block builds (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 45 + }, + "id": 97, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_block_building_failures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Failed block builds (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 45 + }, + "id": 94, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_block_building_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_block_building_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to build a block", + "type": "timeseries" + } + ], + "title": "Block production", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 30 + }, + "id": 114, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 38 + }, + "id": 115, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_block_proposal_attestation_builds_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Attestation selection rounds (/s)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 38 + }, + "id": 116, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_block_proposal_attestation_data_selected_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_block_proposal_attestation_data_selected_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Distinct AttestationData in proposal body (p99)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 38 + }, + "id": 117, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_block_proposal_aggregates_selected_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_block_proposal_aggregates_selected_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Aggregates selected per proposal (p99)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 46 + }, + "id": 118, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_block_proposal_child_payloads_consumed_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Child payloads consumed (/s)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 46 + }, + "id": 119, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (phase, le) (rate(lean_block_proposal_attestation_build_phase_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])))", + "interval": "", + "legendFormat": "{{phase}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (phase, le) (rate(lean_block_proposal_attestation_build_phase_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])))", + "interval": "", + "legendFormat": "{{phase}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Proposal attestation build phase (p99, by phase)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 46 + }, + "id": 120, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_aggregated_proof_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_aggregated_proof_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Aggregated proof size (p99)", + "type": "timeseries" + } + ], + "title": "Block Proposal Internals", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 17, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process block in fork-choice", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 39 + }, + "id": 92, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_fork_choice_block_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_fork_choice_block_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Fork-choice block processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 39 + }, + "id": 68, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_fork_choice_reorgs_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Fork choice reorgs (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 39 + }, + "id": 69, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_fork_choice_reorg_depth_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_fork_choice_reorg_depth_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Depth of fork choice reorgs (in blocks)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 47 + }, + "id": 96, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": " avg_over_time (\n sum by (network, job, instance) (\n (lean_gossip_signatures{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[2m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of gossip signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 47 + }, + "id": 81, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n (lean_latest_new_aggregated_payloads{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[2m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of new aggregated payloads", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 47 + }, + "id": 82, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n (lean_latest_known_aggregated_payloads{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[1m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of known aggregated payloads", + "type": "timeseries" + } + ], + "title": "Fork-Choice", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 32 + }, + "id": 8, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 40 + }, + "id": 9, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_attestations_valid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "FC Valid attestations (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 40 + }, + "id": 10, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_attestations_invalid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "FC Invalid attestations (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 40 + }, + "id": 11, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_attestation_validation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_attestation_validation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "FC Attestations validation time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 48 + }, + "id": 67, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_safe_target_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Safe target slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of attestations processed in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 48 + }, + "id": 27, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job) (rate(lean_state_transition_attestations_processed_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) * 60", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "STF Processed attestations (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process attestations in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 48 + }, + "id": 28, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99,\n rate(lean_state_transition_attestations_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50,\n rate(lean_state_transition_attestations_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "STF Attestations processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 56 + }, + "id": 83, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_attestations_production_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_attestations_production_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to produce attestation", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 56 + }, + "id": 84, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_committee_signatures_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_committee_signatures_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Time taken to aggregate committee signatures", + "type": "timeseries" + } + ], + "title": "Attestations", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 33 + }, + "id": 21, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 41 + }, + "id": 23, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_state_transition_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_state_transition_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "State transition time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process block in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 41 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_state_transition_block_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, \n rate(lean_state_transition_block_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "STF block processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of processed slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 49 + }, + "id": 25, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_state_transition_slots_processed_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))[5m:]\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Processed slots", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 49 + }, + "id": 26, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99,\n rate(lean_state_transition_slots_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50,\n rate(lean_state_transition_slots_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Slots processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of processed slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 57 + }, + "id": 70, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_finalizations_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", result=\"success\"}[$__rate_interval])) * 60", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Finalizations - Success (per minute)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cpm" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 57 + }, + "id": 71, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job) (rate(lean_finalizations_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", result=\"error\"}[$__rate_interval])) * 60", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Finalizations - Errors (per minute)", + "type": "timeseries" + } + ], + "title": "State Transition", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 34 + }, + "id": 125, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 42 + }, + "id": 126, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (table) (lean_table_bytes{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{table}}", + "range": true, + "refId": "A" + } + ], + "title": "Storage size by table", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*p50.*" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 42 + }, + "id": 127, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_tick_interval_duration_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p99", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, rate(lean_tick_interval_duration_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} p50", + "range": true, + "refId": "B" + } + ], + "title": "Tick interval duration (p99)", + "type": "timeseries" + } + ], + "title": "Storage & Tick Health", + "type": "row" + } + ], + "preload": false, + "schemaVersion": 42, + "tags": [ + "interop", + "Client" + ], + "templating": { + "list": [ + { + "allowCustomValue": false, + "current": {}, + "definition": "label_values(up{type=\"app\"}, network)", + "label": "Network", + "name": "network", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(up{type=\"app\"}, network)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query" + }, + { + "allowCustomValue": false, + "current": {}, + "definition": "label_values(lean_node_info{network=~\"$network\"}, job)", + "includeAll": true, + "label": "Job", + "multi": true, + "name": "job", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(lean_node_info{network=~\"$network\"}, job)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": ".*ethlambda.*|.*gean.*|.*grandine.*|.*lantern.*|.*lighthouse.*|.*qlean.*|.*ream.*|.*zeam.*", + "type": "query" + }, + { + "allowCustomValue": false, + "current": {}, + "definition": "label_values(lean_node_info{network=~\"$network\", job=~\"$job\"}, instance)", + "includeAll": true, + "label": "host", + "multi": true, + "name": "instance", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(lean_node_info{network=~\"$network\", job=~\"$job\"}, instance)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query", + "description": "Scrape target = the SERVER, not the node: every node on a host shares one instance value. Use $job to pick a node." + }, + { + "allowCustomValue": false, + "current": { + "text": "", + "value": "${datasource}", + "selected": true + }, + "name": "datasource", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + } + ] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "utc", + "title": "Lean Ethereum Clients Dashboard", + "uid": "lean-ethereum-clients-dashboard", + "version": 35, + "weekStart": "", + "id": null, + "refresh": "30s" +} diff --git a/.claude/skills/multi-server-devnet/scripts/convert.sh b/.claude/skills/multi-server-devnet/scripts/convert.sh new file mode 100755 index 00000000..934bb47c --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/convert.sh @@ -0,0 +1,189 @@ +#!/bin/bash +# Convert nodes of an INDEPENDENT single-host devnet to other lean clients +# (rolling, checkpoint sync, 60s gossip backoff). devnet5 images. +# Node identity stays node_N (node-id, node_N.key, data-dir — genesis keys off it); +# only the container name changes to _N to reflect the running client. +# Ports per node n: gossip 9000+n, api 5052+n, metrics 9200+n (no BASE offset). +# ACC (this devnet's SUBNETS / ATTESTATION_COMMITTEE_COUNT) is read from the +# genesis on disk; set ACC= only to override it. A wrong ACC gives the node a +# different subnet map than the chain, so its votes go to a subnet nobody +# aggregates -- never guess it. +# An :agg node covers the single subnet n % ACC (qlean auto-derives, no flag). +# Specs are validated BEFORE anything is removed: a typo used to be discovered +# after the container and its data were already gone, leaving the node down. +# Args: CS_PORT tuple... tuple = N:client[:agg] client in zeam|ream|qlean|gean|lantern|grandine +set -u +GENESIS=${GENESIS:-/opt/lean-quickstart/genesis} +DATA=${DATA:-/opt/lean-quickstart/data} +CRASH=${CRASH:-/opt/lean-quickstart/crash-logs} +LOG=/tmp/convert.log +TS=$(date +%Y%m%d-%H%M%S) +CLIENTS="zeam ream qlean gean lantern grandine" +ZEAM_IMG=${ZEAM_IMG:-blockblaz/zeam:devnet5} +REAM_IMG=${REAM_IMG:-ghcr.io/reamlabs/ream:latest-devnet5} +QLEAN_IMG=${QLEAN_IMG:-qdrvm/qlean-mini:devnet-5-amd64} +LANTERN_IMG=${LANTERN_IMG:-bitminemavan/lantern:devnet5} +GEAN_IMG=${GEAN_IMG:-ghcr.io/geanlabs/gean:devnet5} +GRANDINE_IMG=${GRANDINE_IMG:-sifrai/lean:devnet-5} +MEMLIMITS=${MEMLIMITS:-"--memory 8g --memory-swap 16g --memory-reservation 2g"} # override per client if needed +# Cap Docker json-file stdout logs. 200m x 5 = 1 GiB/container; central Loki (7d) holds +# the long-term record, so this is just a hot buffer for `docker logs` (see start-devnet.sh). +# EVERY client run must carry this: the default json-file driver is unbounded and has +# filled a host disk before. +LOGOPT=${LOGOPT:-"--log-opt max-size=200m --log-opt max-file=5"} + +CS=$1; shift +log(){ echo "$(date '+%H:%M:%S') $*" | tee -a "$LOG"; } + +: > "$LOG" +# ACC must match the chain. Read it from the genesis this host is running unless +# the caller overrode it. +if [ -z "${ACC:-}" ]; then + ACC=$(sudo awk '/^ATTESTATION_COMMITTEE_COUNT:/{print $2}' "$GENESIS/config.yaml" 2>/dev/null) +fi +case "${ACC:-}" in + ''|*[!0-9]*|0) log "ERROR: ACC unknown (no ATTESTATION_COMMITTEE_COUNT in $GENESIS/config.yaml) -- pass ACC="; exit 1;; +esac + +# Validate every spec before touching a container: an unknown client or a +# non-numeric index used to surface only after the node was already destroyed. +for spec in "$@"; do + n=${spec%%:*}; rest=${spec#*:}; client=${rest%%:*} + case "$n" in ''|*[!0-9]*) log "ERROR: bad node index in spec '$spec'"; exit 1;; esac + case " $CLIENTS " in *" $client "*) ;; *) log "ERROR: unknown client '$client' in spec '$spec' (want: $CLIENTS)"; exit 1;; esac + case "$rest" in "$client"|"$client:agg") ;; *) log "ERROR: bad spec '$spec' (want N:client[:agg])"; exit 1;; esac +done + +convert_node(){ # $1 = N:client[:agg] + local spec=$1 + local n=${spec%%:*} rest=${spec#*:} + local client=${rest%%:*} agg=no + [[ "$rest" == *:agg ]] && agg=yes + local name=node_$n # node identity: node-id, node_N.key, data-dir (client-agnostic) + local cname=${client}_$n # container / display name reflecting the running client + local G=$((9000+n)) A=$((5052+n)) M=$((9200+n)) + local CSURL="http://127.0.0.1:$CS/lean/v0/states/finalized" + # qlean rejects --aggregate-subnet-ids; aggregators here cover one subnet each + local aggflags="" + if [ "$agg" = yes ]; then + aggflags="--is-aggregator" + [ "$client" != qlean ] && aggflags="$aggflags --aggregate-subnet-ids $((n % ACC))" + fi + + # every container for this index (its client prefix may differ); keep one crash + # log each -- converting away from a wedged canary is exactly when you want it + local cids="" cid cnm + while read -r cid cnm; do + [ -n "$cid" ] || continue + cids="${cids:+$cids }$cid" + # Warn if we are removing this devnet's aggregator for the subnet without + # putting one back: the subnet's votes then reach no aggregator and finality + # quietly drops below ceil(2/3 . NODES). + if [ "$agg" = no ] && sudo docker inspect -f '{{join .Config.Cmd " "}}' "$cid" 2>/dev/null \ + | grep -q -- '--is-aggregator'; then + log "WARNING: $cnm was an AGGREGATOR and the new spec has no :agg -- subnet $((n % ACC)) loses its aggregator" + fi + cnm=$(printf '%s' "${cnm:-$cid}" | tr -c 'A-Za-z0-9_.-' '_') + sudo sh -c "docker logs '$cid' > '$CRASH/${cnm}-${TS}.log' 2>&1" \ + && log "saved $CRASH/${cnm}-${TS}.log" + done < <(sudo docker ps -a --filter "name=_${n}$" --format '{{.ID}} {{.Names}}') + local existed=no; [ -n "$cids" ] && existed=yes + [ "$existed" = yes ] && sudo docker rm -f $cids >/dev/null + # wipe the dir itself, not `dir/*` (that glob is expanded unprivileged, so an + # unreadable dir would silently leave the previous client's data in place -- + # wrong on-disk format for the new client). Recreate for the bind mount. + sudo rm -rf "$DATA/$name"; sudo mkdir -p "$DATA/$name" + if [ "$existed" = yes ]; then log "stopped node_$n container(s); 60s gossip backoff"; sleep 60; fi + + case "$client" in + zeam) + # console_log_level=info exposes zeam's own duty publishes (attestation / + # block) on stdout so the duty-timing profiler (reads `docker logs`) can see + # them; the file sink stays warn (not captured). That extra volume is exactly + # why $LOGOPT must be here -- without it this is the one unbounded json log. + sudo docker run -d --restart unless-stopped --name "$cname" --network host \ + --security-opt seccomp=unconfined $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data \ + "$ZEAM_IMG" \ + --log_file_active_level warn --console_log_level info node \ + --custom-genesis /config --validator-config genesis_bootnode \ + --data-dir /data --node-id "$name" --node-key /config/"$name".key \ + --metrics-enable --api-port "$A" --metrics-port "$M" \ + --attestation-committee-count "$ACC" $aggflags \ + --checkpoint-sync-url "$CSURL" --db-backend lmdb >/dev/null + ;; + ream) + sudo docker run -d --restart unless-stopped --name "$cname" --network host \ + $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data \ + "$REAM_IMG" --data-dir /data lean_node \ + --network /config/config.yaml \ + --validator-registry-path /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml \ + --node-id "$name" --node-key /config/"$name".key --socket-port "$G" \ + --metrics --metrics-address 0.0.0.0 --metrics-port "$M" \ + --http-address 0.0.0.0 --http-port "$A" \ + --attestation-committee-count "$ACC" $aggflags \ + --checkpoint-sync-url "$CSURL" >/dev/null + ;; + gean) + sudo docker run -d --restart unless-stopped --name "$cname" --network host \ + $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data \ + "$GEAN_IMG" \ + --custom-network-config-dir /config --data-dir /data \ + --gossipsub-port "$G" --node-id "$name" --node-key /config/"$name".key \ + --http-address 0.0.0.0 --api-port "$A" --metrics-port "$M" \ + --attestation-committee-count "$ACC" $aggflags \ + --checkpoint-sync-url "$CSURL" >/dev/null + ;; + qlean) + sudo docker run -d --restart unless-stopped --name "$cname" --network host \ + $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data \ + "$QLEAN_IMG" \ + --genesis-dir /config --data-dir /data \ + --node-id "$name" --node-key /config/"$name".key \ + --listen-addr /ip4/0.0.0.0/udp/"$G"/quic-v1 \ + --metrics-host 0.0.0.0 --metrics-port "$M" \ + --api-host 0.0.0.0 --api-port "$A" \ + --attestation-committee-count "$ACC" $aggflags \ + --checkpoint-sync-url "$CSURL" -linfo >/dev/null + ;; + lantern) + sudo docker run -d --restart unless-stopped --name "$cname" --network host \ + $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data \ + "$LANTERN_IMG" \ + --data-dir /data --genesis-config /config/config.yaml \ + --nodes-path /config/nodes.yaml --genesis-state /config/genesis.ssz \ + --validator_config /config \ + --node-id "$name" --node-key-path /config/"$name".key \ + --listen-address /ip4/0.0.0.0/udp/"$G"/quic-v1 \ + --http-port "$A" --metrics-port "$M" \ + --hash-sig-key-dir /config/hash-sig-keys \ + --attestation-committee-count "$ACC" --devnet 12345678 $aggflags \ + --checkpoint-sync-url "$CSURL" >/dev/null + ;; + grandine) + sudo docker run -d --restart unless-stopped --name "$cname" --network host \ + $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data \ + "$GRANDINE_IMG" \ + --genesis /config/config.yaml --validator-registry-path /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml --node-id "$name" --node-key /config/"$name".key \ + --port "$G" --address 0.0.0.0 --http-address 0.0.0.0 --http-port "$A" \ + --metrics --metrics-address 0.0.0.0 --metrics-port "$M" \ + --hash-sig-key-dir /config/hash-sig-keys \ + --attestation-committee-count "$ACC" $aggflags \ + --checkpoint-sync-url "$CSURL" >/dev/null + ;; + *) log "ERROR unknown client '$client' for $name"; return 1;; + esac + log "converted $name -> $client$([ "$agg" = yes ] && echo ' [AGGREGATOR]')" +} + +sudo mkdir -p "$CRASH" +log "=== convert start cs_port=$CS acc=$ACC specs=[$*] ===" +for spec in "$@"; do convert_node "$spec"; done +log "=== convert done ===" diff --git a/.claude/skills/multi-server-devnet/scripts/cs-restart.sh b/.claude/skills/multi-server-devnet/scripts/cs-restart.sh new file mode 100755 index 00000000..8e754b2f --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/cs-restart.sh @@ -0,0 +1,147 @@ +#!/bin/bash +# Checkpoint-sync restart of ethlambda nodes on ONE devnet's host, in place. +# Recover crashed/frozen/split nodes, canary an image, or set the aggregator role +# WITHOUT resetting the chain. This is the ONLY restart path: a separate script +# per role drifts out of the one used every day, which is how the role handling +# got lost in the first place. +# +# ALWAYS uses checkpoint sync (a keep-DB resume can panic the consensus actor on +# older images -- `store.rs` `expect("safe target exists")` -- leaving the process +# up with consensus dead). Keeps the mandatory 60s gossip backoff for nodes that +# were running, so they re-graft cleanly (skip it and they attest but aggregators +# never hear them -> finality stalls below the 2/3 threshold). Saves one crash log +# per removed container. +# Ports per node n: gossip 9000+n, api 5052+n, metrics 9200+n (no BASE offset). +# +# THE AGGREGATOR ROLE IS PRESERVED BY DEFAULT. A node restarted without +# --is-aggregator stops storing gossip signatures, so its subnet's votes are never +# aggregated: attestations still verify, the logs look healthy, and the devnet +# quietly stops finalizing with attestation_count=0. So the topology flags +# (--is-aggregator / --aggregate-subnet-ids / --attestation-committee-count) are +# read off the container being replaced and passed through unless you ask +# otherwise. Experiment flags are deliberately NOT carried over -- a restart is +# where an experiment ends. +# +# Usage (on host): cs-restart.sh ... +# CS_PORT api port of a HEALTHY same-devnet node to checkpoint-sync from +# (an aggregator stays up -- use it; but do NOT point an aggregator at +# itself). Comma-separate several for redundancy: ethlambda tries them +# in order and only aborts if every one fails. +# node... node indices to restart, one at a time per host +# +# Env: +# IMAGE image to run (default ghcr.io/lambdaclass/ethlambda:devnet5). Set it +# to canary a PR build on specific nodes, aggregators included. +# AGG role control: +# unset preserve the role the container already has [default] +# auto apply the topology rule (node n aggregates subnet n while +# n < SUBNETS, otherwise non-aggregator). Needs SUBNETS. +# Use this to repair a node that lost its role. +# force --is-aggregator on subnet (promote). With more +# than one node this puts them ALL on -- usually you +# want AGG=auto instead. +# off force non-aggregator (deliberate demotion). +# SUBNETS the devnet's committee count. Authoritative for +# --attestation-committee-count, drives AGG=auto, and is the fallback +# for the role when the container is already gone (nothing to read). +# GENESIS, DATA, CRASH, MEMLIMITS, LOGOPT +set -u +CS=$1; shift; NODES="$*" +IMAGE=${IMAGE:-ghcr.io/lambdaclass/ethlambda:devnet5} +GENESIS=${GENESIS:-/opt/lean-quickstart/genesis} +DATA=${DATA:-/opt/lean-quickstart/data} +MEMLIMITS=${MEMLIMITS:-"--memory 8g --memory-swap 16g --memory-reservation 2g"} +LOGOPT=${LOGOPT:-"--log-opt max-size=200m --log-opt max-file=5"} # 1 GiB hot buffer; Loki (7d) holds the long-term record +CRASH=${CRASH:-/opt/lean-quickstart/crash-logs} +LOG=/tmp/cs-restart.log +TS=$(date +%Y%m%d-%H%M%S) +log(){ echo "$(date '+%H:%M:%S') $*" | tee -a "$LOG"; } + +: > "$LOG" +# Validate everything before touching a container: discovering a bad argument +# halfway through leaves nodes destroyed and not restarted. +for n in $NODES; do + case "$n" in ''|*[!0-9]*) log "ERROR: bad node index '$n'"; exit 1;; esac +done +case "${AGG:-}" in + ''|off|auto) ;; + *[!0-9,]*) log "ERROR: AGG must be a subnet id, 'auto', or 'off' (got '$AGG')"; exit 1;; +esac +if [ "${AGG:-}" = auto ] && [ -z "${SUBNETS:-}" ]; then + log "ERROR: AGG=auto needs SUBNETS= to know which nodes aggregate"; exit 1 +fi + +# One URL per CS_PORT; ethlambda accepts a comma-separated list and falls over to +# the next when a peer is unreachable. +CSURL="" +for p in ${CS//,/ }; do + CSURL="${CSURL:+$CSURL,}http://127.0.0.1:$p/lean/v0/states/finalized" +done + +arg_value(){ # $1 = flag, $2 = full command line -> the token after the flag + awk -v f="$1" '{for(i=1;i<=NF;i++) if($i==f) print $(i+1)}' <<<"$2" | head -1 +} +resolve_role(){ # $1 = container id ("" if none), $2 = node index -> topology flags + local cmd="" acc="" sub="" agg=no + # ENTRYPOINT is the binary, so the node's run args are .Config.Cmd + [ -n "$1" ] && cmd=$(sudo docker inspect -f '{{join .Config.Cmd " "}}' "$1" 2>/dev/null) + if [ -n "$cmd" ]; then + acc=$(arg_value --attestation-committee-count "$cmd") + sub=$(arg_value --aggregate-subnet-ids "$cmd") + case " $cmd " in + *" --is-aggregator=false "*) agg=no ;; + *" --is-aggregator "*|*" --is-aggregator=true "*) agg=yes ;; + esac + elif [ -n "${SUBNETS:-}" ] && [ "$2" -lt "$SUBNETS" ]; then + agg=yes; sub=$2 # nothing to read: fall back to the rule + fi + [ -n "${SUBNETS:-}" ] && acc=$SUBNETS # explicit SUBNETS wins over the old value + case "${AGG:-}" in + off) agg=no ;; + auto) if [ "$2" -lt "$SUBNETS" ]; then agg=yes; sub=$2; else agg=no; fi ;; + ?*) agg=yes; sub=$AGG ;; + esac + # --aggregate-subnet-ids requires --is-aggregator (clap refuses to start otherwise) + [ "$agg" = yes ] || sub="" + printf '%s%s%s' "${acc:+ --attestation-committee-count $acc}" \ + "${sub:+ --aggregate-subnet-ids $sub}" \ + "$([ "$agg" = yes ] && echo ' --is-aggregator')" +} + +sudo mkdir -p "$CRASH" +log "=== cs-restart cs=[$CS] nodes=[$NODES] image=$IMAGE role=${AGG:-preserve} ===" +for n in $NODES; do + name=node_$n; cname=ethlambda_$n; G=$((9000+n)); A=$((5052+n)); M=$((9200+n)) + # A node index can have several containers (e.g. a stopped canary left over from + # a conversion plus the running one) -- they all end in _$n. One crash log each: + # passing several ids to `docker logs` (or letting the id list keep its newline + # inside `sh -c "... > file"`) loses the log, which is the one artifact worth + # having when debugging a crash. + cids="" + while read -r cid cnm; do + [ -n "$cid" ] || continue + cids="${cids:+$cids }$cid" + cnm=$(printf '%s' "${cnm:-$cid}" | tr -c 'A-Za-z0-9_.-' '_') + sudo sh -c "docker logs '$cid' > '$CRASH/${cnm}-${TS}.log' 2>&1" \ + && log "saved $CRASH/${cnm}-${TS}.log" + done < <(sudo docker ps -a --filter "name=_${n}$" --format '{{.ID}} {{.Names}}') + # decide the role from the most recent container BEFORE removing anything + role=$(resolve_role "${cids%% *}" "$n") + existed=no; [ -n "$cids" ] && existed=yes + [ -n "$cids" ] && sudo docker rm -f $cids >/dev/null 2>&1 + # Wipe the whole dir, not `dir/*`: the glob is expanded by THIS (unprivileged) + # shell, so an unreadable dir would silently leave the stale DB in place and the + # node would come back on it. Recreate immediately -- docker must find it there. + sudo rm -rf "$DATA/$name"; sudo mkdir -p "$DATA/$name" + if [ "$existed" = yes ]; then log "stopped node_$n container(s); 60s gossip backoff"; sleep 60; fi + sudo docker run -d --restart unless-stopped --name "$cname" --network host $MEMLIMITS $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$name":/data "$IMAGE" \ + --genesis /config/config.yaml --validators /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml --validator-config /config/validator-config.yaml \ + --hash-sig-keys-dir /config/hash-sig-keys --data-dir /data \ + --gossipsub-port "$G" --node-id "$name" --node-key /config/"$name".key \ + --http-address 0.0.0.0 --api-port "$A" --metrics-port "$M" $role \ + --checkpoint-sync-url "$CSURL" >/dev/null + log "started $cname (node_$n)${role:+ [$role]} (checkpoint sync from $CS)" +done +log "=== cs-restart done ===" diff --git a/.claude/skills/multi-server-devnet/scripts/deploy-finality-alert.sh b/.claude/skills/multi-server-devnet/scripts/deploy-finality-alert.sh new file mode 100755 index 00000000..637b2826 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/deploy-finality-alert.sh @@ -0,0 +1,76 @@ +#!/bin/bash +# Deploy the per-devnet "lost finality" Grafana alert to the central metrics host. +# +# Renders grafana-finality-alert.yaml.template (Slack webhook + Grafana base URL + +# central-prometheus datasource uid), pushes it into Grafana's provisioning/alerting +# dir on the metrics host, and reloads Grafana. The rendered file (with the real +# webhook) is written to a temp file and removed locally; only the host keeps a copy. +# +# Nothing about the deployment is hardcoded: every host-specific value comes from the +# env. Unlike the dashboards, a provisioned ALERT RULE cannot use a datasource +# template variable — it needs a concrete uid — hence PROM_DS_UID. +# +# The five values below normally come from devnet.env (see devnet.env.example) so +# they don't have to be retyped; anything exported in the shell overrides the file. +# +# Usage: +# deploy-finality-alert.sh [WEBHOOK_FILE] # values from devnet.env +# METRICS_HOST=user@host GRAFANA_PROV_DIR=/path/to/grafana-provisioning \ +# GRAFANA_CONTAINER= GRAFANA_BASE_URL=http://host:3000 \ +# PROM_DS_UID= \ +# deploy-finality-alert.sh [WEBHOOK_FILE] # or fully explicit +# +# WEBHOOK_FILE file holding the Slack webhook url (default ./webhook.txt, or +# WEBHOOK_FILE from devnet.env). It is a SECRET — keep it out of git +# (the repo .gitignore covers webhook.txt). +# DRY_RUN=1 render + validate locally and print the plan; ship nothing. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$SCRIPT_DIR/devnet-env.sh" +devnet_load_env || exit 1 + +METRICS_HOST=${METRICS_HOST:?ssh target of the central metrics host, e.g. user@host} +GRAFANA_PROV_DIR=${GRAFANA_PROV_DIR:?grafana provisioning dir on the metrics host} +GRAFANA_CONTAINER=${GRAFANA_CONTAINER:?grafana container name on the metrics host} +GRAFANA_BASE_URL=${GRAFANA_BASE_URL:?grafana base url for the alert dashboard link, e.g. http://host:3000} +PROM_DS_UID=${PROM_DS_UID:?grafana datasource uid of the central prometheus} +WEBHOOK_FILE="${1:-${WEBHOOK_FILE:-webhook.txt}}" + +TEMPLATE="$SCRIPT_DIR/grafana-finality-alert.yaml.template" +REMOTE_ALERT_DIR="${GRAFANA_PROV_DIR%/}/alerting" + +[[ -f "$WEBHOOK_FILE" ]] || { echo "webhook file not found: $WEBHOOK_FILE" >&2; exit 1; } +[[ -f "$TEMPLATE" ]] || { echo "template not found: $TEMPLATE" >&2; exit 1; } + +# Render without echoing the secret. `|` delimiter avoids the URL's `/` chars. +WEBHOOK="$(tr -d '\n\r' < "$WEBHOOK_FILE")" +RENDERED="$(mktemp)" +trap 'rm -f "$RENDERED"' EXIT +sed -e "s|__SLACK_WEBHOOK_URL__|${WEBHOOK}|" \ + -e "s|__GRAFANA_BASE_URL__|${GRAFANA_BASE_URL%/}|" \ + -e "s|__PROM_DS_UID__|${PROM_DS_UID}|g" "$TEMPLATE" > "$RENDERED" + +# Fail before touching the host if a placeholder survived (renders as a broken +# provisioning file that Grafana silently ignores). +if grep -q '__[A-Z_]*__' "$RENDERED"; then + echo "unsubstituted placeholder(s) left in the rendered alert:" >&2 + grep -o '__[A-Z_]*__' "$RENDERED" | sort -u >&2 + exit 1 +fi + +if [[ -n "${DRY_RUN:-}" ]]; then + echo "DRY_RUN: would install as $REMOTE_ALERT_DIR/finality-alerts.yaml on $METRICS_HOST" + echo "DRY_RUN: would restart container $GRAFANA_CONTAINER" + # webhook redacted: the point of the dry run is the structure, not the secret + sed "s|${WEBHOOK}||" "$RENDERED" + exit 0 +fi + +echo "Pushing finality alert to $METRICS_HOST ..." +scp -q "$RENDERED" "$METRICS_HOST:/tmp/finality-alerts.yaml" +ssh "$METRICS_HOST" "sudo mkdir -p '$REMOTE_ALERT_DIR' \ + && sudo mv /tmp/finality-alerts.yaml '$REMOTE_ALERT_DIR/finality-alerts.yaml' \ + && sudo chmod 644 '$REMOTE_ALERT_DIR/finality-alerts.yaml' \ + && sudo docker restart '$GRAFANA_CONTAINER' >/dev/null \ + && echo 'Grafana restarted; alert provisioned.'" diff --git a/.claude/skills/multi-server-devnet/scripts/devnet-env.sh b/.claude/skills/multi-server-devnet/scripts/devnet-env.sh new file mode 100644 index 00000000..2ba9c8fc --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/devnet-env.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# Sourced helper (not executed): load the deployment's operator-specific values from +# an env file so they don't have to be retyped on every invocation. +# +# . "$SCRIPT_DIR/devnet-env.sh"; devnet_load_env +# +# Lookup order, first hit wins: $DEVNET_ENV, ./devnet.env, /devnet.env. +# Copy devnet.env.example -> devnet.env and fill it in (devnet.env is gitignored, +# since it names your hosts and may point at a webhook file). +# +# A value already set in the environment WINS over the file, so a one-off +# `PROM_DS_UID=other ./deploy-finality-alert.sh` still overrides. +# +# Only simple `KEY=value` / `export KEY=value` lines are read. Blank lines and `#` +# comments are skipped, as is a `# comment` tail following an UNQUOTED value; a +# quoted value keeps its contents verbatim, `#` included. Leading indentation is +# fine. A line that looks like an assignment but whose name isn't usable is +# reported on stderr rather than dropped in silence, because a config line that +# goes unread is how you deploy against the wrong deployment. +devnet_load_env() { + local dir file line key val + dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + file="" + if [ -n "${DEVNET_ENV:-}" ]; then + # An explicit path that doesn't exist is a typo, not a reason to silently fall + # back to some other devnet.env and deploy against the wrong deployment. + [ -f "$DEVNET_ENV" ] || { echo "DEVNET_ENV=$DEVNET_ENV does not exist" >&2; return 1; } + file=$DEVNET_ENV + else + for candidate in "./devnet.env" "$dir/devnet.env"; do + [ -f "$candidate" ] && { file=$candidate; break; } + done + fi + [ -n "$file" ] || return 0 + + while IFS= read -r line || [ -n "$line" ]; do + line=${line#"${line%%[![:space:]]*}"} # ltrim, so an indented line is read + case "$line" in ''|'#'*) continue ;; esac + case "$line" in *=*) ;; *) continue ;; esac # not an assignment at all + case "$line" in export[[:space:]]*) + line=${line#export}; line=${line#"${line%%[![:space:]]*}"} ;; + esac + key=${line%%=*} + val=${line#*=} + # Must be a name the shell can assign: a leading digit or any punctuation makes + # `export` fail with its own confusing error, and skipping in silence would hide + # the operator's typo. Name it and move on. + case "$key" in ''|[0-9]*|*[!A-Za-z0-9_]*) + echo "$file: ignoring '$key=...', not a variable name" >&2; continue ;; + esac + # Quotes, when present, DELIMIT the value, so a '#' or a trailing space inside + # them survives. Unquoted, a `# comment` tail is dropped: it used to end up + # inside the value, and for SERVERS that means sweep.sh ssh-ing to a host '#'. + case $val in + \"*) val=${val#\"}; val=${val%%\"*} ;; + \'*) val=${val#\'}; val=${val%%\'*} ;; + *) val=${val%%[[:space:]]#*} # comment tail + val=${val%"${val##*[![:space:]]}"} ;; # rtrim what it left behind + esac + # File provides defaults only: keep an already-set value. No eval anywhere -- + # `${!key}` is bash indirect expansion, and `export` gets ONE quoted + # name=value word, so a multi-word value (SERVERS="host-a host-b") arrives + # whole and nothing in it is ever reparsed as shell. + [ -n "${!key:-}" ] || export "$key=$val" + done < "$file" + echo "loaded deployment env from $file" >&2 +} diff --git a/.claude/skills/multi-server-devnet/scripts/devnet-overview-dashboard.json b/.claude/skills/multi-server-devnet/scripts/devnet-overview-dashboard.json new file mode 100644 index 00000000..3558a7c4 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/devnet-overview-dashboard.json @@ -0,0 +1,608 @@ +{ + "uid": "devnet-finality-overview", + "title": "Devnet Finality Overview", + "tags": [ + "lean", + "devnet" + ], + "timezone": "browser", + "schemaVersion": 39, + "refresh": "10s", + "time": { + "from": "now-30m", + "to": "now" + }, + "templating": { + "list": [ + { + "name": "ds_prom", + "label": "Prometheus", + "type": "datasource", + "query": "prometheus", + "refresh": 1, + "current": {}, + "options": [] + }, + { + "name": "network", + "label": "Devnet", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "query": { + "qryType": 1, + "query": "label_values(lean_head_slot, network)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "definition": "label_values(lean_head_slot, network)", + "refresh": 2, + "sort": 1, + "includeAll": true, + "allValue": ".*", + "multi": true, + "current": { + "text": [ + "All" + ], + "value": [ + "$__all" + ] + }, + "options": [] + } + ] + }, + "panels": [ + { + "id": 3, + "type": "stat", + "title": "Finalized slot", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 0 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_latest_finalized_slot{network=~\"$network\"})", + "instant": true, + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "transformations": [ + { + "id": "seriesToRows", + "options": {} + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "Metric" + } + ] + } + } + ], + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^Value$/", + "values": true + }, + "orientation": "auto", + "textMode": "value_and_name", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 2, + "type": "stat", + "title": "Justified slot", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 0 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_latest_justified_slot{network=~\"$network\"})", + "instant": true, + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "transformations": [ + { + "id": "seriesToRows", + "options": {} + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "Metric" + } + ] + } + } + ], + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^Value$/", + "values": true + }, + "orientation": "auto", + "textMode": "value_and_name", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 1, + "type": "stat", + "title": "Head slot", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 0 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_head_slot{network=~\"$network\"})", + "instant": true, + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "transformations": [ + { + "id": "seriesToRows", + "options": {} + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "Metric" + } + ] + } + } + ], + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^Value$/", + "values": true + }, + "orientation": "auto", + "textMode": "value_and_name", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 6, + "type": "timeseries", + "title": "Finalized slot", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 7 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_latest_finalized_slot{network=~\"$network\"})", + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 5, + "showPoints": "never" + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 5, + "type": "timeseries", + "title": "Justified slot", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 7 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_latest_justified_slot{network=~\"$network\"})", + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 5, + "showPoints": "never" + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 4, + "type": "timeseries", + "title": "Head slot", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 7 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_head_slot{network=~\"$network\"})", + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 5, + "showPoints": "never" + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 7, + "type": "timeseries", + "title": "Justified − Finalized", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 16 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_latest_justified_slot{network=~\"$network\"}) - max by (network) (lean_latest_finalized_slot{network=~\"$network\"})", + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 5, + "showPoints": "never" + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 8, + "type": "timeseries", + "title": "Head − Justified", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 16 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_head_slot{network=~\"$network\"}) - max by (network) (lean_latest_justified_slot{network=~\"$network\"})", + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 5, + "showPoints": "never" + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + }, + { + "id": 9, + "type": "timeseries", + "title": "Head − Finalized", + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 16 + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${ds_prom}" + }, + "expr": "max by (network) (lean_head_slot{network=~\"$network\"}) - max by (network) (lean_latest_finalized_slot{network=~\"$network\"})", + "legendFormat": "{{network}}", + "refId": "A" + } + ], + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 5, + "showPoints": "never" + }, + "unit": "none", + "decimals": 0 + }, + "overrides": [] + } + } + ] +} diff --git a/.claude/skills/multi-server-devnet/scripts/devnet.env.example b/.claude/skills/multi-server-devnet/scripts/devnet.env.example new file mode 100644 index 00000000..ee6a4405 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/devnet.env.example @@ -0,0 +1,34 @@ +# Deployment-specific values for the multi-server-devnet scripts. +# Copy to devnet.env (gitignored) and fill in — everything here is per-deployment, +# which is why none of it is baked into the scripts or dashboards. +# Loaded by the operator-side scripts via devnet-env.sh; an env var already set in +# your shell overrides the value here. + +# --- fleet (one independent devnet per server) ------------------------------- +SERVERS="host-a host-b" # ssh targets, any count/names +SSH_USER=user # login user; docker is invoked with sudo + +# Per-devnet size, as INVENTORY: the scripts take NODES/SUBNETS as positional +# args, and the authority for a running devnet is its own genesis/config.yaml +# (ATTESTATION_COMMITTEE_COUNT). Recorded here so a restart doesn't have to guess. +# host-a: NODES=32 SUBNETS=4 +# host-b: NODES=16 SUBNETS=2 + +# --- central metrics / logs stack ------------------------------------------- +# sweep.sh reads CENTRAL_PROM_URL; the other two are args you pass to +# prometheus-config.sh / promtail-config.sh. +CENTRAL_PROM_URL=http://10.0.0.4:9099 +CENTRAL_WRITE_URL=http://10.0.0.4:9099/api/v1/write +LOKI_PUSH_URL=http://10.0.0.4:3100/loki/api/v1/push + +# --- central Grafana (finality alert + dashboards) --------------------------- +METRICS_HOST=user@10.0.0.4 # ssh target of the host running Grafana +GRAFANA_PROV_DIR=/home/user/metrics/grafana-provisioning +# Where the dashboard JSONs live: the host dir bind-mounted to +# /var/lib/grafana/dashboards, NOT $GRAFANA_PROV_DIR/dashboards (that holds only +# the provider yaml). Read it off `docker inspect $GRAFANA_CONTAINER`. +GRAFANA_DASHBOARDS_DIR=/home/user/metrics/dashboards +GRAFANA_CONTAINER=grafana # Grafana container name on that host +GRAFANA_BASE_URL=http://10.0.0.4:3000 # base url for the dashboard link in Slack +PROM_DS_UID=central-prom # Grafana datasource uid of central prometheus +WEBHOOK_FILE=./webhook.txt # Slack webhook url file — SECRET, gitignored diff --git a/.claude/skills/multi-server-devnet/scripts/grafana-finality-alert.yaml.template b/.claude/skills/multi-server-devnet/scripts/grafana-finality-alert.yaml.template new file mode 100644 index 00000000..4e4cd677 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/grafana-finality-alert.yaml.template @@ -0,0 +1,116 @@ +# Grafana unified-alerting provisioning: per-devnet "lost finality" alert. +# Deployed to the central Grafana on the metrics host (see deploy-finality-alert.sh). +# +# Fires when a devnet's (`network`) CHAIN TIP gap exceeds 512 slots — the network's +# highest head minus its highest finalized — i.e. the canonical chain stopped finalizing. +# A plain magnitude threshold (no slope term) so it can't flap on a sawtoothing gap: a +# real stall stays above 512, a recovering devnet drops below and resolves. +# `lean_head_slot` and `lean_latest_finalized_slot` are the cross-client metrics +# every lean client exports (ethlambda, zeam, ream, qlean, gean, lantern). +# +# IMPORTANT: this is max(head) - max(finalized), NOT max(head - finalized). The per-node +# difference would surface the WORST single client: a canary that forks off and stops +# finalizing (e.g. zeam/qlean stuck at an old finalized while head tracks the tip) shows a +# per-series gap of thousands even when the supermajority chain finalizes fine, firing a +# false alert. Subtracting the two network maxima measures the chain itself — one stuck +# straggler can't drag down max(finalized) — while still catching a true network-wide stall. +# +# Placeholders substituted by deploy-finality-alert.sh (nothing host-specific is +# committed here): __SLACK_WEBHOOK_URL__ from webhook.txt (a secret — the rendered +# file is never committed), __GRAFANA_BASE_URL__ for the dashboard link in the Slack +# message, and __PROM_DS_UID__ for the central-prometheus datasource. A provisioned +# alert rule needs a CONCRETE datasource uid, so unlike the dashboards it can't use a +# datasource template variable. + +apiVersion: 1 + +contactPoints: + - orgId: 1 + name: slack-finality + receivers: + - uid: slack-finality + type: slack + disableResolveMessage: false + settings: + url: __SLACK_WEBHOOK_URL__ + # One single-line message per devnet (grouped by network). Variable-free + # template (Grafana's Slack templater rejects `:=`). No label dump / URLs. + title: 'Devnet finality' + text: | + {{ range .Alerts }}{{ if eq .Status "firing" }}:red_circle: *{{ .Labels.network }}* losing finality (distance from head to finalized exceeds 512 slots). <__GRAFANA_BASE_URL__/d/devnet-finality-overview?var-network={{ .Labels.network }}|dashboard>{{ else }}:large_green_circle: *{{ .Labels.network }}* recovered finality.{{ end }} + {{ end }} + +groups: + - orgId: 1 + name: finality + folder: Devnet Alerts + interval: 1m + rules: + - uid: devnet-lost-finality + title: Devnet lost finality + condition: B + # Require a signal to hold for 5m before firing, so a brief blip doesn't alert. + # (Set to 0s temporarily only when force-testing a fresh notification.) + for: 5m + # Anti-flap: once firing, stay firing until the condition has been false for 15m + # straight. Collapses a gap oscillating around 512 into one alert so it can't spam; + # with repeat_interval that's at most one notification per 12h. NOTE camelCase key + # (file-provisioning uses camelCase: noDataState, isPaused, datasourceUid, ...). + keepFiringFor: 15m + # The query filters to firing series only, so an empty result means "no devnet is + # losing finality" (healthy) — must be OK, not NoData, or it false-alarms when fine. + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __PROM_DS_UID__ + model: + refId: A + datasource: + type: prometheus + uid: __PROM_DS_UID__ + # One series per network whose chain-tip gap (max head - max finalized) + # exceeds 512 slots. Both sides aggregate `by (network)`, so the `network` + # label survives the subtraction and routing/grouping still works. + expr: "(max by (network) (lean_head_slot) - max by (network) (lean_latest_finalized_slot)) > 512" + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + datasourceUid: __expr__ + model: + refId: B + type: threshold + datasource: + type: __expr__ + uid: __expr__ + expression: A + # query A already filters to firing series (all positive values), so any + # value >0 means a signal tripped + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: critical + alert_class: finality + annotations: + summary: "{{ $labels.network }} losing finality (head-to-finalized distance >512 slots)." + description: "{{ $labels.network }}: head-to-finalized distance is {{ $values.A.Value }} slots (>512)." + notification_settings: + receiver: slack-finality + # Group per network so each devnet notifies independently. Without + # `network` here, all devnets share one group (default group_by is + # grafana_folder + alertname), so every notification re-lists every + # currently-firing devnet and the same network appears repeatedly. + group_by: + - grafana_folder + - alertname + - network + # re-notify at most once every 12h while a devnet stays in the firing state + repeat_interval: 12h diff --git a/.claude/skills/multi-server-devnet/scripts/host-check.sh b/.claude/skills/multi-server-devnet/scripts/host-check.sh new file mode 100755 index 00000000..af03591c --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/host-check.sh @@ -0,0 +1,85 @@ +#!/bin/bash +# Per-host: is every node of THIS devnet up and following the chain? +# +# The local counterpart to sweep.sh. sweep.sh reads the CENTRAL Prometheus, which +# is unavailable exactly when it matters most (this host's remote_write is down, or +# the central stack lives on a host that went dark). This one talks only to +# 127.0.0.1, so it works on a host that is otherwise cut off, and it is the check +# to run right after start-devnet.sh / cs-restart.sh / convert.sh. +# +# Client-agnostic: every lean client exports the same `lean_*` metrics, so a +# converted node reports in the same columns. +# +# Usage (on host): host-check.sh [NODES] +# NODES expected node count; indices 0..NODES-1 with no container are reported +# as MISSING. Omit to probe only the containers that exist. +# +# Exit status: 0 all good, 1 something needs attention (container not running or +# missing, metrics unreachable, or a head lagging this host's best by more than the +# sync-gate threshold -- past which the node stops attesting and proposing). +set -u +LAG_THRESHOLD=${LAG_THRESHOLD:-4} # SYNC_LAG_THRESHOLD, crates/blockchain/src/sync_status.rs +EXPECTED=${1:-} + +# Scrape one node's own metrics port. lean_connected_peers is one series PER PEER, +# so the peer count is the number of non-zero series. +probe(){ # $1 = node index -> "head just fin sync agg peers" + curl -s -m 3 "http://127.0.0.1:$((9200+$1))/metrics" 2>/dev/null | awk ' + /^lean_head_slot[ {]/ { head=$NF } + /^lean_latest_justified_slot[ {]/ { just=$NF } + /^lean_latest_finalized_slot[ {]/ { fin=$NF } + /^lean_is_aggregator[ {]/ { agg=($NF==1?"yes":"no") } + /^lean_node_sync_status\{/ { if ($NF==1) { match($0,/status="[^"]+"/) + sync=substr($0,RSTART+8,RLENGTH-9) } } + /^lean_connected_peers[ {]/ { if ($NF!=0) peers++ } + END { printf "%s %s %s %s %s %s", (head==""?"-":head), (just==""?"-":just), + (fin==""?"-":fin), (sync==""?"-":sync), (agg==""?"-":agg), peers+0 }' +} + +# One TSV record per node: idx name status restarts head just fin sync agg peers. +# Node containers are _; infra (prometheus/promtail/cadvisor) has no +# _ suffix and is skipped. +records="" +add_record(){ records="${records}$1"$'\n'; } + +seen="" +while read -r nm status; do + idx=${nm##*_} + case "$nm" in *_*) ;; *) continue ;; esac + case "$idx" in ''|*[!0-9]*) continue ;; esac + seen="$seen $idx" + rs=$(sudo docker inspect -f '{{.RestartCount}}' "$nm" 2>/dev/null) + add_record "$(printf '%s\t%s\t%s\t%s\t%s' "$idx" "$nm" "$status" "${rs:--}" "$(probe "$idx")")" +done < <(sudo docker ps -a --format '{{.Names}} {{.Status}}') + +# Expected-but-absent indices +if [ -n "$EXPECTED" ]; then + for i in $(seq 0 $((EXPECTED-1))); do + case " $seen " in *" $i "*) ;; *) add_record "$(printf '%s\t-\tMISSING (no container)\t-\t- - - - - -' "$i")" ;; esac + done +fi +[ -n "${records//[$'\n' ]/}" ] || { echo "no devnet node containers found"; exit 1; } + +records=$(printf '%s' "$records" | grep -v '^$' | sort -n) +best=$(printf '%s\n' "$records" | awk -F'\t' '{split($5,f," "); if (f[1]+0>m && f[1] ~ /^[0-9]+$/) m=f[1]} END{print m+0}') + +printf '%-5s %-14s %-22s %3s %9s %9s %9s %-8s %4s %5s\n' \ + NODE CONTAINER "DOCKER STATUS" RS HEAD JUST FIN SYNC AGG PEERS +bad=0 +while IFS=$'\t' read -r idx nm status rs vals; do + read -r h j f s a p <<<"$vals" + flag="" + case "$status" in Up*) ;; *) bad=1 ;; esac + case "$h" in + ''|-|*[!0-9]*) [ "$nm" != "-" ] && flag=" <- no metrics"; bad=1 ;; + *) [ $((best - h)) -gt "$LAG_THRESHOLD" ] && { + flag=" <- lags best head by $((best - h)) (>$LAG_THRESHOLD: duties suppressed)"; bad=1; } ;; + esac + printf '%-5s %-14s %-22s %3s %9s %9s %9s %-8s %4s %5s%s\n' \ + "$idx" "$nm" "${status:0:22}" "$rs" "$h" "$j" "$f" "$s" "$a" "$p" "$flag" +done <<<"$records" + +echo +echo "best head on this host: $best (cross-devnet view: sweep.sh / Grafana)" +[ "$bad" = 0 ] && echo "OK: all nodes up, reporting, and within $LAG_THRESHOLD slots of the best head" +exit "$bad" diff --git a/.claude/skills/multi-server-devnet/scripts/logs-dashboard.json b/.claude/skills/multi-server-devnet/scripts/logs-dashboard.json new file mode 100644 index 00000000..d83c8e82 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/logs-dashboard.json @@ -0,0 +1,165 @@ +{ + "uid": "devnet-logs", + "title": "Devnet Logs", + "tags": ["lean", "devnet", "logs"], + "schemaVersion": 39, + "version": 1, + "editable": true, + "graphTooltip": 0, + "refresh": "30s", + "time": { "from": "now-1h", "to": "now" }, + "timepicker": {}, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { "type": "grafana", "uid": "-- Grafana --" }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "templating": { + "list": [ + { + "name": "ds_prom", + "label": "Prometheus", + "type": "datasource", + "query": "prometheus", + "refresh": 1, + "current": {}, + "options": [] + }, + { + "name": "ds_loki", + "label": "Loki", + "type": "datasource", + "query": "loki", + "refresh": 1, + "current": {}, + "options": [] + }, + { + "name": "network", + "type": "query", + "label": "Devnet", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "query": { + "qryType": 1, + "query": "label_values(lean_head_slot, network)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 2, + "sort": 1, + "multi": false, + "includeAll": false, + "current": {} + }, + { + "name": "node", + "type": "query", + "label": "Node", + "datasource": { "type": "loki", "uid": "${ds_loki}" }, + "query": "label_values({network=\"$network\"}, node)", + "refresh": 2, + "sort": 1, + "multi": true, + "includeAll": true, + "allValue": ".*", + "current": { "text": ["All"], "value": ["$__all"] } + }, + { + "name": "stream", + "type": "query", + "label": "Stream", + "datasource": { "type": "loki", "uid": "${ds_loki}" }, + "query": "label_values({network=\"$network\"}, stream)", + "refresh": 2, + "sort": 1, + "multi": true, + "includeAll": true, + "allValue": ".*", + "current": { "text": ["All"], "value": ["$__all"] } + }, + { + "name": "search", + "type": "textbox", + "label": "Search (substring)", + "query": "", + "current": { "text": "", "value": "" }, + "options": [] + } + ] + }, + "panels": [ + { + "id": 1, + "type": "timeseries", + "title": "Log volume by level — $network", + "datasource": { "type": "loki", "uid": "${ds_loki}" }, + "gridPos": { "h": 7, "w": 24, "x": 0, "y": 0 }, + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "bars", + "fillOpacity": 70, + "lineWidth": 0, + "stacking": { "mode": "normal", "group": "A" } + }, + "unit": "short" + }, + "overrides": [ + { "matcher": { "id": "byName", "options": "INFO" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "green" } } ] }, + { "matcher": { "id": "byName", "options": "DEBUG" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "blue" } } ] }, + { "matcher": { "id": "byName", "options": "TRACE" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "purple" } } ] }, + { "matcher": { "id": "byName", "options": "WARN" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } } ] }, + { "matcher": { "id": "byName", "options": "ERROR" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } } ] } + ] + }, + "options": { + "legend": { "showLegend": true, "displayMode": "list", "placement": "bottom", "calcs": ["sum"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "loki", "uid": "${ds_loki}" }, + "editorMode": "code", + "queryType": "range", + "legendFormat": "{{level}}", + "expr": "sum by (level) (count_over_time({network=\"$network\", node=~\"${node:pipe}\", stream=~\"${stream:pipe}\"} |= \"$search\" [$__interval]))" + } + ] + }, + { + "id": 2, + "type": "logs", + "title": "Logs — $network", + "datasource": { "type": "loki", "uid": "${ds_loki}" }, + "gridPos": { "h": 21, "w": 24, "x": 0, "y": 7 }, + "options": { + "showTime": true, + "showLabels": false, + "showCommonLabels": false, + "wrapLogMessage": true, + "prettifyLogMessage": false, + "enableLogDetails": true, + "dedupStrategy": "none", + "sortOrder": "Descending" + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "loki", "uid": "${ds_loki}" }, + "editorMode": "code", + "queryType": "range", + "maxLines": 1000, + "expr": "{network=\"$network\", node=~\"${node:pipe}\", stream=~\"${stream:pipe}\"} |= \"$search\"" + } + ] + } + ] +} diff --git a/.claude/skills/multi-server-devnet/scripts/make-genesis.sh b/.claude/skills/multi-server-devnet/scripts/make-genesis.sh new file mode 100755 index 00000000..766efab5 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/make-genesis.sh @@ -0,0 +1,110 @@ +#!/bin/bash +# OPERATOR-side: build ONE independent devnet's genesis dir. +# Produces a self-contained genesis (own GENESIS_TIME + genesis root) for a +# single host: NODES validators, SUBNETS committees, all ENRs pinned to +# 127.0.0.1 so this devnet never cross-discovers other servers' devnets. +# Aggregators are nodes 0..SUBNETS-1 (node k -> subnet k). Subnet of node n = +# n % SUBNETS. hash-sig keys for validators 0..NODES-1 are HARDLINKED from a +# canonical key set (keygen is the slow part; reuse it). +# +# Requires: yq, python3, docker, and lean-quickstart's generate-genesis.sh +# (set LQ to the lean-quickstart dir; default ~/lean_consensus/lean-quickstart). +# +# Usage: make-genesis.sh [GT_OFFSET] +# OUTDIR genesis build dir to create (e.g. devnet-foo/genesis) +# NODES validator count +# SUBNETS ATTESTATION_COMMITTEE_COUNT +# KEYS_DIR canonical hash-sig-keys dir (has validator__{attester,proposer}_key_{pk,sk}.ssz +# for i in 0..NODES-1 + validator-keys-manifest.yaml) +# SRC_VCONFIG a validator-config.yaml holding one 'privkey' per validator. Its +# 'name' prefix is irrelevant: privkeys are matched by the trailing +# index (name '_'), so an ethlambda_i or node_i source +# both work. +# GT_OFFSET seconds from now until genesis (default 240) +# generate-genesis.sh stamps GENESIS_TIME as now+offset EARLY (before +# the per-validator config loop), so everything after it eats the +# budget. Measured 2026-07-28 on a 32-node host: +# genesis generation after the stamp 13s (128 val) / 28s (256 val) +# ship + teardown + wipe + launch 93s / 105s +# gossip mesh formation (peers=N-1) ~30s +# => ~165s needed; 240 leaves ~75s margin. 600 wasted ~7min of wall +# clock per restart, which is the whole reason this default dropped. +# WIPE THE DATA DIRS BEFORE CALLING THIS: `rm -rf .../data/node_*` is +# the one step that scales with chain age (54 GB took a big chunk of +# that 93s), so doing it pre-stamp keeps the countdown predictable. +# Raise the offset only if shipping to many hosts or over a slow link. +# +# Env: VALIDATORS_PER_NODE (default 1) validators hosted by each node. KEYS_DIR +# must then hold NODES*VALIDATORS_PER_NODE key pairs, and the manifest is read +# positionally, so its indices must run 0..NODES*VALIDATORS_PER_NODE-1. +# +# Node identity is the client-agnostic 'node_' (node-id, node_.key, +# data-dir, validator name). The client running a node is reflected only in its +# container name (_), set by start-devnet.sh/convert.sh, never here. +set -eu +OUTDIR=$1; NODES=$2; SUBNETS=$3; KEYS_DIR=$4; SRC_VCONFIG=$5; GT_OFFSET=${6:-240} +LQ=${LQ:-$HOME/lean_consensus/lean-quickstart} + +mkdir -p "$OUTDIR/hash-sig-keys" + +python3 - "$OUTDIR" "$NODES" "$SUBNETS" "$KEYS_DIR" "$SRC_VCONFIG" <<'PY' +import os, sys, yaml +outdir, nodes, subnets, keys_dir, src_vconfig = sys.argv[1:6] +nodes, subnets = int(nodes), int(subnets) +# Validators per node (env VALIDATORS_PER_NODE, default 1). generate-genesis.sh +# sums the per-row 'count' into VALIDATOR_COUNT and assigns node i the +# contiguous global indices [i*vpn, (i+1)*vpn). +# NOTE: the client derives a validator's attestation subnet from its own global +# index (`vid % ACC`, see p2p attestation_subscription_subnets), NOT from the +# 'subnet' row below -- so at vpn == subnets every node owns one validator in +# every subnet and subscribes to all of them. 'subnet' is metadata for the +# ansible/other-client path only. +vpn = int(os.environ.get('VALIDATORS_PER_NODE', '1')) +# Match privkeys by the trailing index so any source prefix (ethlambda_i / node_i) +# works; node identity emitted here is the client-agnostic node_. +priv = {int(v['name'].rsplit('_', 1)[1]): v['privkey'] + for v in yaml.safe_load(open(src_vconfig))['validators']} +rows = [] +for i in range(nodes): + rows.append({ + 'name': f'node_{i}', 'privkey': priv[i], + 'subnet': i % subnets, + 'enrFields': {'ip': '127.0.0.1', 'quic': 9000 + i}, + 'metricsPort': 9200 + i, 'apiPort': 5052 + i, + 'isAggregator': i < subnets, 'count': vpn, + }) +cfg = {'shuffle': 'roundrobin', 'deployment_mode': 'ansible', + 'config': {'activeEpoch': 18, 'keyType': 'hash-sig', + 'attestation_committee_count': subnets}, + 'validators': rows} +header = (f"# {nodes}-node, {subnets}-subnet independent single-host devnet.\n" + f"# {vpn} validator(s) per node -> {nodes * vpn} validators total;\n" + f"# node i owns global validator indices {vpn}i..{vpn}i+{vpn - 1}.\n" + f"# Subnet of node n = n % {subnets}; aggregators are nodes 0..{subnets-1}.\n" + "# All ENRs pinned to 127.0.0.1 so this devnet is fully isolated.\n" + "# Node identity is the client-agnostic node_ (node-id, node_.key);\n" + "# the client is reflected only in the container name (_).\n") +with open(f'{outdir}/validator-config.yaml', 'w') as f: + f.write(header); yaml.safe_dump(cfg, f, default_flow_style=False, sort_keys=False) +# hardlink keys 0..(nodes*vpn)-1 + manifest (one key pair per VALIDATOR, not per node) +names = ['validator-keys-manifest.yaml'] + [ + f'validator_{i}_{r}_key_{p}.ssz' + for i in range(nodes * vpn) for r in ('attester', 'proposer') for p in ('pk', 'sk')] +linked = 0 +for n in names: + dst = f'{outdir}/hash-sig-keys/{n}' + if not os.path.exists(dst): + os.link(f'{keys_dir}/{n}', dst); linked += 1 +print(f'validator-config + {linked} hardlinked key files ready') +PY + +"$LQ/generate-genesis.sh" "$OUTDIR" --offset "$GT_OFFSET" + +# generate-genesis.sh hands node i the contiguous index block [i*vpn,(i+1)*vpn), +# which at vpn == SUBNETS puts one of its validators in every subnet. Re-derive +# the assignment so each node holds validators from a single subnet instead +# (identity when VALIDATORS_PER_NODE=1). +python3 "$(dirname "$0")/subnet-align-validators.py" \ + "$OUTDIR" "$NODES" "$SUBNETS" "${VALIDATORS_PER_NODE:-1}" + +echo "genesis ready: GT=$(grep ^GENESIS_TIME "$OUTDIR/config.yaml" | awk '{print $2}') ACC=$(grep ^ATTESTATION_COMMITTEE_COUNT "$OUTDIR/config.yaml" | awk '{print $2}') validators=$(grep -c attestation_pubkey "$OUTDIR/config.yaml")" diff --git a/.claude/skills/multi-server-devnet/scripts/merge-keyshards.py b/.claude/skills/multi-server-devnet/scripts/merge-keyshards.py new file mode 100644 index 00000000..f6cedb93 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/merge-keyshards.py @@ -0,0 +1,167 @@ +#!/usr/bin/env python3 +"""Merge hash-sig key shards into one canonical, sequentially-indexed key dir. + +hash-sig-cli always emits validator_0..validator_{N-1}, so generating a key set +across several machines produces colliding indices. This reindexes each shard +into a single 0..TOTAL-1 namespace (shards concatenated in argument order) and +regenerates the manifest that lean-quickstart's generate-genesis.sh reads +positionally. + +Refuses to merge shards whose scheme/pubkey width differ: that mismatch is +exactly how stale 52-byte (Dim46) keys would silently poison a 32-byte +(Dim42) leanVM-main genesis. + +Usage: merge-keyshards.py OUTDIR SHARD_DIR [SHARD_DIR ...] +""" +import re +import shutil +import sys +from pathlib import Path + +import yaml + +ROLES = ("attester", "proposer") +PARTS = ("pk", "sk") + + +class HexSafeLoader(yaml.SafeLoader): + """SafeLoader that does not coerce `0x...` scalars to int. + + The manifest stores pubkeys as unquoted `0x`. PyYAML follows YAML 1.1, + whose int resolver matches `0x[0-9a-fA-F]+`, so a plain safe_load turns a + pubkey into a Python int and round-tripping it writes a DECIMAL -- which yq + (YAML 1.2, keeps it a string) then rejects downstream, and which would also + silently drop any leading zero byte. Dropping the int resolver keeps every + scalar a str; callers int() the few numeric fields they actually use. + """ + + +HexSafeLoader.yaml_implicit_resolvers = { + key: [(tag, regexp) for tag, regexp in resolvers + if tag != "tag:yaml.org,2002:int"] + for key, resolvers in yaml.SafeLoader.yaml_implicit_resolvers.items() +} +# Header fields that must agree across every shard (they define the key format). +CRITICAL = ("key_scheme", "hash_function", "encoding", "pubkey_bytes", + "lifetime", "log_num_active_epochs", "num_active_epochs") + + +def main() -> int: + if len(sys.argv) < 3: + print(__doc__) + return 2 + outdir = Path(sys.argv[1]) + shards = [Path(p) for p in sys.argv[2:]] + + header: dict | None = None + merged: list[dict] = [] + # (src_shard, src_index) per destination index, for the file copy pass. + provenance: list[tuple[Path, int]] = [] + + for shard in shards: + manifest_path = shard / "validator-keys-manifest.yaml" + man = yaml.load(manifest_path.read_text(), Loader=HexSafeLoader) + fields = {k: man.get(k) for k in CRITICAL} + if header is None: + header = fields + elif fields != header: + diff = {k: (header[k], fields[k]) for k in CRITICAL + if header[k] != fields[k]} + print(f"ERROR: {shard} key format differs from first shard: {diff}") + return 1 + + entries = man["validators"] + # Pubkeys must survive as literal 0x-hex of the declared width: a non-str + # here means the int-resolver crept back in, and a short one means a + # leading zero byte was lost. + want_width = 2 * int(header["pubkey_bytes"]) + for entry in entries: + for role in ROLES: + pk = entry[f"{role}_key_pubkey_hex"] + if not isinstance(pk, str) or not re.fullmatch(r"0x[0-9a-fA-F]+", pk): + print(f"ERROR: {shard} validator {entry['index']} {role} " + f"pubkey is not a 0x-hex string: {pk!r}") + return 1 + if len(pk) - 2 != want_width: + print(f"ERROR: {shard} validator {entry['index']} {role} pubkey " + f"is {len(pk) - 2} hex chars, expected {want_width}") + return 1 + # Trust the manifest's own ordering, but key the copy off its recorded + # index rather than list position so a sparse shard can't silently shift. + for entry in entries: + src_index = int(entry["index"]) + dst_index = len(merged) + merged.append({ + "index": dst_index, + "proposer_key_pubkey_hex": entry["proposer_key_pubkey_hex"], + "proposer_key_privkey_file": + f"validator_{dst_index}_proposer_key_sk.ssz", + "attester_key_pubkey_hex": entry["attester_key_pubkey_hex"], + "attester_key_privkey_file": + f"validator_{dst_index}_attester_key_sk.ssz", + }) + provenance.append((shard, src_index)) + print(f" {shard.name}: {len(entries)} validators") + + outdir.mkdir(parents=True, exist_ok=True) + for dst_index, (shard, src_index) in enumerate(provenance): + for role in ROLES: + for part in PARTS: + src = shard / f"validator_{src_index}_{role}_key_{part}.ssz" + dst = outdir / f"validator_{dst_index}_{role}_key_{part}.ssz" + if not src.exists(): + print(f"ERROR: missing {src}") + return 1 + shutil.copy2(src, dst) + + assert header is not None + out_manifest = dict(header) + out_manifest["num_validators"] = len(merged) + # Emit in the same field order hash-sig-cli uses, then the validator list. + lines = [ + "# Hash-Signature Validator Keys Manifest", + "# Generated by hash-sig-cli, reindexed by merge-keyshards.py", + f"# Shards merged in order: {', '.join(s.name for s in shards)}", + "", + ] + for key in CRITICAL: + lines.append(f"{key}: {out_manifest[key]}") + lines.append(f"num_validators: {len(merged)}") + lines.append("") + lines.append("validators:") + for entry in merged: + lines.append(f" - index: {entry['index']}") + lines.append( + f" proposer_key_pubkey_hex: {entry['proposer_key_pubkey_hex']}") + lines.append( + f" proposer_key_privkey_file: {entry['proposer_key_privkey_file']}") + lines.append( + f" attester_key_pubkey_hex: {entry['attester_key_pubkey_hex']}") + lines.append( + f" attester_key_privkey_file: {entry['attester_key_privkey_file']}") + lines.append("") + (outdir / "validator-keys-manifest.yaml").write_text("\n".join(lines)) + + # Re-read through yaml to prove generate-genesis.sh's yq will parse it, and + # that pubkeys survived the reindex unchanged. + check = yaml.load((outdir / "validator-keys-manifest.yaml").read_text(), + Loader=HexSafeLoader) + assert int(check["num_validators"]) == len(merged) + assert [int(v["index"]) for v in check["validators"]] == list(range(len(merged))) + # The regression this guards: pubkeys must remain 0x-hex TEXT on the way out, + # not decimals produced by an int-coercing round-trip. + for v in check["validators"]: + for role in ROLES: + pk = v[f"{role}_key_pubkey_hex"] + assert isinstance(pk, str) and pk.startswith("0x"), \ + f"validator {v['index']} {role} pubkey not 0x-hex text: {pk!r}" + dupes = len({v["attester_key_pubkey_hex"] for v in merged}) != len(merged) + print(f"\nmerged {len(merged)} validators -> {outdir}") + print(f" key_scheme={header['key_scheme']} " + f"pubkey_bytes={header['pubkey_bytes']}") + print(f" duplicate attester pubkeys: {dupes}") + return 1 if dupes else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.claude/skills/multi-server-devnet/scripts/prometheus-config.sh b/.claude/skills/multi-server-devnet/scripts/prometheus-config.sh new file mode 100755 index 00000000..5ded1c28 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/prometheus-config.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# OPERATOR-side: emit one devnet's per-host prometheus.yml to stdout. +# Scrapes nodes 0..NODES-1 (metrics 9200+n) with a per-devnet `network` label +# and per-node `client_type`, plus the host's node_exporter (systemd, on +# HOST_IP:9122 — installed separately via lambdaclass/monitoring-stack) and +# cadvisor (docker, 172.17.0.1:9098), and remote-writes to +# the central Prometheus. Pipe into `ssh host 'sudo tee .../prometheus.yml'`. +# +# Usage: prometheus-config.sh [N:client ...] +# NETWORK label value (e.g. devnet-foo) — Grafana filters on this +# NODES validator count +# HOST_IP this host's address used as the `instance` label +# CENTRAL_WRITE_URL remote_write target (e.g. http://10.0.0.4:9099/api/v1/write) +# N:client optional canary node->client_type overrides +set -eu +python3 - "$@" <<'PY' +import sys +net, nodes, host_ip, central = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4] +canary = dict(s.split(':', 1) for s in sys.argv[5:]) +out = ['global:', ' scrape_interval: 15s', '', 'scrape_configs:'] +for n in range(nodes): + c = canary.get(str(n), 'ethlambda') + out += [f" - job_name: '{c}_{n}'", ' static_configs:', + f" - targets: ['172.17.0.1:{9200+n}']", ' labels:', + " type: 'app'", f" instance: '{host_ip}'", + f" network: '{net}'", f" client_type: '{c}'"] +out += [" - job_name: 'node_exporter'", ' static_configs:', + f" - targets: ['{host_ip}:9122']", ' labels:', + " type: 'node'", f" instance: '{host_ip}'", f" network: '{net}'", + " - job_name: 'cadvisor'", ' static_configs:', + " - targets: ['172.17.0.1:9098']", ' labels:', + " type: 'docker'", f" instance: '{host_ip}'", f" network: '{net}'", + ' metric_relabel_configs:', ' - source_labels: [name]', + " regex: '([a-z]+)_.*'", ' target_label: client_type', + " replacement: '$1'", + 'remote_write:', f' - url: {central}', ''] +print('\n'.join(out)) +PY diff --git a/.claude/skills/multi-server-devnet/scripts/promtail-config.sh b/.claude/skills/multi-server-devnet/scripts/promtail-config.sh new file mode 100755 index 00000000..c85ffea8 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/promtail-config.sh @@ -0,0 +1,142 @@ +#!/bin/bash +# OPERATOR-side: emit one host's promtail.yml to stdout. promtail discovers the +# devnet's node containers via the Docker daemon API (docker_sd) and ships their +# stdout/stderr to the central Loki. Reading via the daemon API (not the json +# files) makes it immune to docker's log-rotation: promtail consumes the live +# stream, so --log-opt max-size/max-file rotation never races it. +# +# Labels mirror prometheus-config.sh so logs and metrics share a vocabulary: +# network per-devnet label (Grafana filters on this) — SAME value as the +# host's prometheus.yml network label +# node container name (_N, e.g. zeam_8 — reflects the client) +# client_type derived from the container-name prefix (self-maintaining) +# instance this host's address +# stream stdout|stderr +# Only containers named _[0-9]+ are kept (infra has no _ suffix). +# +# Pipe into `ssh host 'sudo tee /opt/lean-quickstart/observability/promtail.yml'`. +# That file is a single-file bind-mount: overwrite with tee (preserves inode), +# then `sudo docker restart promtail` — never sed -i (inode trap, see operations.md). +# +# Usage: promtail-config.sh [N:client ...] +# NETWORK per-devnet label value (e.g. devnet-) +# HOST_IP this host's address used as the `instance` label +# LOKI_PUSH_URL central Loki push endpoint +# (e.g. http://10.0.0.4:3100/loki/api/v1/push) +# N:client accepted for call-site compatibility but IGNORED now that +# client_type is derived from the container-name prefix +set -eu +python3 - "$@" <<'PY' +import sys +net, host_ip, push_url = sys.argv[1], sys.argv[2], sys.argv[3] +# Canary N:client args (sys.argv[4:]) are accepted for call-site compatibility +# but no longer used: client_type is derived from the container-name prefix. + +out = [ + 'server:', + ' http_listen_port: 9080', + ' grpc_listen_port: 0', + '', + 'positions:', + ' filename: /promtail-data/positions.yaml', + '', + 'clients:', + f' - url: {push_url}', + '', + 'scrape_configs:', + ' - job_name: ethlambda-logs', + ' docker_sd_configs:', + ' - host: unix:///var/run/docker.sock', + ' refresh_interval: 10s', + ' relabel_configs:', + " # keep only devnet node containers (_N); infra has no _ suffix", + " - source_labels: ['__meta_docker_container_name']", + " regex: '/([a-z]+_[0-9]+)'", + ' action: keep', + " - source_labels: ['__meta_docker_container_name']", + " regex: '/(.*)'", + ' target_label: node', + " # client_type derived from the container-name prefix (self-maintaining:", + " # no per-node overrides now that the name encodes the client)", + " - source_labels: ['node']", + " regex: '([a-z]+)_[0-9]+'", + ' target_label: client_type', + " replacement: '${1}'", + " - source_labels: ['__meta_docker_container_log_stream']", + ' target_label: stream', + ' - target_label: instance', + f" replacement: '{host_ip}'", + ' - target_label: network', + f" replacement: '{net}'", +] +# Drop backlog: docker_sd has no "since", so on first start (empty positions) +# promtail replays each container's FULL retained history (the 600m x N json +# files can reach back days), which Loki rejects as "too old" (>retention) and +# rate-limits. This stage ships only the last hour forward; persisted positions +# mean the one-time backlog read happens once. Trade-off: a promtail outage +# longer than 1h loses the excess on recovery (docker logs + Loki remain the +# buffers). older_than uses the docker log timestamp, so steady-state (current) +# lines are never dropped. +PIPELINE = [ + ' pipeline_stages:', + ' # 1. drop the first-start backlog (docker_sd has no "since"; see header)', + ' - drop:', + ' older_than: 1h', + ' drop_counter_reason: backlog_too_old', + ' # 2. strip ANSI color/style escapes from the client pretty-logger output', + " - replace:", + " expression: '(\\x1b\\[[0-9;]*m)'", + " replace: ''", + ' # 3. merge multi-line events (e.g. fork-choice tree dumps): a line that does NOT', + ' # start with a timestamp is a continuation of the previous entry. Runs after', + ' # ANSI strip so firstline anchors on the bare timestamp; state is per-stream', + ' # (node label), so containers never bleed into each other.', + " - multiline:", + " firstline: '^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}'", + ' max_wait_time: 3s', + ' max_lines: 1024', + ' # 4. lift the tracing level (token after the timestamp) into a low-card label;', + ' # also lets Grafana colourise the logs panel by level. Lines with no level', + ' # (multi-line dumps) simply get no level label.', + " - regex:", + " expression: '^\\S+\\s+(?PTRACE|DEBUG|INFO|WARN|ERROR)\\b'", + ' - labels:', + ' level:', + ' # 5. attach high-cardinality fields as STRUCTURED METADATA (NOT labels):', + ' # filterable in Grafana/LogQL (e.g. | slot="123") without exploding the', + ' # stream cardinality. Wrapped in a match so only lines that actually carry', + ' # key=value fields pay the logfmt cost. logfmt mapping is explicit, so the', + ' # non-logfmt prefix (timestamp/level/target) is ignored.', + ' - match:', + " selector: '{network=~\".+\"} |= \"=\"'", + ' stages:', + ' - logfmt:', + ' mapping:', + ' slot:', + ' validator:', + ' proposer:', + ' block_root:', + ' parent_root:', + ' target_slot:', + ' target_root:', + ' source_slot:', + ' source_root:', + ' attestation_count:', + ' peer_id:', + ' - structured_metadata:', + ' slot:', + ' validator:', + ' proposer:', + ' block_root:', + ' parent_root:', + ' target_slot:', + ' target_root:', + ' source_slot:', + ' source_root:', + ' attestation_count:', + ' peer_id:', +] +out += PIPELINE +out.append('') +print('\n'.join(out)) +PY diff --git a/.claude/skills/multi-server-devnet/scripts/resources-dashboard.json b/.claude/skills/multi-server-devnet/scripts/resources-dashboard.json new file mode 100644 index 00000000..7cf7ca28 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/resources-dashboard.json @@ -0,0 +1,372 @@ +{ + "uid": "devnet-resources", + "title": "Devnet Resources (cAdvisor)", + "tags": ["lean", "devnet", "resources"], + "schemaVersion": 39, + "version": 8, + "editable": true, + "graphTooltip": 1, + "refresh": "30s", + "time": { "from": "now-1h", "to": "now" }, + "timepicker": {}, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { "type": "grafana", "uid": "-- Grafana --" }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "templating": { + "list": [ + { + "name": "ds_prom", + "label": "Prometheus", + "type": "datasource", + "query": "prometheus", + "refresh": 1, + "current": {}, + "options": [] + }, + { + "name": "network", + "type": "query", + "label": "Devnet", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "query": { "qryType": 1, "query": "label_values(lean_head_slot, network)", "refId": "var-network" }, + "refresh": 2, + "sort": 1, + "multi": false, + "includeAll": false, + "current": {} + }, + { + "name": "name", + "type": "query", + "label": "Node", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "query": { "qryType": 1, "query": "label_values(container_memory_working_set_bytes{network=\"$network\"}, name)", "refId": "var-name" }, + "regex": "/^[a-z]+_\\d+$/", + "refresh": 2, + "sort": 5, + "multi": true, + "includeAll": true, + "allValue": "[a-z]+_[0-9]+", + "current": { "text": ["All"], "value": ["$__all"] } + } + ] + }, + "panels": [ + { + "id": 1, + "type": "timeseries", + "title": "CPU usage by node (cores) — $network", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 0 }, + "fieldConfig": { + "defaults": { + "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" }, + "unit": "short", + "decimals": 2 + }, + "overrides": [] + }, + "options": { + "legend": { "showLegend": true, "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "mean", "max"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "legendFormat": "{{name}}", + "expr": "rate(container_cpu_usage_seconds_total{network=\"$network\", name=~\"${name:pipe}\"}[$__rate_interval])" + } + ] + }, + { + "id": 2, + "type": "timeseries", + "title": "Memory working set by node — $network", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 }, + "fieldConfig": { + "defaults": { + "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { "id": "byName", "options": "limit" }, + "properties": [ + { "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }, + { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } }, + { "id": "custom.fillOpacity", "value": 0 } + ] + } + ] + }, + "options": { + "legend": { "showLegend": true, "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "mean", "max"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "legendFormat": "{{name}}", + "expr": "container_memory_working_set_bytes{network=\"$network\", name=~\"${name:pipe}\"}" + }, + { + "refId": "B", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "legendFormat": "limit", + "expr": "max(container_spec_memory_limit_bytes{network=\"$network\", name=~\"${name:pipe}\"})" + } + ] + }, + { + "id": 3, + "type": "timeseries", + "title": "Memory used (% of limit) by node — $network", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 }, + "fieldConfig": { + "defaults": { + "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" }, + "unit": "percent", + "min": 0, + "max": 100, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "orange", "value": 70 }, + { "color": "red", "value": 85 } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { "showLegend": true, "displayMode": "list", "placement": "bottom", "calcs": ["mean"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "legendFormat": "{{name}}", + "expr": "100 * container_memory_working_set_bytes{network=\"$network\", name=~\"${name:pipe}\"} / container_spec_memory_limit_bytes{network=\"$network\", name=~\"${name:pipe}\"}" + } + ] + }, + { + "id": 4, + "type": "bargauge", + "title": "Memory now (% of limit) — $network", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 }, + "fieldConfig": { + "defaults": { + "unit": "percent", + "min": 0, + "max": 100, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "orange", "value": 70 }, + { "color": "red", "value": 85 } + ] + } + }, + "overrides": [] + }, + "options": { + "displayMode": "gradient", + "orientation": "horizontal", + "showUnfilled": true, + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "instant": true, + "legendFormat": "{{name}}", + "expr": "100 * container_memory_working_set_bytes{network=\"$network\", name=~\"${name:pipe}\"} / container_spec_memory_limit_bytes{network=\"$network\", name=~\"${name:pipe}\"}" + } + ] + }, + { + "id": 5, + "type": "timeseries", + "title": "OOM kills by node — $network", + "description": "Per-node OOM-kill events from cAdvisor (container_oom_events_total), counted over each time bucket. Legend 'sum' = total OOM kills in the selected range. Summed by node name so a container that was recreated after an OOM doesn't split into two series.", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 }, + "fieldConfig": { + "defaults": { + "custom": { "drawStyle": "bars", "fillOpacity": 60, "lineWidth": 1, "showPoints": "never", "stacking": { "mode": "normal", "group": "A" } }, + "unit": "short", + "decimals": 0, + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { "showLegend": true, "displayMode": "table", "placement": "right", "calcs": ["sum", "max"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "legendFormat": "{{name}}", + "expr": "sum by (name) (increase(container_oom_events_total{network=\"$network\", name=~\"${name:pipe}\"}[$__rate_interval]))" + } + ] + }, + { + "id": 6, + "type": "timeseries", + "title": "Restart events by node — $network", + "description": "(Re)starts detected as jumps in the newest start time per node: changes(max by(name)(container_start_time_seconds)). Cgroup-agnostic and mechanism-agnostic — catches BOTH in-place restarts (docker restart / crash+restart-policy, same container id) AND full recreations (docker rm+run: image upgrade, client conversion, config change), which the naive per-series changes() misses. Legend 'sum' = total (re)starts in the selected range. Does not count a node's very first start within the window if its start time never changed.", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 }, + "fieldConfig": { + "defaults": { + "custom": { "drawStyle": "bars", "fillOpacity": 60, "lineWidth": 1, "showPoints": "never", "stacking": { "mode": "normal", "group": "A" } }, + "unit": "short", + "decimals": 0, + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { "showLegend": true, "displayMode": "table", "placement": "right", "calcs": ["sum", "lastNotNull"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "legendFormat": "{{name}}", + "expr": "changes((max by (name) (container_start_time_seconds{network=\"$network\", name=~\"${name:pipe}\"}))[$__rate_interval:15s])" + } + ] + }, + { + "id": 7, + "type": "stat", + "title": "Current uptime by node — $network", + "description": "Time since each container last (re)started = time() - container_start_time_seconds. A low value means a recent restart or a fresh (re)deploy. Red < 5m, orange < 30m, green otherwise.", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 }, + "fieldConfig": { + "defaults": { + "unit": "s", + "decimals": 0, + "color": { "mode": "thresholds" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "red", "value": null }, + { "color": "orange", "value": 300 }, + { "color": "green", "value": 1800 } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "orientation": "auto", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "textMode": "value_and_name" + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "instant": true, + "legendFormat": "{{name}}", + "expr": "time() - max by (name) (container_start_time_seconds{network=\"$network\", name=~\"${name:pipe}\"})" + } + ] + }, + { + "id": 8, + "type": "table", + "title": "Image build (commit per node) — $network", + "description": "Which build each node is actually running, read from the OCI labels baked into its container image (org.opencontainers.image.revision / .ref.name): docker copies an image's labels onto every container created from it, and cAdvisor re-exports them as container_label_*. It is the label of the image the container was CREATED with, so it stays correct after a `docker pull` that was not followed by a recreate, unlike the mutable tag in Image. An empty Commit means that client's Dockerfile does not set the label. Read it next to 'Restart events' / 'Current uptime': a fresh uptime with an unchanged Commit means the node came back on the SAME build, a changed one means the restart also upgraded it.", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "gridPos": { "h": 9, "w": 24, "x": 0, "y": 40 }, + "fieldConfig": { + "defaults": { + "custom": { "align": "auto", "cellOptions": { "type": "auto" }, "filterable": true, "inspect": false }, + "noValue": "No cAdvisor data for this network. Is its cadvisor container being scraped?" + }, + "overrides": [ + { "matcher": { "id": "byName", "options": "Node" }, "properties": [{ "id": "custom.width", "value": 130 }] }, + { "matcher": { "id": "byName", "options": "Client" }, "properties": [{ "id": "custom.width", "value": 110 }] }, + { "matcher": { "id": "byName", "options": "Commit" }, "properties": [{ "id": "custom.width", "value": 350 }] }, + { "matcher": { "id": "byName", "options": "Branch / tag" }, "properties": [{ "id": "custom.width", "value": 230 }] }, + { "matcher": { "id": "byName", "options": "Image built" }, "properties": [{ "id": "custom.width", "value": 230 }] } + ] + }, + "options": { + "cellHeight": "sm", + "showHeader": true, + "sortBy": [{ "desc": false, "displayName": "Node" }] + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${ds_prom}" }, + "editorMode": "code", + "instant": true, + "legendFormat": "__auto", + "expr": "last_over_time(container_last_seen{network=\"$network\", name=~\"${name:pipe}\"}[5m])" + } + ], + "transformations": [ + { "id": "labelsToFields", "options": { "keepLabels": ["name", "client_type", "container_label_org_opencontainers_image_revision", "container_label_org_opencontainers_image_ref_name", "image", "container_label_org_opencontainers_image_created"] } }, + { "id": "merge", "options": {} }, + { "id": "filterFieldsByName", "options": { "include": { "names": ["name", "client_type", "container_label_org_opencontainers_image_revision", "container_label_org_opencontainers_image_ref_name", "image", "container_label_org_opencontainers_image_created"] } } }, + { + "id": "organize", + "options": { + "excludeByName": {}, + "includeByName": {}, + "indexByName": { "name": 0, "client_type": 1, "container_label_org_opencontainers_image_revision": 2, "container_label_org_opencontainers_image_ref_name": 3, "image": 4, "container_label_org_opencontainers_image_created": 5 }, + "renameByName": { + "name": "Node", + "client_type": "Client", + "container_label_org_opencontainers_image_revision": "Commit", + "container_label_org_opencontainers_image_ref_name": "Branch / tag", + "image": "Image", + "container_label_org_opencontainers_image_created": "Image built" + } + } + } + ] + } + ] +} diff --git a/.claude/skills/multi-server-devnet/scripts/start-devnet.sh b/.claude/skills/multi-server-devnet/scripts/start-devnet.sh new file mode 100755 index 00000000..f7cd48bf --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/start-devnet.sh @@ -0,0 +1,222 @@ +#!/bin/bash +# Launch a full INDEPENDENT single-host devnet on THIS host (fresh start, no +# checkpoint sync — every node boots from the genesis already in /config). +# Nodes 0..NODES-1, ports gossip 9000+n / api 5052+n / metrics 9200+n. +# Aggregators: nodes 0..SUBNETS-1 by default, each covering subnet n % SUBNETS +# (i.e. its own index). Override the SET with AGG_NODES when the default lands +# them all on one client — see below. +# Three-layer naming: node identity = node_N (node-id, node_N.key, data-dir, +# genesis validator name); container/display = _N (docker name, cAdvisor +# `name`, promtail `node`); ports = index N. All genesis artifacts key off node_N. +# +# Usage (on host): start-devnet.sh [N:client ...] +# NODES validators on this server +# SUBNETS ATTESTATION_COMMITTEE_COUNT (= number of aggregators) +# N:client optional: run node N as client in zeam|ream|qlean|gean|lantern|grandine +# (everything else runs ethlambda) +# +# AGG_NODES="0 4 8" which node indices aggregate (default: 0..SUBNETS-1). The +# subnet a node covers is always n % SUBNETS — never passed separately, because +# a node must aggregate the subnet its OWN validators attest on (a client +# subscribes to that one subnet, so aggregating a different one listens to a +# mesh it is not in). The set must therefore cover every subnet exactly once, +# which is checked below. Use this to spread the aggregators across clients on a +# mixed devnet: the default 0..SUBNETS-1 is a contiguous prefix, so on a layout +# whose first SUBNETS nodes are one client it hands that client every subnet. +set -u +NODES=$1; SUBNETS=$2; shift 2 +# Paths are env-overridable (same as cs-restart.sh) so the launch can be +# exercised against a fixture dir without touching the real devnet. +GENESIS=${GENESIS:-/opt/lean-quickstart/genesis} +DATA=${DATA:-/opt/lean-quickstart/data} +# Per-container limits, env-overridable (same pattern as the *_IMG vars below). +# No CPU caps: block production is latency-bound on the leanVM Type-2 merge +# (~6.5s per packed attestation), not CPU-bound, so caps only starved nodes +# without taming the cost. (History: --cpus 1.5/8 were added 2026-06-18 after a +# non-finality spiral drove 32 nodes to load 50 on 16 cores; removed 2026-06-24.) +# Memory is uniform across roles: 8g RAM / 16g RAM+swap / 2g reservation. +MEM="${MEM:---memory 8g --memory-swap 16g --memory-reservation 2g}" +MEM_AGG="${MEM_AGG:---memory 8g --memory-swap 16g --memory-reservation 2g}" +# Cap Docker json-file stdout logs (default driver is unbounded -> filled 936G disks). +# 200m x 5 = 1 GiB max per container. Applied to every `docker run` below. +# History: started 100m x 3 (300 MiB); bumped to 600m x 10 (6 GiB) on 2026-06-17 so a +# multi-hour stall stayed inspectable via `docker logs` (300 MiB only held ~7-15h). +# Reduced to 1 GiB on 2026-06-29 once the central Loki (7d retention) took over that +# role: the long-term stall record now lives in Loki/Grafana, so the local json-file is +# just a hot buffer for `docker logs --tail` + a promtail-outage cushion (~a day at this +# rate). promtail ships via the Docker daemon API, so this rotation never races it. +# Env-overridable (same pattern as MEM above). +LOGOPT="${LOGOPT:---log-opt max-size=200m --log-opt max-file=5}" +LOG=/tmp/start-devnet.log +log(){ echo "$(date '+%H:%M:%S') $*" | tee -a "$LOG"; } +: > "$LOG" + +ETH_IMG=${ETH_IMG:-ghcr.io/lambdaclass/ethlambda:devnet5} +# Extra CLI flags appended to every ethlambda node (env-overridable). Used to +# inject experiment-only flags like --disable-duty-sync-gate without hardcoding. +EXTRA_ETH_FLAGS=${EXTRA_ETH_FLAGS:-} +ZEAM_IMG=${ZEAM_IMG:-blockblaz/zeam:devnet5} +REAM_IMG=${REAM_IMG:-ghcr.io/reamlabs/ream:latest-devnet5} +QLEAN_IMG=${QLEAN_IMG:-qdrvm/qlean-mini:devnet-5-amd64} +GEAN_IMG=${GEAN_IMG:-ghcr.io/geanlabs/gean:devnet5} +LANTERN_IMG=${LANTERN_IMG:-bitminemavan/lantern:devnet5} +GRANDINE_IMG=${GRANDINE_IMG:-sifrai/lean:devnet-5} + +CLIENTS="ethlambda zeam ream qlean gean lantern grandine" + +# map node index -> canary client (empty = ethlambda) +declare -A CANARY +for spec in "$@"; do + n=${spec%%:*}; client=${spec#*:} + case "$n" in ''|*[!0-9]*) log "ERROR: bad node index in spec '$spec'"; exit 1;; esac + case " $CLIENTS " in *" $client "*) ;; *) log "ERROR: unknown client '$client' in spec '$spec' (want: $CLIENTS)"; exit 1;; esac + CANARY[$n]=$client +done + +# map node index -> aggregator (subnet covered is always index % SUBNETS) +declare -A IS_AGG +AGG_NODES=${AGG_NODES:-$(seq 0 $((SUBNETS-1)))} +declare -A SEEN_SUBNET +for n in $AGG_NODES; do + case "$n" in ''|*[!0-9]*) log "ERROR: bad node index '$n' in AGG_NODES"; exit 1;; esac + [ "$n" -lt "$NODES" ] || { log "ERROR: AGG_NODES has node $n but NODES=$NODES"; exit 1; } + s=$((n % SUBNETS)) + # Two aggregators on one subnet leaves another with none: attestations there + # verify but nobody stores them, blocks go out with attestation_count=0 and + # finality dies quietly. Cheaper to catch here than in Grafana an hour later. + [ -z "${SEEN_SUBNET[$s]:-}" ] || { + log "ERROR: AGG_NODES has both node ${SEEN_SUBNET[$s]} and node $n on subnet $s" + log " (subnet = index % SUBNETS), which leaves some subnet unaggregated." + exit 1; } + SEEN_SUBNET[$s]=$n; IS_AGG[$n]=1 +done +for s in $(seq 0 $((SUBNETS-1))); do + [ -n "${SEEN_SUBNET[$s]:-}" ] || { log "ERROR: no aggregator for subnet $s in AGG_NODES=[$AGG_NODES]"; exit 1; } +done + +# --- preflight: the two mistakes that cost a whole devnet --------------------- +cfg_val(){ sudo awk -v k="$1:" '$1==k{print $2}' "$GENESIS/config.yaml" 2>/dev/null; } +GT=$(cfg_val GENESIS_TIME); GACC=$(cfg_val ATTESTATION_COMMITTEE_COUNT) +[ -n "$GT" ] || { log "ERROR: no GENESIS_TIME in $GENESIS/config.yaml -- is the genesis shipped?"; exit 1; } +# SUBNETS drives the aggregator set and every client's --attestation-committee-count. +# If it disagrees with the genesis the chain was built with, nodes compute a different +# subnet map than the aggregators listen on and votes vanish. +if [ -n "$GACC" ] && [ "$GACC" != "$SUBNETS" ]; then + log "ERROR: SUBNETS=$SUBNETS but the genesis says ATTESTATION_COMMITTEE_COUNT=$GACC." + log " Pass $GACC (or ship the genesis you meant to run)." + exit 1 +fi +# A node that resumes an old DB against a NEW genesis forks onto its own chain +# (the genesis-time landmine). Fresh start => data dirs must be empty. +stale=$(sudo find "$DATA" -mindepth 2 -maxdepth 2 -path "$DATA/node_*" -print -quit 2>/dev/null) +if [ -n "$stale" ] && [ -z "${ALLOW_STALE_DATA:-}" ]; then + log "ERROR: leftover node data under $DATA (e.g. $stale)." + log " Wipe it first: sudo rm -rf $DATA/node_*" + log " Do that BEFORE generating genesis -- it is the one step that scales with chain" + log " age, so it must not run against the GENESIS_TIME countdown." + log " To relaunch on the EXISTING chain instead, use cs-restart.sh (checkpoint sync)," + log " or override with ALLOW_STALE_DATA=1 if you know the data matches this genesis." + exit 1 +fi +gt_delta=$((GT - $(date +%s))) +if [ "$gt_delta" -lt -3600 ]; then + log "WARNING: GENESIS_TIME is $(( -gt_delta / 60 ))min in the PAST with empty data dirs --" + log " nodes will boot thousands of slots behind and sit in the sync gate. Stale genesis dir?" +fi + +log "=== start-devnet NODES=$NODES SUBNETS=$SUBNETS canaries=[$*] aggregators=[$(echo $AGG_NODES)] GT=$GT (genesis in ${gt_delta}s) ===" + +for n in $(seq 0 $((NODES-1))); do + G=$((9000+n)); A=$((5052+n)); M=$((9200+n)) + client=${CANARY[$n]:-ethlambda} + NAME=node_$n # node identity: node-id, node_N.key, data-dir (client-agnostic) + CNAME=${client}_$n # container / display name (docker, cAdvisor `name`, promtail `node`) + # remove any prior container for this node index, whatever client it ran + cids=$(sudo docker ps -aq --filter "name=_${n}$"); [ -n "$cids" ] && sudo docker rm -f $cids >/dev/null 2>&1 + sudo mkdir -p "$DATA/$NAME" + # aggregator per $AGG_NODES; subnet covered = the one its own validators attest on + aggflags=""; lim="$MEM"; sub=$((n % SUBNETS)) + if [ -n "${IS_AGG[$n]:-}" ]; then + lim="$MEM_AGG"; aggflags="--is-aggregator" + [ "$client" != qlean ] && aggflags="$aggflags --aggregate-subnet-ids $sub" + fi + case "$client" in + ethlambda) + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$ETH_IMG" \ + --genesis /config/config.yaml --validators /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml --validator-config /config/validator-config.yaml \ + --hash-sig-keys-dir /config/hash-sig-keys --data-dir /data \ + --gossipsub-port "$G" --node-id "$NAME" --node-key /config/"$NAME".key \ + --http-address 0.0.0.0 --api-port "$A" --metrics-port "$M" \ + --attestation-committee-count "$SUBNETS" $aggflags $EXTRA_ETH_FLAGS >/dev/null \ + && log "$NAME ethlambda started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME ethlambda";; + zeam) + # console_log_level=info exposes zeam's own duty publishes (attestation / + # block) on stdout so the duty-timing profiler (reads `docker logs`) can see + # them; the file sink stays warn (not captured). Costs more stdout volume, + # bounded here by the json-log rotation in $LOGOPT. + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + --security-opt seccomp=unconfined \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$ZEAM_IMG" \ + --log_file_active_level warn --console_log_level info \ + node --custom-genesis /config --validator-config genesis_bootnode --data-dir /data \ + --node-id "$NAME" --node-key /config/"$NAME".key --metrics-enable \ + --api-port "$A" --metrics-port "$M" --attestation-committee-count "$SUBNETS" \ + $aggflags --db-backend lmdb >/dev/null \ + && log "$NAME zeam started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME zeam";; + ream) + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$REAM_IMG" \ + --data-dir /data lean_node --network /config/config.yaml \ + --validator-registry-path /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml \ + --node-id "$NAME" --node-key /config/"$NAME".key --socket-port "$G" \ + --metrics --metrics-address 0.0.0.0 --metrics-port "$M" \ + --http-address 0.0.0.0 --http-port "$A" \ + --attestation-committee-count "$SUBNETS" $aggflags >/dev/null \ + && log "$NAME ream started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME ream";; + qlean) + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$QLEAN_IMG" \ + --genesis-dir /config --data-dir /data --node-id "$NAME" \ + --node-key /config/"$NAME".key \ + --listen-addr /ip4/0.0.0.0/udp/"$G"/quic-v1 \ + --metrics-host 0.0.0.0 --metrics-port "$M" \ + --api-host 0.0.0.0 --api-port "$A" \ + --attestation-committee-count "$SUBNETS" $aggflags -linfo >/dev/null \ + && log "$NAME qlean started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME qlean";; + gean) + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$GEAN_IMG" \ + --custom-network-config-dir /config --data-dir /data \ + --gossipsub-port "$G" --node-id "$NAME" --node-key /config/"$NAME".key \ + --http-address 0.0.0.0 --api-port "$A" --metrics-port "$M" \ + --attestation-committee-count "$SUBNETS" $aggflags >/dev/null \ + && log "$NAME gean started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME gean";; + lantern) + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$LANTERN_IMG" \ + --data-dir /data --genesis-config /config/config.yaml \ + --nodes-path /config/nodes.yaml --genesis-state /config/genesis.ssz \ + --validator_config /config \ + --node-id "$NAME" --node-key-path /config/"$NAME".key \ + --listen-address /ip4/0.0.0.0/udp/"$G"/quic-v1 \ + --http-port "$A" --metrics-port "$M" \ + --hash-sig-key-dir /config/hash-sig-keys --attestation-committee-count "$SUBNETS" \ + --devnet 12345678 $aggflags >/dev/null \ + && log "$NAME lantern started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME lantern";; + grandine) + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $lim $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$GRANDINE_IMG" \ + --genesis /config/config.yaml --validator-registry-path /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml --node-id "$NAME" --node-key /config/"$NAME".key \ + --port "$G" --address 0.0.0.0 --http-address 0.0.0.0 --http-port "$A" \ + --metrics --metrics-address 0.0.0.0 --metrics-port "$M" \ + --hash-sig-key-dir /config/hash-sig-keys --attestation-committee-count "$SUBNETS" \ + $aggflags >/dev/null \ + && log "$NAME grandine started${aggflags:+ (agg subnet $sub)}" || log "FAIL $NAME grandine";; + *) log "ERROR unknown client '$client' for $NAME";; + esac +done +log "=== all $NODES nodes launched ===" diff --git a/.claude/skills/multi-server-devnet/scripts/start-observability.sh b/.claude/skills/multi-server-devnet/scripts/start-observability.sh new file mode 100755 index 00000000..f6786c93 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/start-observability.sh @@ -0,0 +1,85 @@ +#!/bin/bash +# Per-host: (re)launch this devnet's metrics scrapers. +# +# prometheus scrapes the devnet's nodes (172.17.0.1:9200+n) + node_exporter + +# cadvisor from /opt/lean-quickstart/observability/prometheus.yml +# (generate it with prometheus-config.sh) and remote-writes to the +# central Prometheus. Local retention is short on purpose: the +# central prom is the long-term store. +# cadvisor per-container CPU + memory on :9098 (per-container network/fs is +# meaningless under --network host -- use node_exporter for those). +# +# node_exporter is NOT here: it is a systemd service installed from +# lambdaclass/monitoring-stack (binds :9122), not a container. +# Log shipping is start-promtail.sh. The CENTRAL prometheus/grafana/loki stack is +# a separate one-time deployment (see references/operations.md). +# +# Safe on a live devnet: neither container is a gossip peer, so no 60s backoff. +# +# Usage (on host): start-observability.sh +# RECREATE=1 replace containers that already exist. Without it an existing +# prometheus/cadvisor is LEFT ALONE: it may have been created with +# different flags (retention, ports), and silently rewriting a live +# scraper's configuration is not something to do as a side effect. +# To pick up a prometheus.yml change you only need +# `sudo docker restart prometheus` (single-file bind mount -- see +# the inode trap in references/operations.md). +# PROM_IMG / CADVISOR_IMG / PROM_PORT / CADVISOR_PORT / PROM_RETENTION +set -u +OBS=${OBS:-/opt/lean-quickstart/observability} +PROM_IMG=${PROM_IMG:-prom/prometheus:v3.1.0} +CADVISOR_IMG=${CADVISOR_IMG:-gcr.io/cadvisor/cadvisor:v0.49.1} +PROM_PORT=${PROM_PORT:-9090} +CADVISOR_PORT=${CADVISOR_PORT:-9098} +PROM_RETENTION=${PROM_RETENTION:-2d} +CFG="$OBS/prometheus.yml" + +exists(){ [ -n "$(sudo docker ps -aq --filter "name=^${1}$")" ]; } +keep_or_replace(){ # $1 = container name -> 0 = go ahead and (re)create + if exists "$1"; then + if [ -z "${RECREATE:-}" ]; then + echo "SKIP $1: already exists (RECREATE=1 to replace; 'docker restart $1' to reload its config)" + return 1 + fi + sudo docker rm -f "$1" >/dev/null 2>&1 + fi + return 0 +} + +if [ ! -f "$CFG" ]; then + echo "ERROR: $CFG missing -- generate it first:" + echo " prometheus-config.sh NETWORK NODES HOST_IP CENTRAL_WRITE_URL [N:client ...] | ssh host 'sudo tee $CFG'" + exit 1 +fi +sudo mkdir -p "$OBS/prometheus-data" + +if keep_or_replace prometheus; then + # --network host so it can reach the nodes' metrics ports, the host's + # node_exporter, and the central prometheus over the same routes the operator uses. + sudo docker run -d --restart unless-stopped --name prometheus --network host \ + --memory 2g --memory-swap 4g --log-opt max-size=50m --log-opt max-file=3 \ + -v "$CFG":/etc/prometheus/prometheus.yml \ + -v "$OBS/prometheus-data":/prometheus \ + "$PROM_IMG" \ + --config.file=/etc/prometheus/prometheus.yml \ + --storage.tsdb.path=/prometheus \ + --storage.tsdb.retention.time="$PROM_RETENTION" \ + --web.listen-address=":$PROM_PORT" >/dev/null \ + && echo "prometheus started ($PROM_IMG, :$PROM_PORT, retention $PROM_RETENTION)" \ + || echo "FAIL to start prometheus" +fi + +if keep_or_replace cadvisor; then + sudo docker run -d --restart unless-stopped --name cadvisor \ + -p "$CADVISOR_PORT":8080 --privileged --device /dev/kmsg \ + --memory 1g --memory-swap 2g --log-opt max-size=50m --log-opt max-file=3 \ + -v /:/rootfs:ro -v /var/run:/var/run:ro -v /sys:/sys:ro \ + -v /var/lib/docker/:/var/lib/docker:ro -v /dev/disk/:/dev/disk:ro \ + "$CADVISOR_IMG" >/dev/null \ + && echo "cadvisor started ($CADVISOR_IMG, :$CADVISOR_PORT)" \ + || echo "FAIL to start cadvisor" +fi + +echo +echo "verify: curl -s 127.0.0.1:$PROM_PORT/api/v1/targets | grep -o '\"health\":\"[a-z]*\"' | sort | uniq -c" +echo " (every node job should be 'up'; check job/client_type labels per scrapeUrl)" diff --git a/.claude/skills/multi-server-devnet/scripts/start-promtail.sh b/.claude/skills/multi-server-devnet/scripts/start-promtail.sh new file mode 100755 index 00000000..b127fbf3 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/start-promtail.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# Per-host: (re)launch the promtail log shipper. Reads /opt/lean-quickstart/ +# observability/promtail.yml (push the generated file there first with +# promtail-config.sh | ssh ... tee) and ships node-container logs to the central +# Loki. Safe to run on a live devnet: promtail is NOT a gossip peer, so there is +# no 60s backoff and no devnet impact — it only reads the Docker socket. +# +# Usage (on host): start-promtail.sh +# PROMTAIL_IMG env overrides the image (default pinned to the Loki version). +set -u +OBS=/opt/lean-quickstart/observability +IMG=${PROMTAIL_IMG:-grafana/promtail:3.4.2} +CFG="$OBS/promtail.yml" + +if [ ! -f "$CFG" ]; then echo "ERROR: $CFG missing (generate + push it first)"; exit 1; fi +sudo mkdir -p "$OBS/promtail-data" + +sudo docker rm -f promtail >/dev/null 2>&1 +# --user root: read the docker socket. Cap promtail's own logs too. +sudo docker run -d --restart unless-stopped --name promtail --network host --user root \ + --memory 512m --memory-swap 1g --memory-reservation 128m \ + --log-opt max-size=50m --log-opt max-file=3 \ + -v "$CFG":/etc/promtail/config.yml:ro \ + -v "$OBS/promtail-data":/promtail-data \ + -v /var/run/docker.sock:/var/run/docker.sock:ro \ + "$IMG" -config.file=/etc/promtail/config.yml >/dev/null \ + && echo "promtail started ($IMG)" || echo "FAIL to start promtail" diff --git a/.claude/skills/multi-server-devnet/scripts/start-range.sh b/.claude/skills/multi-server-devnet/scripts/start-range.sh new file mode 100755 index 00000000..d6f83eb9 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/start-range.sh @@ -0,0 +1,87 @@ +#!/bin/bash +# HOST-side: launch an ethlambda node RANGE of a CROSS-SERVER devnet (fresh start, +# no checkpoint sync -- every node boots from the genesis already in /config). +# The skill's start-devnet.sh assumes one host owns nodes 0..NODES-1; a chain split +# across hosts needs each host to own a slice, and aggregator->subnet mapping is no +# longer "node k -> subnet k" (eth-5 owns nodes 32-63 but subnets 4-7), so the +# aggregators are passed explicitly. +# +# Usage (on host): start-range.sh [N:subnet ...] +# START END inclusive node index range this host owns (e.g. 0 31 / 32 63) +# SUBNETS ATTESTATION_COMMITTEE_COUNT (checked against the genesis on disk) +# N:subnet node N aggregates subnet `subnet` (e.g. 0:0 1:1 / 36:4 37:5) +# +# Ports: gossip 9000+n, api 5052+n, metrics 9200+n (global index, no per-host base). +# Naming: identity node_N (node-id, node_N.key, data-dir, genesis validator name); +# container ethlambda_N (docker name, cAdvisor `name`, promtail `node`). +set -u +START=$1; END=$2; SUBNETS=$3; shift 3 + +GENESIS=${GENESIS:-/opt/lean-quickstart/genesis} +DATA=${DATA:-/opt/lean-quickstart/data} +ETH_IMG=${ETH_IMG:-ghcr.io/lambdaclass/ethlambda:devnet5} +MEM="${MEM:---memory 8g --memory-swap 16g --memory-reservation 2g}" +LOGOPT="${LOGOPT:---log-opt max-size=200m --log-opt max-file=5}" +EXTRA_ETH_FLAGS=${EXTRA_ETH_FLAGS:-} +LOG=/tmp/start-range.log +log(){ echo "$(date '+%H:%M:%S') $*" | tee -a "$LOG"; } +: > "$LOG" + +# map node index -> aggregated subnet id +declare -A AGG +for spec in "$@"; do + n=${spec%%:*}; s=${spec#*:} + case "$n" in ''|*[!0-9]*) log "ERROR: bad node index in agg spec '$spec'"; exit 1;; esac + case "$s" in ''|*[!0-9]*) log "ERROR: bad subnet in agg spec '$spec'"; exit 1;; esac + [ "$n" -ge "$START" ] && [ "$n" -le "$END" ] || { log "ERROR: agg node $n outside range $START-$END"; exit 1; } + [ "$s" -lt "$SUBNETS" ] || { log "ERROR: subnet $s >= SUBNETS=$SUBNETS"; exit 1; } + AGG[$n]=$s +done + +# --- preflight (same failures start-devnet.sh guards, adapted to a range) ------ +cfg_val(){ sudo awk -v k="$1:" '$1==k{print $2}' "$GENESIS/config.yaml" 2>/dev/null; } +GT=$(cfg_val GENESIS_TIME); GACC=$(cfg_val ATTESTATION_COMMITTEE_COUNT) +GVC=$(cfg_val VALIDATOR_COUNT) +[ -n "$GT" ] || { log "ERROR: no GENESIS_TIME in $GENESIS/config.yaml -- is the genesis shipped?"; exit 1; } +if [ -n "$GACC" ] && [ "$GACC" != "$SUBNETS" ]; then + log "ERROR: SUBNETS=$SUBNETS but genesis says ATTESTATION_COMMITTEE_COUNT=$GACC"; exit 1 +fi +if [ -n "$GVC" ] && [ "$END" -ge "$GVC" ]; then + log "ERROR: END=$END but genesis only has VALIDATOR_COUNT=$GVC"; exit 1 +fi +# pubkey size must match the binary's PUBLIC_KEY_SIZE or the node dies parsing genesis +# (52B = pre-leanVM-main / leanSig, 64 hex; 32B = leanVM-main, 104 hex incl. quotes). +PKHEX=$(sudo grep -m1 -o 'attestation_pubkey: "[0-9a-f]*"' "$GENESIS/config.yaml" 2>/dev/null | sed 's/.*"\(.*\)"/\1/') +log "genesis pubkey size: $(( ${#PKHEX} / 2 )) bytes (image $ETH_IMG must agree)" +for n in $(seq "$START" "$END"); do + if [ -n "$(sudo find "$DATA/node_$n" -mindepth 1 -maxdepth 1 -print -quit 2>/dev/null)" ] \ + && [ -z "${ALLOW_STALE_DATA:-}" ]; then + log "ERROR: leftover data in $DATA/node_$n -- a node resuming an old DB against a NEW" + log " genesis forks onto its own chain. Wipe first: sudo rm -rf $DATA/node_*" + log " (override with ALLOW_STALE_DATA=1 only if the data matches THIS genesis)" + exit 1 + fi +done +gt_delta=$((GT - $(date +%s))) +[ "$gt_delta" -lt -3600 ] && log "WARNING: GENESIS_TIME is $(( -gt_delta / 60 ))min in the PAST with empty data dirs" + +log "=== start-range $START-$END SUBNETS=$SUBNETS aggs=[$*] IMG=$ETH_IMG GT=$GT (in ${gt_delta}s) ===" + +for n in $(seq "$START" "$END"); do + G=$((9000+n)); A=$((5052+n)); M=$((9200+n)) + NAME=node_$n; CNAME=ethlambda_$n + cids=$(sudo docker ps -aq --filter "name=_${n}$"); [ -n "$cids" ] && sudo docker rm -f $cids >/dev/null 2>&1 + sudo mkdir -p "$DATA/$NAME" + aggflags="" + if [ -n "${AGG[$n]:-}" ]; then aggflags="--is-aggregator --aggregate-subnet-ids ${AGG[$n]}"; fi + sudo docker run -d --restart unless-stopped --name "$CNAME" --network host $MEM $LOGOPT \ + -v "$GENESIS":/config -v "$DATA/$NAME":/data "$ETH_IMG" \ + --genesis /config/config.yaml --validators /config/annotated_validators.yaml \ + --bootnodes /config/nodes.yaml --validator-config /config/validator-config.yaml \ + --hash-sig-keys-dir /config/hash-sig-keys --data-dir /data \ + --gossipsub-port "$G" --node-id "$NAME" --node-key /config/"$NAME".key \ + --http-address 0.0.0.0 --api-port "$A" --metrics-port "$M" \ + --attestation-committee-count "$SUBNETS" $aggflags $EXTRA_ETH_FLAGS >/dev/null \ + && log "$NAME started${aggflags:+ (agg subnet ${AGG[$n]})}" || log "FAIL $NAME" +done +log "=== launched $((END-START+1)) nodes; aggregators: ${!AGG[*]} ===" diff --git a/.claude/skills/multi-server-devnet/scripts/subnet-align-validators.py b/.claude/skills/multi-server-devnet/scripts/subnet-align-validators.py new file mode 100644 index 00000000..b12b8ba5 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/subnet-align-validators.py @@ -0,0 +1,156 @@ +#!/usr/bin/env python3 +"""Rewrite annotated_validators.yaml so each node's validators share one subnet. + +WHY: the client derives a validator's attestation subnet from its own global +index (`vid % ACC`, see p2p::attestation_subscription_subnets) -- the `subnet` +field in validator-config.yaml is not consulted. lean-quickstart's +generate-genesis.sh hands node i the CONTIGUOUS block [i*vpn, (i+1)*vpn), so at +vpn == ACC every node ends up owning exactly one validator in every subnet and +therefore subscribes to all of them. Permuting the key files cannot fix that: +the subnet is a property of the index, not of the key sitting at it. + +So we reassign which indices each node owns: + + s = i % S subnet of node i (round-robin, so the + aggregators 0..S-1 stay one-per-subnet) + g = i // S which block of nodes within that subnet + indices = [s + S*(g*vpn + k) for k in range(vpn)] + +Every index in that set is congruent to s mod S, so the node holds validators +from subnet s only and subscribes to exactly that one subnet. The map is a +bijection onto 0..N*vpn-1 (index = s + S*m with m = g*vpn + k covering 0..N*vpn/S-1 +once per s), and it degenerates to the identity when vpn == 1. + +Rebuilt from the key manifest rather than by permuting the existing file, then +cross-checked against config.yaml's GENESIS_VALIDATORS so an index/pubkey skew +fails loudly instead of producing a chain whose signatures never verify. + +Usage: subnet-align-validators.py GENESIS_DIR NODES SUBNETS VALIDATORS_PER_NODE +""" +import sys +from collections import defaultdict +from pathlib import Path + +import yaml + + +class HexSafeLoader(yaml.SafeLoader): + """SafeLoader that does not coerce `0x...` scalars to int. + + The key manifest stores pubkeys as unquoted `0x`, and PyYAML's YAML 1.1 + int resolver matches that -- turning a pubkey into an int and breaking the + comparison against config.yaml's quoted string form. yq (YAML 1.2, used by + generate-genesis.sh) keeps it a string, so this loader matches yq. + """ + + +HexSafeLoader.yaml_implicit_resolvers = { + key: [(tag, regexp) for tag, regexp in resolvers + if tag != "tag:yaml.org,2002:int"] + for key, resolvers in yaml.SafeLoader.yaml_implicit_resolvers.items() +} + + +def main() -> int: + if len(sys.argv) != 5: + print(__doc__) + return 2 + gdir = Path(sys.argv[1]) + nodes, subnets, vpn = (int(x) for x in sys.argv[2:5]) + total = nodes * vpn + + if nodes % subnets: + print(f"ERROR: NODES ({nodes}) must be divisible by SUBNETS ({subnets}) " + "for an equal-sized, subnet-homogeneous split") + return 1 + + manifest = yaml.load( + (gdir / "hash-sig-keys" / "validator-keys-manifest.yaml").read_text(), + Loader=HexSafeLoader) + entries = {int(v["index"]): v for v in manifest["validators"]} + if len(entries) < total: + print(f"ERROR: manifest has {len(entries)} validators, need {total}") + return 1 + + # --- assignment --------------------------------------------------------- + assignment: dict[str, list[int]] = {} + for i in range(nodes): + s, g = i % subnets, i // subnets + assignment[f"node_{i}"] = [s + subnets * (g * vpn + k) for k in range(vpn)] + + # bijection + homogeneity checks + flat = [idx for idxs in assignment.values() for idx in idxs] + if sorted(flat) != list(range(total)): + print("ERROR: assignment is not a bijection onto 0..%d" % (total - 1)) + return 1 + for name, idxs in assignment.items(): + if len({idx % subnets for idx in idxs}) != 1: + print(f"ERROR: {name} spans multiple subnets: {idxs}") + return 1 + + # --- cross-check index -> pubkey against config.yaml -------------------- + cfg = yaml.safe_load((gdir / "config.yaml").read_text()) + gvs = cfg["GENESIS_VALIDATORS"] + if len(gvs) != total: + print(f"ERROR: config.yaml has {len(gvs)} GENESIS_VALIDATORS, need {total}") + return 1 + for idx in range(total): + want_att = entries[idx]["attester_key_pubkey_hex"].removeprefix("0x").lower() + want_pro = entries[idx]["proposer_key_pubkey_hex"].removeprefix("0x").lower() + got_att = str(gvs[idx]["attestation_pubkey"]).removeprefix("0x").lower() + got_pro = str(gvs[idx]["proposal_pubkey"]).removeprefix("0x").lower() + if (want_att, want_pro) != (got_att, got_pro): + print(f"ERROR: index {idx} pubkey mismatch between manifest and " + f"config.yaml\n manifest att={want_att}\n config att={got_att}") + return 1 + + # --- emit --------------------------------------------------------------- + per_subnet: dict[int, list[str]] = defaultdict(list) + lines = [ + "# node -> validator assignment, SUBNET-HOMOGENEOUS.", + f"# {nodes} nodes x {vpn} validators = {total} validators over {subnets} subnets.", + "# Node i owns only validators of subnet i % " + f"{subnets} (indices s + {subnets}*(g*{vpn}+k)), so it subscribes to that", + "# one subnet instead of all of them. Generated by subnet-align-validators.py;", + "# do not hand-edit -- regenerate.", + "", + ] + for i in range(nodes): + name = f"node_{i}" + idxs = assignment[name] + per_subnet[i % subnets].append(name) + lines.append(f"{name}: # subnet {i % subnets}, validators {idxs}") + for idx in idxs: + e = entries[idx] + for role in ("attester", "proposer"): + pk = e[f"{role}_key_pubkey_hex"].removeprefix("0x") + lines.append(f" - index: {idx}") + lines.append(f" pubkey_hex: {pk}") + lines.append(f" privkey_file: {e[f'{role}_key_privkey_file']}") + lines.append("") + out = gdir / "annotated_validators.yaml" + out.write_text("\n".join(lines)) + + # Re-parse to prove the client's serde_yaml_ng will accept the shape. + back = yaml.safe_load(out.read_text()) + assert len(back) == nodes, f"{len(back)} nodes in output, expected {nodes}" + for name, idxs in assignment.items(): + got = back[name] + assert len(got) == 2 * vpn, f"{name}: {len(got)} entries, want {2 * vpn}" + assert sorted({e["index"] for e in got}) == sorted(idxs) + atts = [e for e in got if "attester" in e["privkey_file"]] + pros = [e for e in got if "proposer" in e["privkey_file"]] + assert len(atts) == len(pros) == vpn, f"{name}: role split wrong" + + print(f"annotated_validators.yaml rewritten: {nodes} nodes x {vpn} validators") + for s in sorted(per_subnet): + members = per_subnet[s] + print(f" subnet {s}: {len(members)} nodes ({members[0]}..{members[-1]}), " + f"{len(members) * vpn} validators") + print(f" node_0 -> {assignment['node_0']} node_1 -> {assignment['node_1']}") + print(f" node_{nodes - 1} -> {assignment[f'node_{nodes - 1}']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.claude/skills/multi-server-devnet/scripts/sweep.sh b/.claude/skills/multi-server-devnet/scripts/sweep.sh new file mode 100755 index 00000000..53e399fc --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/sweep.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# Run from the OPERATOR machine. Audit ALL independent devnets at once via the +# central Prometheus (the one Grafana reads): per devnet `network`, prints head / +# justified / finalized slot and the client mix. This is the fast cross-devnet +# health check — one query, no per-node ssh. +# +# Env: +# CENTRAL_PROM_URL central Prometheus base, e.g. http://10.0.0.4:9099 (required; +# normally comes from devnet.env, see devnet.env.example) +# +# For a single devnet's per-node detail (running state, stuck node), ssh the host +# and read /tmp/start-devnet.log + curl 127.0.0.1:$((9200+n))/metrics directly. +set -u +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/devnet-env.sh" +devnet_load_env || exit 1 +P=${CENTRAL_PROM_URL:?set CENTRAL_PROM_URL to the central Prometheus base url} + +q(){ curl -s -m 5 "$P/api/v1/query" --data-urlencode "query=$1"; } + +for metric in lean_head_slot lean_latest_justified_slot lean_latest_finalized_slot; do + echo "$metric:" + q "max by (network) ($metric)" | python3 -c ' +import json,sys +rs=json.load(sys.stdin).get("data",{}).get("result",[]) +for r in sorted(rs, key=lambda x: x["metric"].get("network","")): + print(" ", r["metric"].get("network","?"), r["value"][1])' +done + +echo "client mix per devnet (running app targets):" +q 'count by (network, client_type) (up{type="app"} == 1)' | python3 -c ' +import json,sys +rows={} +for r in json.load(sys.stdin).get("data",{}).get("result",[]): + m=r["metric"]; rows.setdefault(m.get("network","?"),{})[m.get("client_type","?")]=r["value"][1] +for net in sorted(rows): print(" ", net, rows[net])' diff --git a/.claude/skills/multi-server-devnet/scripts/teardown.sh b/.claude/skills/multi-server-devnet/scripts/teardown.sh new file mode 100755 index 00000000..cda0dc19 --- /dev/null +++ b/.claude/skills/multi-server-devnet/scripts/teardown.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# HOST-side: retire this host's slice of a cross-server devnet, in the order that +# keeps the GENESIS_TIME countdown honest (the data wipe is the step that scales +# with chain age, so it runs BEFORE the operator stamps the new genesis). +# +# Usage (on host): teardown.sh +# Env: KEEP_GENESIS=1 (default) archives /opt/lean-quickstart/genesis to +# genesis.bak- instead of deleting it -- that dir holds the retired +# chain's hash-sig keys, which are deliberately preserved for reuse. +# STAGED=/opt/lean-quickstart/genesis-new is promoted into place if present. +set -u +LQ=${LQ:-/opt/lean-quickstart} +GENESIS=$LQ/genesis +STAGED=${STAGED:-$LQ/genesis-new} +DATA=$LQ/data +TS=$(date +%Y%m%d-%H%M%S) +log(){ echo "$(date '+%H:%M:%S') $*"; } + +# 1. containers: `--restart unless-stopped` respawns them, so clear the policy first +cids=$(sudo docker ps -aq --filter "name=_[0-9]" | tr '\n' ' ') +if [ -n "$cids" ]; then + log "clearing restart policy on $(echo $cids | wc -w) containers" + for c in $cids; do sudo docker update --restart=no "$c" >/dev/null; done + log "removing containers" + sudo docker rm -f $cids >/dev/null +else + log "no node containers found" +fi + +# 2. archive the retired genesis (holds the old chain's hash-sig keys -- KEEP) +if [ -d "$GENESIS" ]; then + sudo mv "$GENESIS" "$GENESIS.bak-$TS" + log "archived genesis -> $GENESIS.bak-$TS ($(sudo du -sh "$GENESIS.bak-$TS" | cut -f1), keys kept)" +fi + +# 3. promote the staged genesis (config.yaml GT is re-stamped + reshipped after the wipe) +if [ -d "$STAGED" ]; then + sudo mv "$STAGED" "$GENESIS" + PK=$(sudo grep -m1 -o 'attestation_pubkey: "[0-9a-f]*"' "$GENESIS/config.yaml" | sed 's/.*"\(.*\)"/\1/') + log "promoted staged genesis (pubkey $(( ${#PK} / 2 ))B, GT $(sudo awk '/GENESIS_TIME:/{print $2}' "$GENESIS/config.yaml") -- stale until reship)" +fi + +# 4. the slow part: wipe every node DB so nothing resumes an old chain +if [ -d "$DATA" ]; then + log "wiping $(sudo du -sh "$DATA" | cut -f1) of node data..." + sudo rm -rf "$DATA"/node_* + log "wiped; remaining: $(sudo du -sh "$DATA" 2>/dev/null | cut -f1)" +fi +log "=== teardown complete; ready for genesis reship + start-range.sh ===" diff --git a/.gitignore b/.gitignore index eb1f3df5..b3f89904 100644 --- a/.gitignore +++ b/.gitignore @@ -39,3 +39,10 @@ devnet.log # mdbook build output book/ + +# Slack webhook consumed by the devnet finality alert deploy script (a secret) +webhook.txt +finality-alerts.yaml + +# Filled-in copy of the multi-server-devnet deployment env (names hosts/paths) +devnet.env