feat(tooling): add multi-server-devnet skill - #565
Conversation
Captures the operating procedure for the federated devnets (one independent single-host devnet per server, federated into one Grafana), which until now lived only on the hosts: genesis generation, per-host prometheus/promtail config emitters, client conversion, checkpoint-sync restarts, finality alert. Vendors the four provisioned Grafana dashboards so the repo is their source of truth. Their directory is bind-mounted read-only into the container, so UI edits get reverted by the provisioner and nothing syncs server-side changes back; resources-dashboard.json had already drifted three panels behind the deployed copy. Nothing about a particular deployment is committed. Hosts, urls and Grafana ids live in a gitignored devnet.env (devnet.env.example documents every key); the dashboards resolve their datasource through a template variable instead of a pinned uid, so they drop into any Grafana unedited; and the finality alert's Slack webhook, Grafana base url and prometheus datasource uid are substituted at deploy time, since a provisioned alert rule cannot use a datasource variable. webhook.txt and devnet.env are gitignored so the webhook can't be committed by accident.
Halve the client dashboard's Overview stat and slot-graph rows so the
finality-delay graphs sit closer to the fold, and reframe both start-time
panels around the question they are actually asked: how long has this been
running?
- "Latest start time" -> "Oldest node start". max() reported the *newest*
process start, so a single restarted node masked a fleet that had been up
for days; min() gives the devnet's uninterrupted age instead. Rendered with
dateTimeFromNow ("6 days ago") rather than an ISO timestamp the reader has
to subtract by hand.
- "Start time" -> "Node start times", timeseries -> table. changes(...[1m])
drew flat zeros whenever nothing had restarted recently, and its spike is
only as wide as the range window, so Grafana's step aliased the event away
at any dashboard range past ~1h: the panel read "no restarts" whether or
not restarts had occurred. A per-node table of start times, newest first,
cannot hide one. Also replaces that panel's description, which was a
copy-paste of the processed-slots text.
Both panels carry a note that only ethlambda exports
lean_node_start_time_seconds, so converted canaries never appear.
Replace the per-node start-time table with a stat panel that draws one tile per node, so a restart is visible at a glance instead of requiring the reader to compare timestamps down a column. The tile value is an age (time() - lean_node_start_time_seconds) rather than the start timestamp: thresholds compare a field's own value, so a tile holding an epoch timestamp could only be coloured against a fixed calendar date, which goes stale immediately. Age makes the bands meaningful and permanent -- red under 15m, orange under 1h, yellow under 6h, dark green beyond -- so a restarted node lights up and decays back to calm over six hours. Trade-off: Grafana cannot sort stat tiles, so ordering follows the query's series order. Colour, not position, is what surfaces the event.
🤖 Kimi Code Review
This PR adds operational tooling (documentation, shell scripts, Grafana dashboards) for managing multi-server devnets. Since this is infrastructure code rather than core consensus logic, the review focuses on operational safety, script correctness, and documentation maintainability. Critical Issues1. Fragile Line Number References in DocumentationFiles: Specific line number references to Rust source files will become stale when code changes, leading to operational confusion:
Recommendation: Replace line numbers with function names and module paths only (e.g., 2. Unsafe File Deletion in agg-restart.shFile: sudo rm -rf "$DATA/$name"/* 2>/dev/nullIssues:
Recommendation: # Validate path before deletion
if [[ -d "$DATA/$name" && "$name" == node_* ]]; then
sudo find "$DATA/$name" -mindepth 1 -delete
else
log "ERROR: Invalid data directory $DATA/$name"
exit 1
fi3. Unquoted Variables in Subshell CommandsFile: [ -n "$cid" ] && sudo sh -c "docker logs $cid > '$CRASH/node_${n}-${TS}.log' 2>&1"
Recommendation: [ -n "$cid" ] && sudo sh -c "docker logs \"$cid\" > '$CRASH/node_${n}-${TS}.log' 2>&1"Or better, avoid the subshell: [ -n "$cid" ] && sudo docker logs "$cid" > "$CRASH/node_${n}-${TS}.log" 2>&1Security & Safety Concerns4. Missing Error Handling in agg-restart.shFile:
Recommendation: Add health checks: # Verify checkpoint source is healthy before proceeding
if ! curl -sf "http://127.0.0.1:$CS/lean/v0/states/finalized" >/dev/null; then
log "ERROR: Checkpoint sync source on port $CS is not healthy"
exit 1
fi5. Input Validation MissingFile: CS=$1; shift; NODES="$*"No validation that Recommendation: Add validation: if ! [[ "$CS" =~ ^[0-9]+$ ]]; then
echo "ERROR: CS_PORT must be a number"
exit 1
fi
if [[ -z "${IMAGE:-}" ]]; then
echo "ERROR: IMAGE environment variable required"
exit 1
fiOperational Risks6. Gossip Backoff Timing Edge CaseFile: The 60s sleep only occurs if Recommendation: Check container state explicitly: if [ "$existed" = yes ] && sudo docker inspect -f '{{.State.Running}}' "$cid" 2>/dev/null | grep -q "true"; then
log "stopped node_$n container; 60s gossip backoff"
sleep 60
fi7. Dashboard JSON MaintenanceFile: The dashboard contains hardcoded metric names (e.g., Recommendation: Add a comment block at the top of the JSON indicating which version of ethlambda metrics this dashboard was built against, or include a test in CI that validates metric names against the dashboard JSON. Minor Improvements8. Prometheus Query EfficiencyFile: High-cardinality histograms over 10m can be expensive at scale. Consider if 5m is sufficient for the health check use case. 9. Inconsistent Use of sudoFile: The script mixes Recommendation: Add a check at the start: if ! sudo -n docker ps >/dev/null 2>&1; then
echo "ERROR: Requires passwordless sudo access to docker"
exit 1
fi10. Documentation Clarity on Finality ThresholdFile: The finality threshold is mentioned as Positive Aspects
SummaryThis PR provides valuable operational tooling but needs hardening against script failures and documentation rot. The critical fixes needed are:
Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code Review
No consensus Rust paths were changed in this PR, so I did not find fork-choice / state-transition / SSZ / XMSS regressions in the client itself. The meaningful risk here is operational correctness and host security. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
Greptile SummaryThe PR adds an operational skill for running isolated single-host devnets across multiple servers, with lifecycle, genesis, client-conversion, observability, alerting, and audit tooling. One shared configuration-loader defect currently truncates documented multi-host values and can reinterpret configuration as shell syntax.
Confidence Score: 4/5The configuration loader must be fixed before merging because documented multi-server values are truncated, causing fleet operations to silently omit hosts. Removing configuration quotes and evaluating the resulting text changes a space-separated server list into multiple shell words, leaving only the first hostname assigned and also allowing values to be interpreted as shell syntax. Files Needing Attention: .claude/skills/multi-server-devnet/scripts/devnet-env.sh
|
| Filename | Overview |
|---|---|
| .claude/skills/multi-server-devnet/scripts/devnet-env.sh | Adds shared deployment configuration loading, but eval reparses and truncates documented space-containing values. |
| .claude/skills/multi-server-devnet/scripts/start-devnet.sh | Launches isolated devnet nodes with per-client command shapes, aggregator roles, resource limits, and bounded logs. |
| .claude/skills/multi-server-devnet/scripts/make-genesis.sh | Builds per-host genesis artifacts and delegates multi-validator subnet alignment to the new Python helper. |
| .claude/skills/multi-server-devnet/scripts/convert.sh | Implements rolling checkpoint-synced conversion across six alternative clients while retaining node identity. |
| .claude/skills/multi-server-devnet/scripts/promtail-config.sh | Generates Docker-discovered log shipping with normalized labels, backlog filtering, multiline handling, and structured metadata. |
| .claude/skills/multi-server-devnet/scripts/deploy-finality-alert.sh | Renders and remotely provisions the centralized Grafana finality alert from deployment-specific settings. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Env[devnet.env] --> Loader[devnet_load_env]
Loader --> Lifecycle[Lifecycle and audit scripts]
Lifecycle --> Hosts[Independent host devnets]
Hosts --> Metrics[Per-host Prometheus and Promtail]
Metrics --> Central[Central Prometheus, Loki, and Grafana]
Central --> Dashboards[Dashboards and finality alerts]
Prompt To Fix All With AI
### Issue 1
.claude/skills/multi-server-devnet/scripts/devnet-env.sh:39
**Eval breaks environment values**
When the documented `SERVERS="host-a host-b"` value is loaded, stripping its quotes and reparsing it through `eval` leaves only the first host assigned, so fleet operations silently skip subsequent devnets; the same reparsing also executes shell substitutions or separators embedded in values.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat(tooling): flag recent devnet node r..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Restarting a node silently dropped its aggregator role. A node that comes back without --is-aggregator stops storing gossip signatures, so its subnet's votes are never aggregated: attestations still verify, every log line looks healthy, and the devnet quietly stops finalizing with attestation_count=0. The role now travels with the node, read off the container being replaced, with AGG=auto|<id>|off to set it deliberately. The separate aggregator restart script is gone: with one script per role the role handling drifts out of whichever one gets used every day, which is how it got lost in the first place. Launch and convert now validate before destroying anything, since a typo used to surface only after the container and its data were already gone, leaving the node down. start-devnet.sh checks SUBNETS against the genesis' ATTESTATION_COMMITTEE_COUNT (a mismatch makes nodes compute a different subnet map than the aggregators listen on, so votes vanish) and refuses to start on leftover node data, since resuming an old DB against a new genesis forks that node onto its own chain. convert.sh reads ACC from the genesis on disk instead of defaulting to 2, and warns when a conversion takes away a subnet's only aggregator. Adds host-check.sh, the local counterpart to sweep.sh: it reads only 127.0.0.1, so it still answers when the central Prometheus or this host's remote_write is the thing that broke. Adds start-observability.sh to relaunch a host's scrapers.
d441c34 to
bce946e
Compare
…silently The env loader was correct but unreadable: `eval "export $key=\$val"` escapes the value so it expands after eval has parsed, and an `export name=value` assignment suppresses word splitting, so multi-word values did arrive whole. Nobody should have to know that to trust the line, and a reviewer already read it as the interpolated form that would break SERVERS="host-a host-b". Bash indirect expansion plus one quoted export word says the same thing with nothing left to reparse. While in there, the header's promise that comments and malformed lines are skipped was only half true. A `# note` after an unquoted value landed inside the value, so SERVERS picked up a host called '#' for sweep.sh to ssh to; quoted values now delimit themselves, keeping a '#' that belongs to the value (http://x/y#frag). A bare word with no '=' became a variable assigned its own name. A leading-digit name reached `export` and produced its error instead of ours. Indented lines were dropped outright. Names that can't be assigned are now reported on stderr, because a config line that goes unread is how you deploy against the wrong deployment.
#566 added `lean_gossip_*_arrival_delay_seconds` and `lean_gossip_*_arrival_total`, which answer "are votes arriving late, or not arriving at all?" from inside each node. Nothing graphed them, so answering it still meant an out-of-band event-monitor run against a collector clock. Adds a "Gossip Arrival Timing" row to `client-dashboard.json`: a 3x4 grid, one column per message kind (block / attestation / aggregate), one row per view. - delay p99/p50 per node, with a dashed line at one interval (0.8s): above it the typical message misses the interval it was due in - delay distribution heatmap, which exposes the bimodal profile that percentiles average away - arrival position stacked by `position`, the only view that recovers the sign the absolute-value histogram discards (rising `before` is clock skew, rising `after` is propagation or CPU). The aggregate panel omits `before`, unreachable by construction since aggregates anchor to the latest aggregation-interval boundary rather than their own data slot - on-time fraction per node, so one late node separates from a fleet-wide drop All 15 queries were run against the central Prometheus before committing. Also adds a receiver-side timing block to the node-health checklist, whose item 5 covered only the node's own duties, and corrects where dashboards get deployed: the JSONs live in the host dir bind-mounted at `/var/lib/grafana/dashboards`, not in `<GRAFANA_PROV_DIR>/dashboards`, which holds only the provider yaml. A JSON dropped in the provisioning tree is silently ignored, which reads as a working copy that never appears. Recorded as `GRAFANA_DASHBOARDS_DIR`.
Two rows had a band that did not fill the 24-column grid, which renders as dead space next to a panel: Fork-Choice's `Fork-choice block processing time` sat alone at w=12, and Aggregation Coverage's odd seventh panel, `Early aggregation start lead (p99)`, sat alone at w=8. Fork-Choice repacks from a ragged 12 | 12+12 | 8+8+8 into two uniform bands of 3x8, the layout most rows in this dashboard already use. The lone coverage panel widens to the full 24 instead, since seven panels cannot divide evenly into thirds. No query, unit or threshold changed; this is gridPos only. Verified by checking every band in the dashboard sums to 24 columns with no x holes and no vertical gap between bands, so the new Gossip Arrival Timing row is covered by the same check.
The skill only modelled one independent devnet per server, so resetting a chain that spans hosts meant hand-writing the launcher every time: its `start-devnet.sh` assumes one host owns nodes 0..NODES-1 and derives each aggregator's subnet from the node index, which is wrong once a host owns a slice (node 36 aggregating subnet 4). Those throwaway scripts were written twice and lost twice with their sessions. - start-range.sh: launch one host's slice, aggregator->subnet map passed explicitly. Its preflight logs the genesis pubkey size, so a 52-byte (leanSig) vs 32-byte (leanVM-main) image mismatch surfaces before launch rather than as a genesis parse error on every node. - check-range.sh: host-check.sh for a START..END range. - teardown.sh: retire a host's slice in wipe-before-stamp order, archiving genesis to genesis.bak-<ts> so the retired chain's hash-sig keys survive. Also document the variant: ENRs must carry each host's real ip (so make-genesis.sh, which pins 127.0.0.1, is not usable as-is), the 2/3 threshold spans hosts, and the reset workflow pre-flights the image against a staged genesis while the old chain is still running, since a pubkey-size mismatch fails upstream of any checkpoint-sync path. Measured on a 64-node two-host reset: teardown including a 184 GB RocksDB wipe takes 7s per host, so the genesis countdown needs far less padding than the existing note implies.
2 participants
Adds the
multi-server-devnetskill: the operational knowledge for thelong-lived devnets that run as detached docker containers on remote hosts, one
independent single-host devnet per server, federated into one Grafana.
Nothing in it is deployment-specific. Hosts, node counts, subnet counts, central
prometheus/loki urls and Grafana ids all come from a gitignored
scripts/devnet.env(template committed asdevnet.env.example), and everydashboard picks its datasource through a template variable rather than a pinned
uid, so the JSON drops into any Grafana unedited.
Contents
127.0.0.1; three-layer naming (identitynode_N/ container<client>_N/ indexN)make-genesis.sh,merge-keyshards.py(hash-sig-cli has no--start-index),subnet-align-validators.pystart-devnet.sh,cs-restart.sh,agg-restart.sh,convert.sh(per-client CLI shapes for zeam/ream/qlean/grandine/gean/lantern)prometheus-config.sh,promtail-config.sh,start-promtail.sh, the finality Slack alert, and four dashboards (client, finality, logs, resources)sweep.shfor per-devnet head/justified/finalized + client mixoperations.md,clients.md,node-health.md(per-node "is this node working" checklist with a query and a log grep per item)The golden rules in
SKILL.mdeach came out of an outage: always pass acheckpoint-sync URL on restart, wait 60s between stop and start so the node
rejoins the gossip meshes, one stable
GENESIS_TIMEper devnet, and keep theswap + per-container memory guards so one client's leak OOMs its own container
instead of starving the host.
Client dashboard start-time panels
The two follow-up commits rework how the client dashboard reports node starts,
since both panels answered a question nobody asks:
max(), i.e. thenewest process start, so one restarted node made a fleet that had been up
for days look fresh. Now
min(), rendered relative ("6 days ago"), which isthe devnet's uninterrupted age.
changes(lean_node_start_time_seconds[1m]), which drew flat zeros whennothing had restarted and, worse, hid restarts that had happened: the spike
is only as wide as the range window, so Grafana's step steps over it at any
dashboard range past about an hour. Verified on live data over 7 days — 2
non-zero points at
step=60s, zero atstep=300s. Replaced with one tile pernode showing age since start, coloured red < 15m / orange < 1h / yellow < 6h /
dark green beyond, so a restart lights up and decays back to calm.
Both panels note that only ethlambda exports
lean_node_start_time_seconds, so converted canaries never appear in them.Testing
The skill's dashboards are deployed and serving on the central Grafana; the
committed
client-dashboard.jsonis byte-identical to the live copy there.Panel queries were checked against live Prometheus across devnet-eth2 /
devnet-eth35 / devnet-eth4.