diff --git a/CLAUDE.md b/CLAUDE.md index 34ad0704..263090ac 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -52,15 +52,64 @@ crates/ - Communication via `mpsc::unbounded_channel` - Shared storage via `Arc` (clone Store, share backend) -### Tick-Based Validator Duties (4-second slots, 5 intervals per slot) +### Tick-Based Validator Duties (4-second slots, 4 intervals of 1000ms) ``` -Interval 0: Block published (at the slot boundary). The build+publish code path is merged into the previous slot's interval 4 (see below) and aligned to publish here; no attestation acceptance happens at interval 0. -Interval 1: Attestation production (all validators, including proposer) -Interval 2: Aggregation (aggregators create proofs from gossip signatures) -Interval 3: Safe target update (fork choice) -Interval 4: Accept accumulated attestations; build the NEXT slot's block and publish it aligned to that slot's interval 0 (build and publish merged into this tick) +Interval 0: Block published (at the slot boundary). The build+publish code path is merged into the previous slot's interval 3 (see below) and aligned to publish here. Block import merges the block's slot-(T-1) committee bits into the heartbeat vote store; new payloads are promoted to known. +Interval 1: Attestation production (all validators, including proposer). Committee members republish the same signature to the global heartbeat topic. The early-aggregation window check is scheduled. +Interval 2: Safe target update from current-slot heartbeat votes; the aggregation session starts (or already started early) and its aggregates begin publishing. +Interval 3: lagging_head (RLMD window) then fast_head (GHOST-Eph); accept accumulated attestations; build the NEXT slot's block and publish it aligned to that slot's interval 0 (build and publish merged into this tick) ``` +The slot stays 4000ms across the 5→4 interval change: the XMSS epoch is the slot, so +key lifetime is untouched, `GENESIS_TIME` configs stay valid, and slot numbers still +line up with other clients even while the interval grid inside the slot diverges. + +### Two-Tier Fork Choice (heartbeat) + +| value | interval | base | vote window | pool | `min_score` | +|---|---|---|---|---|---| +| `safe_target` | 2 | `latest_justified` | `slot == S` only | heartbeat (gossip) | `ceil(3K'/4)` | +| `lagging_head` | 3 | `latest_justified` | `[S-N, S)` | heartbeat ∪ `known_payloads` | `ceil(2n/3)` | +| `fast_head` | 3 | `lagging_head` | `slot == S-1`, expanding | heartbeat | `0` | + +### Aggregation triggers + +One session per slot, started at interval 2 or earlier once a threshold is met. +Two roles trigger it: + +| role | early threshold | jobs | subnet ordering | +|---|---|---|---| +| committee aggregator | 2/3 of signatures expected from subscribed subnets | up to `MAX_AGGREGATION_JOBS` from the subnet pool | `SlotOrdering::TierOnly` | +| next slot's proposer | `ceil(3K'/4)` heartbeat votes (the safe-target threshold) | exactly 1, over the heartbeat committee votes | `SlotOrdering::CurrentSlotFirst` (fallback only) | + +The proposer's job is built by `heartbeat_fold::heartbeat_aggregation_snapshot`, +which picks the `AttestationData` with the most buffered committee signers and +aggregates **raw signatures only** — it never recurses into an existing type-1, so +the job's `children` list is always empty and its coverage is exactly the buffered +signer set. The one existing-proof check left is dominance: when a *single* type-1 +for that data already covers every buffered signer the fold is skipped +(`lean_heartbeat_fold_skipped_total`), since with proposer aggregation off the +builder keeps one best-coverage proof per data rather than merging them. It falls +back to a single subnet job when nothing is foldable. The result flows through the +ordinary `AggregateProduced` path into `new_payloads`, is promoted at interval 3, +and reaches the builder as one candidate among many; `Tier::Heartbeat` is what +makes it win. + +Heartbeat signatures are the proposer's alone: `heartbeat_aggregation_snapshot` is +the only reader of that buffer. An aggregator that does not propose the next slot +sees the same committee votes only where they duplicate into its subnet pool, and +`SlotOrdering::TierOnly` denies them the recency bucket there, so they are +aggregated only when they win on consensus value (Finalize > Justify > Build). +Recency is worth a queue jump only to the proposer, which is the one node that has +to pack those votes; everyone else already has them raw off the global topic. + +`K` is `HEARTBEAT_COMMITTEE_SIZE` from the genesis config (default 16); `K' = min(K, n)` +and every threshold is denominated in `K'`, never the raw `K`. `N` is +`RLMD_LOOKBACK_LIMIT` (8). Heartbeat votes ride in `body.attestations` — there is no +dedicated `BlockBody` field — and are extracted on import by +`store::extract_heartbeat_votes`, whose `data.slot == block.slot - 1` gate exactly +mirrors the packer's `Tier::Heartbeat`. + ### Attestation Pipeline ``` Gossip → Signature verification → new_payloads (pending) @@ -275,7 +324,7 @@ actual_slot = finalized_slot + 1 + relative_index ### Protocols - **Transport**: QUIC over UDP (TLS 1.3) - **Gossipsub**: Blocks + Attestations (snappy raw compression) - - Topic: `/leanconsensus/{fork_digest}/{block|aggregation|attestation_N}/ssz_snappy` + - Topic: `/leanconsensus/{fork_digest}/{block|aggregation|heartbeat|attestation_N}/ssz_snappy` - `fork_digest` is a 4-byte hex string (no `0x` prefix); currently the dummy `12345678` agreed across clients - Mesh size: 8 (6-12 bounds), heartbeat: 700ms - **Req/Resp**: Status, BlocksByRoot, BlocksByRange (snappy frame compression + varint length) diff --git a/bin/ethlambda/src/main.rs b/bin/ethlambda/src/main.rs index 5b40ad37..2a7661c0 100644 --- a/bin/ethlambda/src/main.rs +++ b/bin/ethlambda/src/main.rs @@ -205,10 +205,22 @@ async fn main() -> eyre::Result<()> { .filter(|url| !url.is_empty()) .collect(); - let store = fetch_initial_state(&clean_checkpoint_urls, &genesis_config, backend.clone()) + let mut store = fetch_initial_state(&clean_checkpoint_urls, &genesis_config, backend.clone()) .await .inspect_err(|err| error!(%err, "Failed to initialize state"))?; + // Adopt the genesis config's HEARTBEAT_COMMITTEE_SIZE on first boot only; a + // persisted value wins thereafter, because committee membership decides which + // bits of an imported block count as heartbeat votes and a restart must not + // change that silently. A mismatch is warned about, not applied. + store + .reconcile_heartbeat_committee_size(genesis_config.heartbeat_committee_size) + .inspect_err(|err| error!(%err, "Failed to persist heartbeat committee size"))?; + info!( + heartbeat_committee_size = store.heartbeat_committee_size(), + "Heartbeat committee configured" + ); + let validator_ids: Vec = validator_keys.keys().copied().collect(); // Shared, runtime-mutable aggregator flag. Seeded from the CLI and @@ -250,6 +262,9 @@ async fn main() -> eyre::Result<()> { proposer_config: ProposerConfig { enable_proposer_aggregation: options.enable_proposer_aggregation, max_attestations_per_block: options.max_attestations_per_block, + // Read once here rather than per build: the persisted value is + // authoritative from first boot and cannot change at runtime. + heartbeat_committee_size: store.heartbeat_committee_size(), }, }; @@ -758,6 +773,7 @@ async fn fetch_initial_state( mod tests { use super::*; use ethlambda_storage::backend::InMemoryBackend; + use ethlambda_types::constants::DEFAULT_HEARTBEAT_COMMITTEE_SIZE; use ethlambda_types::genesis::GenesisValidatorEntry; /// Validator-config snippet matching `lean-quickstart`'s ansible-devnet @@ -883,6 +899,7 @@ validators: fn test_genesis(genesis_time: u64) -> GenesisConfig { GenesisConfig { genesis_time, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, genesis_validators: vec![GenesisValidatorEntry { attestation_pubkey: [1u8; 52], proposal_pubkey: [2u8; 52], diff --git a/crates/blockchain/src/aggregation.rs b/crates/blockchain/src/aggregation.rs index d8249c76..474d6ef3 100644 --- a/crates/blockchain/src/aggregation.rs +++ b/crates/blockchain/src/aggregation.rs @@ -1,21 +1,32 @@ //! Committee-signature aggregation: off-thread worker orchestration and the //! pure functions it runs. //! -//! The blockchain actor fires one aggregation session per slot — at interval 2, -//! or up to [`EARLY_AGGREGATION_WINDOW`] early when the 2/3 signature -//! threshold is met — via +//! The blockchain actor fires one aggregation session per slot via //! [`run_aggregation_worker`]. The actor stays on its message loop; the worker //! runs the expensive XMSS proofs on a `spawn_blocking` thread and streams //! results back as [`AggregateProduced`] / [`AggregationDone`] messages. //! +//! Two roles trigger a session, and either can start it at interval 2 or up to +//! [`EARLY_AGGREGATION_WINDOW`] early once its own threshold is met: +//! +//! - a **committee aggregator**, whose threshold is 2/3 of the signatures expected +//! from its subscribed subnets, and which selects up to [`MAX_AGGREGATION_JOBS`] +//! jobs from the subnet pool; +//! - the **next slot's proposer**, whose threshold is the safe target's +//! `ceil(3K'/4)` of the heartbeat committee, and which runs exactly one job over +//! those heartbeat votes (see [`crate::heartbeat_fold`]). +//! +//! Both job kinds share this worker, deadline, and result path, so a folded +//! heartbeat aggregate reaches the block builder through the ordinary payload pool +//! rather than a private channel. +//! //! [`snapshot_aggregation_inputs`] builds the session's job list with a tiered //! greedy selector modeled on `block_builder::select_attestations`: an //! up-front store pass resolves every candidate `AttestationData`'s //! aggregation material once (raw-first + trim, see [`resolve_job`]), then a //! pure in-memory loop scores and orders candidates by consensus value -//! (current-slot before stale, then Finalize > Justify > Build), emitting at -//! most `max_jobs` jobs — [`MAX_AGGREGATION_JOBS`] normally, dropping to a -//! single job in the slot before one of our validators proposes. +//! (Finalize > Justify > Build, with this slot's groups jumping the queue only +//! for the proposer — see [`SlotOrdering`]), emitting at most `max_jobs` jobs. use std::collections::{HashMap, HashSet}; use std::time::{Duration, Instant, SystemTime}; @@ -45,7 +56,10 @@ use crate::{MILLISECONDS_PER_INTERVAL, metrics}; /// started early (see `maybe_start_early_aggregation`) ends correspondingly /// earlier. The deadline only stops new jobs from starting — a job mid-proof /// finishes and publishes right after. -pub(crate) const AGGREGATION_DEADLINE: Duration = Duration::from_millis(800); +/// +/// Derived from the interval width rather than written as a literal so it keeps +/// meaning "one interval" across changes to the interval grid. +pub(crate) const AGGREGATION_DEADLINE: Duration = Duration::from_millis(MILLISECONDS_PER_INTERVAL); /// Upper bound we wait for a prior worker to exit if it is still running when /// the next session is about to start. Reached only in pathological cases /// (mismatched timers, stuck proofs); we warn before blocking. @@ -66,6 +80,46 @@ const _: () = assert!( "EARLY_AGGREGATION_WINDOW must not exceed one interval" ); +/// Where a job's raw signatures came from. +/// +/// Only affects which histogram the prover time lands in: the heartbeat fold has +/// its own deadline pressure (it must finish before the proposer's interval-3 +/// build), so merging its timing into the committee session's would hide exactly +/// the signal that tells you whether a chosen `K` still fits. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum JobSource { + /// Gossip signatures and payload-only merges from the attestation subnets. + Subnet, + /// Raw committee signatures from the global heartbeat topic. + Heartbeat, +} + +/// How a session ranks this slot's candidate groups against stale ones. +/// +/// This slot's groups are the committee's view-merge payload: the next slot's +/// proposer needs them, and every other node already receives them raw on the +/// global heartbeat topic. So recency is worth a queue jump only to the proposer, +/// which is why the two roles order candidates differently. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SlotOrdering { + /// Current-slot groups precede stale ones; tier decides within a bucket. + /// + /// The next slot's proposer's ordering. It reaches the subnet pool only as a + /// fallback (nothing foldable from the heartbeat topic), and the one job it + /// spends there is still meant to cover this slot's committee, since that is + /// what `Tier::Heartbeat` packs into the block it is about to build. + CurrentSlotFirst, + /// Tier alone decides: Finalize > Justify > Build, no recency bucket. + /// + /// A committee aggregator that does not propose the next slot. Its scarce + /// resource is leanVM time, and its contribution to the network is pushing + /// targets over 2/3 — not re-publishing votes the heartbeat topic already + /// delivered. Under this ordering a current-slot group is aggregated only when + /// it wins on consensus value: it finalizes, it justifies, or it adds coverage + /// no other candidate does. + TierOnly, +} + /// A single pre-prepared aggregation group. /// /// Built on the actor thread from a store snapshot; consumed by an off-thread @@ -83,6 +137,7 @@ pub struct AggregationJob { pub(crate) raw_ids: Vec, /// Gossip-signature keys to delete on successful aggregation. pub(crate) keys_to_delete: Vec<(u64, H256)>, + pub(crate) source: JobSource, } impl AggregationJob { @@ -177,6 +232,10 @@ impl Message for EarlyAggregationCheck { /// leanVM prover work against [`AGGREGATION_DEADLINE`]: the greedy loop in /// [`snapshot_aggregation_inputs`] stops after this many rounds even if /// scoring candidates remain. +/// +/// A session run because we propose the next slot uses one job instead, spent on +/// the heartbeat committee's votes; see +/// `BlockChainServer::start_aggregation_session`. pub(crate) const MAX_AGGREGATION_JOBS: usize = 2; /// Build a snapshot of everything needed to aggregate. Runs on the actor @@ -193,20 +252,24 @@ pub(crate) const MAX_AGGREGATION_JOBS: usize = 2; /// at least two existing proofs to merge). /// 2. **Greedy loop**, at most `max_jobs` rounds: each round /// scores every unselected candidate against the projected state and -/// keeps the lowest ordering key (current-slot before stale, then -/// Finalize > Justify > Build, mirroring the block builder). The winning -/// [`AggregationJob`] is emitted as-is; the projection is updated with its -/// realized coverage. +/// keeps the lowest ordering key (Finalize > Justify > Build, mirroring the +/// block builder, behind whatever recency bucket `slot_ordering` asks for). +/// The winning [`AggregationJob`] is emitted as-is; the projection is updated +/// with its realized coverage. /// /// Stops early when no remaining candidate scores (converged). /// -/// `max_jobs` is [`MAX_AGGREGATION_JOBS`] for an ordinary session and `1` when -/// the caller is about to build a block at interval 4 (see -/// `BlockChainServer::start_aggregation_session`). +/// `max_jobs` is [`MAX_AGGREGATION_JOBS`] for an ordinary session, run with +/// [`SlotOrdering::TierOnly`]. A session run because we propose the next slot +/// prefers a single heartbeat job instead +/// ([`crate::heartbeat_fold::heartbeat_aggregation_snapshot`]) and only falls back +/// here, with `max_jobs = 1` and [`SlotOrdering::CurrentSlotFirst`], when nothing +/// is foldable. pub fn snapshot_aggregation_inputs( store: &Store, current_slot: u64, max_jobs: usize, + slot_ordering: SlotOrdering, ) -> Option { let gossip_groups = store.iter_gossip_signatures(); let new_payload_keys = store.new_payload_keys(); @@ -284,6 +347,7 @@ pub fn snapshot_aggregation_inputs( &extended_historical_block_hashes, current_slot, validator_count, + slot_ordering, ) else { trace!( jobs_selected = jobs.len(), @@ -312,7 +376,7 @@ pub fn snapshot_aggregation_inputs( // Fold the job's realized coverage into the shared projection so // same-target candidates re-tier across rounds exactly as the block // builder's post-state would. - projected.advance(score.tier, att_data, coverage.iter().copied()); + projected.advance(score.effect, att_data, coverage.iter().copied()); jobs.push(job); } @@ -334,8 +398,8 @@ pub fn snapshot_aggregation_inputs( /// voters (relative to the candidate's realized [`AggregationJob::coverage`], /// not the full proof union — see [`resolve_job`]). Among the rest, returns /// `(data_root, score)` for the entry with the lowest composite key: -/// current-slot groups precede stale ones, then `EntryScore::ordering_key` -/// (tier, then tier-dependent dims, then `data_root`) decides. +/// `slot_ordering`'s recency bucket, then `EntryScore::ordering_key` (tier, then +/// tier-dependent dims, then `data_root`). fn pick_best_candidate( candidates: &HashMap, projected: &block_builder::ProjectedState, @@ -343,6 +407,7 @@ fn pick_best_candidate( extended_historical_block_hashes: &[H256], current_slot: u64, validator_count: usize, + slot_ordering: SlotOrdering, ) -> Option<(H256, EntryScore)> { let mut best: Option<(H256, EntryScore)> = None; let mut best_key: Option<(u8, block_builder::OrderingKey)> = None; @@ -365,11 +430,11 @@ fn pick_best_candidate( continue; }; - // Current-slot groups always precede stale ones (goal: consider - // current-slot signatures first); within a bucket, `EntryScore` - // decides. - let slot_bucket: u8 = if att_data.slot == current_slot { 0 } else { 1 }; - let candidate_key = candidate_ordering_key(slot_bucket, &score, *data_root); + let candidate_key = candidate_ordering_key( + slot_bucket(slot_ordering, att_data.slot, current_slot), + &score, + *data_root, + ); if best_key.as_ref().is_none_or(|k| candidate_key < *k) { best = Some((*data_root, score)); best_key = Some(candidate_key); @@ -379,9 +444,21 @@ fn pick_best_candidate( best } -/// Composite ordering key (lower is better): current-slot groups (`0`) -/// precede stale ones (`1`); within a bucket, `EntryScore::ordering_key` -/// (tier, then tier-dependent dims, then `data_root`) decides. +/// Recency bucket for a candidate (lower is better), per [`SlotOrdering`]. +/// +/// [`SlotOrdering::TierOnly`] collapses every candidate into bucket `0`, so the +/// composite key degenerates to `EntryScore::ordering_key` alone and a current-slot +/// group has to out-tier a stale one to be picked. +fn slot_bucket(slot_ordering: SlotOrdering, att_slot: u64, current_slot: u64) -> u8 { + match slot_ordering { + SlotOrdering::CurrentSlotFirst => u8::from(att_slot != current_slot), + SlotOrdering::TierOnly => 0, + } +} + +/// Composite ordering key (lower is better): the [`slot_bucket`] leads; within a +/// bucket, `EntryScore::ordering_key` (tier, then tier-dependent dims, then +/// `data_root`) decides. fn candidate_ordering_key( slot_bucket: u8, score: &EntryScore, @@ -480,6 +557,7 @@ fn resolve_job( raw_sigs, raw_ids, keys_to_delete, + source: JobSource::Subnet, }) } @@ -530,7 +608,10 @@ pub fn aggregate_job(job: AggregationJob) -> Option { let data_root = job.hashed.root(); let proof_data = { - let _timing = metrics::time_pq_sig_aggregated_signatures_building(); + let _timing = match job.source { + JobSource::Subnet => metrics::time_pq_sig_aggregated_signatures_building(), + JobSource::Heartbeat => metrics::time_heartbeat_fold(), + }; aggregate_mixed( job.children, job.raw_pubkeys, @@ -973,16 +1054,27 @@ mod tests { // ---- ordering ---- - /// The slot bucket dominates the within-bucket score: a current-slot - /// candidate is picked ahead of a stale candidate that has *more* new - /// voters (which, absent the bucket, would win the Build-tier - /// `new_voters` dimension). Exercises `candidate_ordering_key` through the - /// real `pick_best_candidate` path rather than constructing an - /// `EntryScore` directly. - #[test] - fn pick_best_candidate_prefers_current_slot_over_higher_stale_score() { - const NUM_VALIDATORS: usize = 100; - const CURRENT_SLOT: u64 = 3; + const ORDERING_NUM_VALIDATORS: usize = 100; + const ORDERING_CURRENT_SLOT: u64 = 3; + + /// Everything `pick_best_candidate` needs for the two ordering tests. + struct OrderingFixture { + candidates: HashMap, + projected: block_builder::ProjectedState, + known_block_roots: HashSet, + historical_block_hashes: Vec, + root_current: H256, + root_stale: H256, + } + + /// One current-slot candidate with a single new voter against one stale + /// candidate with five, on independent target roots so their voter buckets + /// never interact. Both score `Tier::Build`, so the stale one wins the + /// within-tier `new_voters` dimension and only the recency bucket can flip + /// the outcome — which is exactly what the two [`SlotOrdering`] variants + /// disagree about. + fn ordering_fixture() -> OrderingFixture { + const CURRENT_SLOT: u64 = ORDERING_CURRENT_SLOT; let genesis_root = H256([1u8; 32]); let target_root = H256([7u8; 32]); @@ -1035,6 +1127,7 @@ mod tests { raw_sigs: Vec::new(), raw_ids: coverage.into_iter().collect(), keys_to_delete: Vec::new(), + source: JobSource::Subnet, } }; @@ -1056,23 +1149,72 @@ mod tests { current_votes: HashMap::new(), }; + OrderingFixture { + candidates, + projected, + known_block_roots, + historical_block_hashes, + root_current, + root_stale, + } + } + + /// Under [`SlotOrdering::CurrentSlotFirst`] the recency bucket dominates the + /// within-bucket score: the current-slot candidate is picked ahead of a stale + /// candidate that has *more* new voters. Exercises `candidate_ordering_key` + /// through the real `pick_best_candidate` path rather than constructing an + /// `EntryScore` directly. + #[test] + fn pick_best_candidate_prefers_current_slot_over_higher_stale_score() { + let fixture = ordering_fixture(); + let (picked_root, score) = pick_best_candidate( - &candidates, - &projected, - &known_block_roots, - &historical_block_hashes, - CURRENT_SLOT, - NUM_VALIDATORS, + &fixture.candidates, + &fixture.projected, + &fixture.known_block_roots, + &fixture.historical_block_hashes, + ORDERING_CURRENT_SLOT, + ORDERING_NUM_VALIDATORS, + SlotOrdering::CurrentSlotFirst, ) .expect("both candidates are viable Build-tier entries"); assert_eq!(score.tier, block_builder::Tier::Build); assert_eq!( - picked_root, root_current, + picked_root, fixture.root_current, "the current-slot group must be picked ahead of a stale group with more new voters" ); } + /// Under [`SlotOrdering::TierOnly`] recency buys nothing: the same pool picks + /// the stale candidate, because it wins Build tier's `new_voters` dimension. + /// + /// This is what keeps a non-proposing aggregator's jobs on consensus value + /// rather than on the committee's view-merge payload — those votes reach every + /// peer raw on the global heartbeat topic, so re-proving them buys the network + /// nothing an aggregator alone could provide. + #[test] + fn pick_best_candidate_tier_only_ignores_slot_recency() { + let fixture = ordering_fixture(); + + let (picked_root, score) = pick_best_candidate( + &fixture.candidates, + &fixture.projected, + &fixture.known_block_roots, + &fixture.historical_block_hashes, + ORDERING_CURRENT_SLOT, + ORDERING_NUM_VALIDATORS, + SlotOrdering::TierOnly, + ) + .expect("both candidates are viable Build-tier entries"); + + assert_eq!(score.tier, block_builder::Tier::Build); + assert_eq!( + picked_root, fixture.root_stale, + "with no recency bucket the higher-coverage group wins, current-slot or not" + ); + } + // ---- projection ---- /// Two candidates targeting the same root accumulate coverage: the @@ -1128,6 +1270,7 @@ mod tests { raw_sigs: Vec::new(), raw_ids: coverage.into_iter().collect(), keys_to_delete: Vec::new(), + source: JobSource::Subnet, } }; @@ -1156,6 +1299,7 @@ mod tests { &historical_block_hashes, 999, NUM_VALIDATORS, + SlotOrdering::TierOnly, ) .expect("round 1 should find a candidate"); assert_eq!(picked_root, root_a); @@ -1179,6 +1323,7 @@ mod tests { &historical_block_hashes, 999, NUM_VALIDATORS, + SlotOrdering::TierOnly, ) .expect("round 2 should find B"); assert_eq!(picked_root, root_b); @@ -1197,7 +1342,9 @@ mod tests { fn snapshot_returns_none_for_empty_store() { let hashes = vec![H256([1u8; 32])]; let store = new_test_store(make_head_state(0, 4, &hashes)); - assert!(snapshot_aggregation_inputs(&store, 0, MAX_AGGREGATION_JOBS).is_none()); + let snapshot = + snapshot_aggregation_inputs(&store, 0, MAX_AGGREGATION_JOBS, SlotOrdering::TierOnly); + assert!(snapshot.is_none()); } /// A single gossip signature with no other material to merge is dropped @@ -1226,7 +1373,9 @@ mod tests { let hashed = HashedAttestationData::new(att_data); store.insert_gossip_signature(hashed, 0, dummy_sig()); - assert!(snapshot_aggregation_inputs(&store, 0, MAX_AGGREGATION_JOBS).is_none()); + let snapshot = + snapshot_aggregation_inputs(&store, 0, MAX_AGGREGATION_JOBS, SlotOrdering::TierOnly); + assert!(snapshot.is_none()); } /// A group whose target is already justified (here: at or behind the @@ -1269,7 +1418,8 @@ mod tests { store.insert_gossip_signature(hashed, 1, dummy_sig()); assert!( - snapshot_aggregation_inputs(&store, 999, MAX_AGGREGATION_JOBS).is_none(), + snapshot_aggregation_inputs(&store, 999, MAX_AGGREGATION_JOBS, SlotOrdering::TierOnly) + .is_none(), "a group targeting an already-justified slot must never become a job" ); } @@ -1325,8 +1475,13 @@ mod tests { store.insert_gossip_signature(hashed.clone(), 0, dummy_sig()); store.insert_gossip_signature(hashed, 1, dummy_sig()); - let snapshot = snapshot_aggregation_inputs(&store, HEAD_SLOT, MAX_AGGREGATION_JOBS) - .expect("a vote for the current head must produce a job (chain view covers the tip)"); + let snapshot = snapshot_aggregation_inputs( + &store, + HEAD_SLOT, + MAX_AGGREGATION_JOBS, + SlotOrdering::TierOnly, + ) + .expect("a vote for the current head must produce a job (chain view covers the tip)"); assert_eq!(snapshot.jobs.len(), 1); assert_eq!( snapshot.jobs[0].hashed.data().target.slot, @@ -1386,8 +1541,9 @@ mod tests { fn snapshot_caps_jobs_at_max_aggregation_jobs() { let store = store_with_competing_build_tier_groups(); - let snapshot = snapshot_aggregation_inputs(&store, 999, MAX_AGGREGATION_JOBS) - .expect("should produce jobs"); + let snapshot = + snapshot_aggregation_inputs(&store, 999, MAX_AGGREGATION_JOBS, SlotOrdering::TierOnly) + .expect("should produce jobs"); assert_eq!(snapshot.groups_considered, NUM_GROUPS); assert_eq!(snapshot.jobs.len(), MAX_AGGREGATION_JOBS); @@ -1412,7 +1568,8 @@ mod tests { fn snapshot_caps_jobs_at_one_for_proposer() { let store = store_with_competing_build_tier_groups(); - let snapshot = snapshot_aggregation_inputs(&store, 999, 1).expect("should produce a job"); + let snapshot = snapshot_aggregation_inputs(&store, 999, 1, SlotOrdering::CurrentSlotFirst) + .expect("should produce a job"); assert_eq!(snapshot.groups_considered, NUM_GROUPS); assert_eq!(snapshot.jobs.len(), 1); assert_eq!( diff --git a/crates/blockchain/src/block_builder.rs b/crates/blockchain/src/block_builder.rs index f064b3d7..bd84aaf4 100644 --- a/crates/blockchain/src/block_builder.rs +++ b/crates/blockchain/src/block_builder.rs @@ -17,8 +17,8 @@ use std::{ use ethlambda_crypto::{aggregate_proofs, signature::ValidatorPublicKey}; use ethlambda_state_transition::{ - attestation_data_matches_chain, justified_slots_ops, process_block, process_slots, - slot_is_justifiable_after, + attestation_data_matches_chain, is_heartbeat_committee_member, justified_slots_ops, + process_block, process_slots, slot_is_justifiable_after, }; use ethlambda_types::{ ShortRoot, @@ -56,6 +56,10 @@ pub struct ProposerConfig { /// Proposer-side self-limit only; clamped to `MAX_ATTESTATIONS_DATA` during /// selection so the block never exceeds the cap `on_block` enforces. pub max_attestations_per_block: usize, + /// The network's `K`. Used only to classify entries into + /// [`Tier::Heartbeat`], never to validate anything, so a stale value builds a + /// worse block rather than an invalid one. + pub heartbeat_committee_size: u64, } /// Build a valid block on top of this state. @@ -91,6 +95,7 @@ pub(crate) fn build_block( aggregated_payloads: &HashMap)>, config: ProposerConfig, ) -> Result<(Block, Vec, PostBlockCheckpoints), StoreError> { + let heartbeat_committee_size = config.heartbeat_committee_size; info!(slot, proposer_index, "Building block"); let select_start = Instant::now(); @@ -100,6 +105,7 @@ pub(crate) fn build_block( parent_root, known_block_roots, aggregated_payloads, + heartbeat_committee_size, config.max_attestations_per_block, ); metrics::observe_block_proposal_phase("select_payloads", select_start.elapsed()); @@ -174,6 +180,7 @@ fn select_attestations( parent_root: H256, known_block_roots: &HashSet, aggregated_payloads: &HashMap)>, + heartbeat_committee_size: u64, max_attestations_per_block: usize, ) -> Vec<(AggregatedAttestation, SingleMessageAggregate)> { let mut selected: Vec<(AggregatedAttestation, SingleMessageAggregate)> = Vec::new(); @@ -192,11 +199,13 @@ fn select_attestations( extended_historical_block_hashes.push(parent_root); extended_historical_block_hashes.extend(std::iter::repeat_n(H256::ZERO, num_empty_slots)); + let validator_count = head_state.validators.len(); let chain = ChainContext { aggregated_payloads, known_block_roots, extended_historical_block_hashes: &extended_historical_block_hashes, - validator_count: head_state.validators.len(), + validator_count, + heartbeat: HeartbeatTier::for_block(slot, validator_count as u64, heartbeat_committee_size), }; // Running per-target-root voter set, seeded from state and updated @@ -204,6 +213,7 @@ fn select_attestations( // participation flags in Prysm/Lighthouse-style packing. let mut projected = ProjectedState::from_head_state(head_state); let mut processed_data_roots: HashSet = HashSet::new(); + let mut heartbeat_entries = 0usize; // A block may carry at most `MAX_ATTESTATIONS_DATA` distinct entries // (`on_block` rejects more), so the proposer-side limit never exceeds it. @@ -227,8 +237,14 @@ fn select_attestations( extend_proofs_greedily(proofs, &mut selected, att_data); let target_root = att_data.target.root; + if score.tier == Tier::Heartbeat { + heartbeat_entries += 1; + } + trace!( tier = ?score.tier, + effect = ?score.effect, + committee_voters = score.committee_voters, new_voters = score.new_voters, target_slot = score.target_slot, target_root = %ShortRoot(&target_root.0), @@ -237,9 +253,13 @@ fn select_attestations( "selected" ); - projected.advance(score.tier, att_data, new_voters); + projected.advance(score.effect, att_data, new_voters); } + // Cap-pressure signal: at MAX_ATTESTATIONS_DATA the committee votes are being + // truncated and `Tier::Finalize` is being starved of entry slots. + metrics::observe_heartbeat_entries_in_block(heartbeat_entries); + selected } @@ -275,9 +295,12 @@ fn pick_best_candidate( .iter() .flat_map(|proof| proof.participant_indices()) .collect(); - let Some((score, new_voters)) = - projected.score_entry(att_data, &coverage, chain.validator_count) - else { + let Some((score, new_voters)) = projected.score_entry_with_heartbeat( + att_data, + &coverage, + chain.validator_count, + Some(&chain.heartbeat), + ) else { trace_skipped_attestation("zero_new_voters", att_data, data_root); continue; }; @@ -300,6 +323,9 @@ struct ChainContext<'a> { known_block_roots: &'a HashSet, extended_historical_block_hashes: &'a [H256], validator_count: usize, + /// Committee context for [`Tier::Heartbeat`], derived from the block being + /// built rather than from the store clock. + heartbeat: HeartbeatTier, } /// Mutable projection of the post-state that a tiered greedy selector @@ -337,7 +363,7 @@ impl ProjectedState { /// resulting per-target voter set is the same union either way. pub(crate) fn advance( &mut self, - tier: Tier, + effect: EntryEffect, att_data: &AttestationData, new_voters: impl IntoIterator, ) { @@ -349,7 +375,7 @@ impl ProjectedState { // Finalize implies Justify (target is justified, AND source is // finalized). - if tier <= Tier::Justify { + if effect <= EntryEffect::Justifies { justified_slots_ops::extend_to_slot( &mut self.justified_slots, self.finalized_slot, @@ -364,7 +390,7 @@ impl ProjectedState { // scoring (no further entry can target it: filter rejects). self.current_votes.remove(&target_root); } - if tier == Tier::Finalize { + if effect == EntryEffect::Finalizes { let new_finalized = att_data.source.slot; let delta = new_finalized.saturating_sub(self.finalized_slot) as usize; justified_slots_ops::shift_window(&mut self.justified_slots, delta); @@ -392,6 +418,28 @@ impl ProjectedState { att_data: &AttestationData, coverage: &HashSet, validator_count: usize, + ) -> Option<(EntryScore, HashSet)> { + self.score_entry_with_heartbeat(att_data, coverage, validator_count, None) + } + + /// [`Self::score_entry`], plus the proposer-side heartbeat tier. + /// + /// When `heartbeat` is `Some` and the entry qualifies (its data slot is + /// exactly the committee slot and it covers at least one committee member), + /// the entry sorts into [`Tier::Heartbeat`] and the zero-new-voters drop is + /// bypassed. The drop is about *justification*, and a committee vote whose + /// signers are already justification-covered still has to reach other nodes' + /// heartbeat vote store via the body, so a positive `committee_voters` is + /// itself the positive score. + /// + /// The entry's consensus [`EntryEffect`] is computed identically either way: + /// the tier reorders packing, it never changes what applying the entry does. + pub(crate) fn score_entry_with_heartbeat( + &self, + att_data: &AttestationData, + coverage: &HashSet, + validator_count: usize, + heartbeat: Option<&HeartbeatTier>, ) -> Option<(EntryScore, HashSet)> { let prior_voters = self.current_votes.get(&att_data.target.root); let prior_count = prior_voters.map_or(0, HashSet::len); @@ -401,7 +449,12 @@ impl ProjectedState { .copied() .filter(|vid| prior_voters.is_none_or(|prior| !prior.contains(vid))) .collect(); - if new_voters.is_empty() { + + let committee_voters = heartbeat + .filter(|hb| hb.covers(att_data)) + .map_or(0, |hb| hb.count_members(coverage)); + + if new_voters.is_empty() && committee_voters == 0 { return None; } @@ -419,16 +472,28 @@ impl ProjectedState { && (att_data.source.slot + 1..att_data.target.slot) .all(|s| !slot_is_justifiable_after(s, self.finalized_slot)); - let tier = if is_genesis_self_vote(att_data) || !crosses_2_3 { - Tier::Build + let effect = if is_genesis_self_vote(att_data) || !crosses_2_3 { + EntryEffect::Builds } else if finalizes { - Tier::Finalize + EntryEffect::Finalizes + } else { + EntryEffect::Justifies + }; + + let tier = if committee_voters > 0 { + Tier::Heartbeat } else { - Tier::Justify + match effect { + EntryEffect::Finalizes => Tier::Finalize, + EntryEffect::Justifies => Tier::Justify, + EntryEffect::Builds => Tier::Build, + } }; let score = EntryScore { tier, + effect, + committee_voters, new_voters: new_voters.len(), target_slot: att_data.target.slot, att_slot: att_data.slot, @@ -508,9 +573,22 @@ impl ProjectedState { #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] #[repr(u8)] pub(crate) enum Tier { - /// Applying the entry crosses 2/3 on target AND finalizes the source - /// (no slot strictly between source.slot and target.slot is still - /// justifiable given projected finalized_slot). + /// Carries committee bits for exactly `block.slot - 1`: the view-merge + /// payload the next slot's fast head is computed from. + /// + /// Outranks everything, so arbitrary eviction of the view-merge payload in + /// favour of older aggregates — the fast head silently losing its evidence + /// for reasons that look like a network problem — cannot happen. The cost is + /// the mirror-image risk: a wide committee split can starve + /// [`Self::Finalize`] of entry slots and delay finality by a slot. That is a + /// bounded, self-clearing cost (the split resolves, the tier empties) against + /// an unbounded one, which is why the ordering is this way round. + /// + /// Classification is proposer-side only. No validity rule reads it, so a + /// proposer with a stale committee size builds a worse block, not an invalid + /// one — the one place where committee-membership disagreement is harmless. + Heartbeat = 0, + /// Applying the entry crosses 2/3 on target AND finalizes the source. Finalize = 1, /// Applying the entry crosses 2/3 on target but does not finalize. Justify = 2, @@ -518,13 +596,92 @@ pub(crate) enum Tier { Build = 3, } +/// What applying an entry does to the projected post-state. +/// +/// Split from [`Tier`] because [`Tier::Heartbeat`] reorders packing without +/// having any consensus effect of its own: a heartbeat entry may well be a plain +/// [`Self::Builds`] entry. Keying [`ProjectedState::advance`] off the sort tier +/// would let a heartbeat entry that never crosses 2/3 mark its target justified +/// in the projection. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +#[repr(u8)] +pub(crate) enum EntryEffect { + /// Crosses 2/3 on target and finalizes the source. + Finalizes = 0, + /// Crosses 2/3 on target without finalizing. + Justifies = 1, + /// Adds marginal voters only. + Builds = 2, +} + +/// Committee context for the proposer's heartbeat tier. +/// +/// Built from the block *being built*, never from the clock: `propose_block` +/// advances the store to the next slot's interval 0 before building, so a builder +/// reading "current slot" from `store.time()` mid-build sees `S+1` and would gate +/// on an empty set — a silently empty payload with no error anywhere. +pub(crate) struct HeartbeatTier { + /// Exactly `block.slot - 1`. `None` for a block at slot 0. + cttee_slot: Option, + num_validators: u64, + committee_size: u64, +} + +impl HeartbeatTier { + pub(crate) fn for_block(block_slot: u64, num_validators: u64, committee_size: u64) -> Self { + Self { + cttee_slot: block_slot.checked_sub(1), + num_validators, + committee_size, + } + } + + /// Whether this entry is in the tier's slot: `data.slot == block.slot - 1`, + /// strictly. No window and no fallback on a skipped slot, so this is an exact + /// mirror of `store::extract_heartbeat_votes`'s gate — whatever the builder + /// puts in the tier is exactly what import reads back out. + fn covers(&self, att_data: &AttestationData) -> bool { + self.cttee_slot == Some(att_data.slot) + } + + /// Committee members within `coverage`. + /// + /// Committee members only: a non-committee signer contributes nothing to the + /// tier ordering, because non-committee bits are invisible to the fast head. + /// Nothing is lost by ignoring them here — they still count via `new_voters` + /// lower in the same ordering key, and via the other tiers for every other + /// purpose. + fn count_members(&self, coverage: &HashSet) -> usize { + let Some(cttee_slot) = self.cttee_slot else { + return 0; + }; + coverage + .iter() + .filter(|vid| { + is_heartbeat_committee_member( + **vid, + cttee_slot, + self.num_validators, + self.committee_size, + ) + }) + .count() + } +} + /// Tiered score for a candidate `AttestationData` entry during block building. /// -/// Lower `tier` wins. Entries with zero new voters relative to the running -/// per-target-root voter set are dropped (returned as `None`). +/// Lower `tier` wins. Entries that add no new voters relative to the running +/// per-target-root voter set are dropped (returned as `None`) — except in +/// [`Tier::Heartbeat`], where positive committee coverage is itself the score. /// /// The within-tier ordering is tier-dependent (leanSpec PR #1149): /// +/// - **Heartbeat**: committee coverage leads, then newer target, then +/// `new_voters` (which is where non-committee bits get their say). Under cap +/// pressure the rounds that survive are the highest-coverage ones, so the fast +/// head keeps the votes carrying the most tie-breaking weight and loses only +/// the thinnest fragments. /// - **Finalize / Justify**: the entry already crosses 2/3 on its target, so /// newer chain progress leads: larger `target_slot`, then larger `att_slot`, /// then more `new_voters`. Pushing the justified slot as far forward as @@ -533,10 +690,15 @@ pub(crate) enum Tier { /// coverage leads: more `new_voters`, then larger `target_slot`, then larger /// `att_slot`. /// -/// In both tiers `data_root` (ascending) is the final deterministic tiebreak. +/// In every tier `data_root` (ascending) is the final deterministic tiebreak. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) struct EntryScore { pub(crate) tier: Tier, + /// What applying the entry does to the projection, independent of `tier`. + pub(crate) effect: EntryEffect, + /// Committee members covered, counted only when the entry is in the + /// heartbeat tier's slot. Zero for every non-heartbeat entry. + pub(crate) committee_voters: usize, pub(crate) new_voters: usize, /// Read only inside [`EntryScore::ordering_key`]; kept private. target_slot: u64, @@ -555,9 +717,17 @@ impl EntryScore { /// the type-level docs), all encoded as `Reverse` so "larger is better". pub(crate) fn ordering_key(&self, data_root: H256) -> OrderingKey { let more_new_voters = Reverse(self.new_voters as u64); + let more_committee_voters = Reverse(self.committee_voters as u64); let newer_target = Reverse(self.target_slot); let newer_att = Reverse(self.att_slot); match self.tier { + Tier::Heartbeat => ( + self.tier, + more_committee_voters, + newer_target, + more_new_voters, + data_root, + ), Tier::Build => ( self.tier, more_new_voters, @@ -890,6 +1060,7 @@ fn trace_skipped_attestation(reason: &'static str, att: &AttestationData, data_r #[cfg(test)] mod tests { use super::*; + use ethlambda_types::constants::DEFAULT_HEARTBEAT_COMMITTEE_SIZE; use ethlambda_types::{ attestation::{AggregatedAttestation, AggregationBits, AttestationData}, block::{ByteList512KiB, MultiMessageAggregate, SignedBlock, SingleMessageAggregate}, @@ -915,6 +1086,386 @@ mod tests { bits } + // ============ Tier::Heartbeat Tests ============ + + /// `n = 8, K = 4`, so the committee for slot `s` is `{s%8, .., s%8+3}`. + /// At the committee slot 7 used below that is `{7, 0, 1, 2}`. + const HB_VALIDATORS: u64 = 8; + const HB_COMMITTEE: u64 = 4; + /// Block slot 8, so the committee slot is 7. + const HB_BLOCK_SLOT: u64 = 8; + + fn hb_tier() -> HeartbeatTier { + HeartbeatTier::for_block(HB_BLOCK_SLOT, HB_VALIDATORS, HB_COMMITTEE) + } + + fn empty_projection() -> ProjectedState { + ProjectedState { + justified_slots: JustifiedSlots::new(), + finalized_slot: 0, + current_votes: HashMap::new(), + } + } + + fn score( + tier: Tier, + committee_voters: usize, + new_voters: usize, + target_slot: u64, + ) -> EntryScore { + EntryScore { + tier, + effect: EntryEffect::Builds, + committee_voters, + new_voters, + target_slot, + att_slot: 0, + } + } + + #[test] + fn heartbeat_tier_outranks_every_other_tier() { + let root = H256([0u8; 32]); + let heartbeat = score(Tier::Heartbeat, 1, 1, 0).ordering_key(root); + for other in [Tier::Finalize, Tier::Justify, Tier::Build] { + // A maximally attractive entry in another tier still loses: nothing + // outranks the view-merge payload. + let rival = score(other, 0, usize::MAX, u64::MAX).ordering_key(root); + assert!( + heartbeat < rival, + "Tier::Heartbeat must outrank {other:?}, so the payload is never evicted" + ); + } + } + + #[test] + fn heartbeat_tier_orders_by_committee_coverage_first() { + let root = H256([0u8; 32]); + // Thicker committee coverage wins even against a much newer target. + let thick = score(Tier::Heartbeat, 4, 0, 1).ordering_key(root); + let thin = score(Tier::Heartbeat, 3, 0, 999).ordering_key(root); + assert!(thick < thin); + } + + #[test] + fn heartbeat_tier_falls_through_to_target_then_new_voters() { + let root = H256([0u8; 32]); + // Equal committee coverage: newer target leads. + assert!( + score(Tier::Heartbeat, 4, 0, 9).ordering_key(root) + < score(Tier::Heartbeat, 4, 0, 8).ordering_key(root) + ); + // Equal coverage and target: non-committee bits get their say here. + assert!( + score(Tier::Heartbeat, 4, 10, 9).ordering_key(root) + < score(Tier::Heartbeat, 4, 2, 9).ordering_key(root) + ); + // Fully equal: deterministic tiebreak on data_root, ascending. + let low = score(Tier::Heartbeat, 4, 2, 9).ordering_key(H256([1u8; 32])); + let high = score(Tier::Heartbeat, 4, 2, 9).ordering_key(H256([2u8; 32])); + assert!(low < high); + } + + #[test] + fn heartbeat_coverage_counts_committee_members_only() { + let projected = empty_projection(); + let tier = hb_tier(); + let data = make_att_data(HB_BLOCK_SLOT - 1); + + // Committee at slot 7 is {7, 0, 1, 2}. Three members plus two outsiders. + let three_members: HashSet = HashSet::from([7, 0, 1, 3, 4]); + let (three, _) = projected + .score_entry_with_heartbeat(&data, &three_members, HB_VALIDATORS as usize, Some(&tier)) + .expect("covers committee members"); + assert_eq!(three.committee_voters, 3, "outsiders must not be counted"); + + // Four members and no outsiders at all. + let four_members: HashSet = HashSet::from([7, 0, 1, 2]); + let (four, _) = projected + .score_entry_with_heartbeat(&data, &four_members, HB_VALIDATORS as usize, Some(&tier)) + .expect("covers committee members"); + assert_eq!(four.committee_voters, 4); + + // And the thicker committee coverage wins despite fewer total voters. + let root = H256([0u8; 32]); + assert!(four.ordering_key(root) < three.ordering_key(root)); + } + + #[test] + fn heartbeat_tier_gate_is_strictly_the_previous_slot() { + let projected = empty_projection(); + let tier = hb_tier(); + let coverage: HashSet = HashSet::from([7, 0, 1, 2]); + + // Exactly block.slot - 1 enters the tier. + let (on_slot, _) = projected + .score_entry_with_heartbeat( + &make_att_data(HB_BLOCK_SLOT - 1), + &coverage, + HB_VALIDATORS as usize, + Some(&tier), + ) + .unwrap(); + assert_eq!(on_slot.tier, Tier::Heartbeat); + + // block.slot - 2 does not, even though its signers are committee members + // for *that* slot too. No window, no fallback: this mirrors + // `store::extract_heartbeat_votes` exactly. + let (two_back, _) = projected + .score_entry_with_heartbeat( + &make_att_data(HB_BLOCK_SLOT - 2), + &coverage, + HB_VALIDATORS as usize, + Some(&tier), + ) + .unwrap(); + assert_ne!(two_back.tier, Tier::Heartbeat); + assert_eq!(two_back.committee_voters, 0); + + // A block at the block's own slot is not the previous slot either. + let (same_slot, _) = projected + .score_entry_with_heartbeat( + &make_att_data(HB_BLOCK_SLOT), + &coverage, + HB_VALIDATORS as usize, + Some(&tier), + ) + .unwrap(); + assert_ne!(same_slot.tier, Tier::Heartbeat); + } + + #[test] + fn heartbeat_tier_gate_derives_from_the_block_not_the_clock() { + // `propose_block` advances the store to the next slot's interval 0 before + // building, so a gate read from `store.time()` would see S+1 and match + // nothing. `HeartbeatTier` only ever sees the block slot, and this pins + // that: building block T gates on T-1 for every T. + for block_slot in 1..12u64 { + let tier = HeartbeatTier::for_block(block_slot, HB_VALIDATORS, HB_COMMITTEE); + assert!(tier.covers(&make_att_data(block_slot - 1))); + assert!(!tier.covers(&make_att_data(block_slot))); + } + // Slot 0 has no predecessor committee and must not panic. + let genesis_tier = HeartbeatTier::for_block(0, HB_VALIDATORS, HB_COMMITTEE); + assert!(!genesis_tier.covers(&make_att_data(0))); + assert_eq!(genesis_tier.count_members(&HashSet::from([0, 1])), 0); + } + + #[test] + fn heartbeat_tier_bypasses_the_zero_new_voters_drop() { + let data = make_att_data(HB_BLOCK_SLOT - 1); + let coverage: HashSet = HashSet::from([7, 0, 1, 2]); + + // Every signer is already justification-covered for this target root. + let mut current_votes = HashMap::new(); + current_votes.insert(data.target.root, coverage.clone()); + let projected = ProjectedState { + justified_slots: JustifiedSlots::new(), + finalized_slot: 0, + current_votes, + }; + + // Without the heartbeat tier the entry has no marginal justification value + // and is dropped. + assert!( + projected + .score_entry(&data, &coverage, HB_VALIDATORS as usize) + .is_none(), + "the zero-new-voters drop should still apply outside the tier" + ); + + // With it, positive committee coverage is itself the score: the vote has to + // reach other nodes' heartbeat store via the body regardless of whether it + // adds anything to justification. This is the regression that would + // silently empty view-merge while every other test stayed green. + let (score, new_voters) = projected + .score_entry_with_heartbeat(&data, &coverage, HB_VALIDATORS as usize, Some(&hb_tier())) + .expect("heartbeat entries bypass the drop"); + assert_eq!(score.tier, Tier::Heartbeat); + assert_eq!(score.committee_voters, 4); + assert!(new_voters.is_empty(), "genuinely adds no new voters"); + } + + #[test] + fn heartbeat_tier_does_not_justify_the_projection_by_itself() { + // `Tier::Heartbeat = 0` sorts below `Tier::Justify`, so keying `advance` + // off the sort tier would let a thin committee vote mark its target + // justified. `EntryEffect` is what `advance` reads, and it is computed + // independently of the tier. + // + // Target slot 5 against finalized slot 0, so "is the target justified" is + // a real question — slot 0 would be trivially justified as the finalized + // slot itself. + let data = AttestationData { + slot: HB_BLOCK_SLOT - 1, + head: Checkpoint::default(), + target: Checkpoint { + slot: 5, + root: H256([5u8; 32]), + }, + source: Checkpoint::default(), + }; + // One of eight validators: nowhere near 2/3. + let coverage: HashSet = HashSet::from([7]); + + let (score, new_voters) = empty_projection() + .score_entry_with_heartbeat(&data, &coverage, HB_VALIDATORS as usize, Some(&hb_tier())) + .unwrap(); + assert_eq!(score.tier, Tier::Heartbeat); + assert_eq!( + score.effect, + EntryEffect::Builds, + "a thin committee vote justifies nothing" + ); + + let mut advanced = empty_projection(); + advanced.advance(score.effect, &data, new_voters); + assert!( + !justified_slots_ops::is_slot_justified( + &advanced.justified_slots, + advanced.finalized_slot, + data.target.slot, + ) + // Same reading as the packer's pre-filter: a slot the projection + // window does not track is not justified yet. + .unwrap_or(false), + "a heartbeat entry must not justify its target on tier alone" + ); + + // Contrast: the same entry carrying a real justifying effect does advance + // the projection, so the assertion above is about the effect and not about + // `advance` being inert. + let mut justified = empty_projection(); + justified.advance(EntryEffect::Justifies, &data, HashSet::from([7])); + assert!( + justified_slots_ops::is_slot_justified( + &justified.justified_slots, + justified.finalized_slot, + data.target.slot, + ) + .unwrap_or(false) + ); + } + + /// Under cap pressure the surviving heartbeat entries must be the + /// *highest-coverage* ones, not an arbitrary subset. That is what turns risk 5 + /// from "the fast head silently loses its evidence" into "the fast head loses + /// the thinnest fragments": the assertion is on *which* entries survive, not + /// just how many. + #[test] + fn select_attestations_under_cap_keeps_the_thickest_committee_coverage() { + use ethlambda_types::{ + block::BlockHeader, + state::{ChainConfig, JustificationValidators, JustifiedSlots, Validator}, + }; + use libssz_types::SszList; + + // n = 8, K = 4, block slot 8 -> committee slot 7 -> committee {7, 0, 1, 2}. + const NUM_VALIDATORS: usize = 8; + const HEAD_SLOT: u64 = 7; + const BLOCK_SLOT: u64 = 8; + const LIMIT: usize = 2; + + let validators: Vec<_> = (0..NUM_VALIDATORS) + .map(|i| Validator { + attestation_pubkey: [i as u8; 52], + proposal_pubkey: [i as u8; 52], + index: i as u64, + }) + .collect(); + + let hashes: Vec = (0..HEAD_SLOT).map(|i| H256([(i + 1) as u8; 32])).collect(); + let head_header = BlockHeader { + slot: HEAD_SLOT, + proposer_index: 0, + parent_root: H256::ZERO, + state_root: H256::ZERO, + body_root: BlockBody::default().hash_tree_root(), + }; + let head_state = State { + config: ChainConfig { genesis_time: 1000 }, + slot: HEAD_SLOT, + latest_block_header: head_header, + latest_justified: Checkpoint::default(), + latest_finalized: Checkpoint::default(), + historical_block_hashes: SszList::try_from(hashes.clone()).unwrap(), + justified_slots: JustifiedSlots::new(), + validators: SszList::try_from(validators).unwrap(), + justifications_roots: Default::default(), + justifications_validators: JustificationValidators::new(), + }; + + let mut header_for_root = head_state.latest_block_header.clone(); + header_for_root.state_root = head_state.hash_tree_root(); + let parent_root = header_for_root.hash_tree_root(); + + // Source == target == slot 0 makes each entry a genesis self-vote, which is + // exempt from the already-justified filter; varying `head` across chain + // slots is what makes the four datas distinct while all staying on the + // committee slot. + let anchor = Checkpoint { + root: hashes[0], + slot: 0, + }; + let mut known_block_roots = HashSet::new(); + known_block_roots.insert(parent_root); + + // Committee subsets of decreasing size, each with a distinct `head`. + let cohorts: [&[usize]; 4] = [&[7, 0, 1, 2], &[7, 0, 1], &[7, 0], &[7]]; + let mut aggregated_payloads: HashMap)> = + HashMap::new(); + let mut expected_by_coverage: Vec<(usize, H256)> = Vec::new(); + + for (i, signers) in cohorts.iter().enumerate() { + let head = Checkpoint { + root: hashes[i], + slot: i as u64, + }; + known_block_roots.insert(head.root); + let att_data = AttestationData { + slot: HEAD_SLOT, + head, + target: anchor, + source: anchor, + }; + let data_root = att_data.hash_tree_root(); + let proof_data = SszList::try_from(vec![0xABu8; 8]).expect("proof fits"); + let proof = SingleMessageAggregate::new(make_bits(signers), proof_data); + aggregated_payloads.insert(data_root, (att_data, vec![proof])); + expected_by_coverage.push((signers.len(), data_root)); + } + + let selected = select_attestations( + &head_state, + BLOCK_SLOT, + parent_root, + &known_block_roots, + &aggregated_payloads, + HB_COMMITTEE, + LIMIT, + ); + + let picked: HashSet = selected + .iter() + .map(|(att, _)| att.data.hash_tree_root()) + .collect(); + assert_eq!(picked.len(), LIMIT, "the limit should bind"); + + expected_by_coverage.sort_by_key(|(coverage, _)| std::cmp::Reverse(*coverage)); + for (coverage, data_root) in &expected_by_coverage[..LIMIT] { + assert!( + picked.contains(data_root), + "entry with {coverage} committee members should survive the cap" + ); + } + for (coverage, data_root) in &expected_by_coverage[LIMIT..] { + assert!( + !picked.contains(data_root), + "entry with only {coverage} committee members should be dropped first" + ); + } + } + /// Regression (leanSpec #802): a supermajority entry whose source sits at /// the finalized boundary must be scored `Justify`, not `Finalize`. Such a /// source is already final, so it advances nothing; the empty scan range @@ -1088,6 +1639,7 @@ mod tests { ProposerConfig { enable_proposer_aggregation: true, max_attestations_per_block: MAX_ATTESTATIONS_DATA, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, }, ) .expect("build_block should succeed"); @@ -1234,6 +1786,7 @@ mod tests { ProposerConfig { enable_proposer_aggregation: false, max_attestations_per_block: limit, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, }, ) .expect("build_block should succeed") @@ -1360,6 +1913,7 @@ mod tests { ProposerConfig { enable_proposer_aggregation: false, max_attestations_per_block: MAX_ATTESTATIONS_DATA, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, }, ) .expect("build_block should succeed"); @@ -1666,6 +2220,7 @@ mod tests { ProposerConfig { enable_proposer_aggregation: true, max_attestations_per_block: MAX_ATTESTATIONS_DATA, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, }, ) .expect("build_block should succeed"); @@ -1802,6 +2357,7 @@ mod tests { ProposerConfig { enable_proposer_aggregation: true, max_attestations_per_block: MAX_ATTESTATIONS_DATA, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, }, ) .expect("build_block should succeed"); diff --git a/crates/blockchain/src/heartbeat_fold.rs b/crates/blockchain/src/heartbeat_fold.rs new file mode 100644 index 00000000..0808c0b4 --- /dev/null +++ b/crates/blockchain/src/heartbeat_fold.rs @@ -0,0 +1,414 @@ +//! Heartbeat fold: the next-slot proposer's job source for the interval-2 +//! aggregation pipeline. +//! +//! Turns the raw committee signatures collected on the global heartbeat topic +//! into one type-1 aggregate, so the proposer can pack the committee's votes +//! without waiting for subnet aggregation to build and propagate one. +//! +//! This is a *job source*, not a second session: the produced job runs through +//! the same [`crate::aggregation`] worker, deadline, and `AggregateProduced` +//! machinery as a subnet job. That is what makes the result reach the builder +//! through the ordinary payload pool — inserted into `new_payloads` at the +//! interval-2 boundary, promoted to `known_payloads` at interval 3, and scored by +//! `select_attestations` like any other candidate — with no special path into the +//! block builder. +//! +//! Only the next slot's proposer runs it. Every other node's copy of these votes +//! reaches fork choice directly from the gossip topic, so folding them would be +//! prover work with no consumer. +//! +//! **Raw signatures only: the fold never recurses into an existing type-1.** The +//! job it emits carries an empty `children` list, which degenerates the shared +//! `aggregate_mixed` call to a plain single-message aggregation over raw XMSS, and +//! its coverage is exactly the buffered signers it was handed. The cost is that +//! the fold cannot union its buffer with a partially-overlapping existing proof: +//! both become separate candidates for the same `AttestationData`, and the +//! builder's per-data pick (`keep_best_proof_per_data`) keeps whichever covers +//! more. +//! +//! The common case is still free. Honest committee members with the same view +//! produce byte-identical `AttestationData`, so when a single existing type-1 +//! already covers every buffered signer the fold would only reproduce coverage the +//! block can already pack, and no job is emitted at all. + +use std::collections::{BTreeMap, HashMap, HashSet}; + +use ethlambda_crypto::signature::{ValidatorPublicKey, ValidatorSignature}; +use ethlambda_storage::Store; +use ethlambda_types::{ + ShortRoot, + attestation::{AttestationData, HashedAttestationData}, + block::SingleMessageAggregate, + primitives::H256, + state::Validator, +}; +use tracing::trace; + +use crate::aggregation::{AggregationJob, AggregationSnapshot, JobSource}; +use crate::metrics; + +/// Build the next-slot proposer's single aggregation job over `slot`'s heartbeat +/// committee votes. Runs on the actor thread; touches the store, does no heavy +/// cryptography. +/// +/// A committee split can leave several distinct `AttestationData` among the +/// buffered signatures. Exactly one job is emitted, choosing the data with the +/// most buffered committee signers — the same coverage-led preference +/// `Tier::Heartbeat` applies when packing, so the job produced is the entry the +/// block most wants. Ties break on `data_root` ascending for determinism. +/// +/// Returns `None` when nothing is foldable, either because no heartbeat signature +/// is buffered or because a single existing type-1 already covers every buffered +/// signer. The caller falls back to an ordinary subnet job in that case. +pub fn heartbeat_aggregation_snapshot(store: &Store, slot: u64) -> Option { + let buffered = store.heartbeat_signatures_at(slot); + if buffered.is_empty() { + return None; + } + + let validators = store.head_state().validators; + + // `AttestationData` per buffered data_root, recovered from the vote store: + // both are written together in `on_gossip_heartbeat_attestation`. + let data_by_root: HashMap = store + .heartbeat_votes_at(slot) + .into_values() + .map(|data| (HashedAttestationData::new(data.clone()).root(), data)) + .collect(); + + // Descending buffered-signer count, then ascending data_root. + let mut by_coverage: Vec<(&H256, &BTreeMap)> = + buffered.iter().collect(); + by_coverage.sort_by_key(|(root, sigs)| (std::cmp::Reverse(sigs.len()), **root)); + + let groups_considered = by_coverage.len(); + for (data_root, sigs_by_validator) in by_coverage { + let Some(data) = data_by_root.get(data_root) else { + // The vote was pruned out from under the signature buffer; a + // signature is unusable without its data. + continue; + }; + // Cheap pre-check: with no existing proof for this data there is nothing + // the fold could be dominated by, so skip the clone of every proof blob. + let existing_proofs = if store.proof_count_for_data(data_root) == 0 { + Vec::new() + } else { + let (new_proofs, known_proofs) = store.existing_proofs_for_data(data_root); + new_proofs.into_iter().chain(known_proofs).collect() + }; + if let Some(job) = resolve_fold_job( + HashedAttestationData::new(data.clone()), + sigs_by_validator, + &existing_proofs, + &validators, + ) { + trace!( + %slot, + data_root = %ShortRoot(&data_root.0), + raw_count = job.raw_ids.len(), + "Heartbeat fold job selected" + ); + return Some(AggregationSnapshot { + jobs: vec![job], + groups_considered, + }); + } + } + + None +} + +/// Build one raw-only fold job, or `None` when there is nothing worth proving. +/// +/// Every buffered signature the validator registry can resolve to a pubkey goes in +/// raw, in ascending validator order (XMSS aggregation requires it, which the +/// `BTreeMap` iteration order already gives us). `children` stays empty by +/// construction, so `aggregate_mixed`'s "at least one raw signature OR at least two +/// children" precondition reduces to "at least one resolvable signature" — the +/// exact condition under which a job is emitted. +/// +/// A lone raw signature is still a job, unlike on the subnet path: a committee +/// vote can only enter a block as a type-1, so a 1-of-1 aggregate is the cheapest +/// form in which the proposer can pack it. +fn resolve_fold_job( + hashed: HashedAttestationData, + sigs_by_validator: &BTreeMap, + existing_proofs: &[SingleMessageAggregate], + validators: &[Validator], +) -> Option { + let mut raw_pubkeys = Vec::new(); + let mut raw_sigs = Vec::new(); + let mut raw_ids = Vec::new(); + for (validator_id, signature) in sigs_by_validator { + let Some(validator) = validators.get(*validator_id as usize) else { + continue; + }; + let Ok(pubkey) = ValidatorPublicKey::from_bytes(&validator.attestation_pubkey) else { + continue; + }; + raw_pubkeys.push(pubkey); + raw_sigs.push(signature.clone()); + raw_ids.push(*validator_id); + } + + if raw_ids.is_empty() { + return None; + } + + // A single existing type-1 covering every buffered signer already carries + // everything a raw-only fold could produce, so proving it again buys the block + // nothing. Dominance has to come from *one* proof: with proposer aggregation + // off (the default) the builder keeps the single best-coverage proof per data + // rather than merging them, so coverage spread across two proofs is not + // coverage the block can pack. + if is_dominated(&raw_ids, existing_proofs) { + metrics::inc_heartbeat_fold_skipped(); + trace!( + data_root = %ShortRoot(&hashed.root().0), + raw_count = raw_ids.len(), + "Heartbeat fold skipped: an existing type-1 already covers every buffered signer" + ); + return None; + } + + let slot = hashed.data().slot; + Some(AggregationJob { + hashed, + slot, + // Raw signatures only — see the module docs. Empty children make the + // shared `aggregate_mixed` call a plain single-message aggregation. + children: Vec::new(), + accepted_child_ids: Vec::new(), + raw_pubkeys, + raw_sigs, + raw_ids, + // Heartbeat signatures live in their own slot-keyed buffer, pruned by the + // RLMD window rather than consumed on aggregation, so there is nothing to + // delete from the gossip pool here. + keys_to_delete: Vec::new(), + source: JobSource::Heartbeat, + }) +} + +/// Whether some single existing type-1's participants are a superset of `raw_ids`. +fn is_dominated(raw_ids: &[u64], existing_proofs: &[SingleMessageAggregate]) -> bool { + existing_proofs.iter().any(|proof| { + let participants: HashSet = proof.participant_indices().collect(); + raw_ids.iter().all(|vid| participants.contains(vid)) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use ethlambda_types::checkpoint::Checkpoint; + use libssz_types::SszList; + + fn validators(n: usize) -> Vec { + (0..n) + .map(|i| Validator { + attestation_pubkey: [i as u8; 52], + proposal_pubkey: [i as u8; 52], + index: i as u64, + }) + .collect() + } + + fn hashed(slot: u64) -> HashedAttestationData { + HashedAttestationData::new(AttestationData { + slot, + head: Checkpoint::default(), + target: Checkpoint::default(), + source: Checkpoint::default(), + }) + } + + /// An existing type-1 covering exactly `signers`. + fn existing_proof(signers: &[usize]) -> SingleMessageAggregate { + let max = signers.iter().copied().max().unwrap_or(0); + let mut bits = ethlambda_types::attestation::AggregationBits::with_length(max + 1).unwrap(); + for &i in signers { + bits.set(i, true).unwrap(); + } + let proof = SszList::try_from(vec![0xABu8; 8]).expect("proof fits"); + SingleMessageAggregate::new(bits, proof) + } + + /// A cheap-but-real XMSS signature (tiny lifetime, cached) for tests that only + /// need `ValidatorSignature::from_bytes` to succeed. `resolve_fold_job` never + /// checks validity, only that the signature clones and carries a resolvable + /// id — mirrors `aggregation::tests::dummy_sig`. + fn dummy_sig() -> ValidatorSignature { + use ethlambda_crypto::signature::LeanSignatureScheme; + use leansig::{serialization::Serializable, signature::SignatureScheme}; + use rand::{SeedableRng, rngs::StdRng}; + + static CACHED_SIG: std::sync::LazyLock> = std::sync::LazyLock::new(|| { + let mut rng = StdRng::seed_from_u64(42); + let lifetime = 1 << 5; // small for speed + let (_pk, sk) = LeanSignatureScheme::key_gen(&mut rng, 0, lifetime); + let sig = LeanSignatureScheme::sign(&sk, 0, &[0u8; 32]).unwrap(); + sig.to_bytes() + }); + + ValidatorSignature::from_bytes(&CACHED_SIG).expect("cached test signature") + } + + /// Buffered raw heartbeat signatures for `signers`. + fn buffered(signers: &[u64]) -> BTreeMap { + signers.iter().map(|vid| (*vid, dummy_sig())).collect() + } + + #[test] + fn fold_keeps_every_buffered_signer_raw() { + // An existing type-1 covers {0, 1} and the buffer holds {0, 1, 2}. The fold + // does not recurse into it, so all three signatures go in raw and the + // resulting aggregate covers exactly the buffer. + let job = resolve_fold_job( + hashed(7), + &buffered(&[0, 1, 2]), + &[existing_proof(&[0, 1])], + &validators(4), + ) + .expect("the existing proof does not cover 2, so the fold runs"); + + assert_eq!(job.raw_ids, vec![0, 1, 2], "no signer is dropped"); + assert!( + job.children.is_empty() && job.accepted_child_ids.is_empty(), + "raw signatures only: no existing type-1 is recursed into" + ); + // The pubkey and signature lists stay aligned with raw_ids; a mismatch is + // an `aggregate_mixed` CountMismatch error. + assert_eq!(job.raw_pubkeys.len(), job.raw_ids.len()); + assert_eq!(job.raw_sigs.len(), job.raw_ids.len()); + assert_eq!(job.source, JobSource::Heartbeat); + assert!( + job.keys_to_delete.is_empty(), + "heartbeat signatures are window-pruned, not consumed from the gossip pool" + ); + } + + #[test] + fn fold_is_skipped_when_one_proof_covers_every_signer() { + // The common case: a subnet aggregate for the same data already covers the + // whole buffer, so a raw-only fold would only reproduce coverage the block + // can already pack. + let job = resolve_fold_job( + hashed(7), + &buffered(&[0, 1]), + &[existing_proof(&[0, 1, 2])], + &validators(4), + ); + assert!(job.is_none(), "a dominating type-1 means no work"); + } + + #[test] + fn fold_runs_when_coverage_is_split_across_proofs() { + // Two proofs jointly cover {0, 1, 2}, but neither covers it alone. Their + // union is only reachable by recursion, which the fold does not do, and the + // builder keeps one proof per data rather than merging them — so the raw + // fold is what gets the whole buffer into a single packable entry. + let job = resolve_fold_job( + hashed(7), + &buffered(&[0, 1, 2]), + &[existing_proof(&[0, 1]), existing_proof(&[2])], + &validators(4), + ) + .expect("no single proof dominates the buffer"); + assert_eq!(job.raw_ids, vec![0, 1, 2]); + assert!(job.children.is_empty()); + } + + #[test] + fn fold_with_no_existing_proof_aggregates_raw_signatures() { + // A committee member whose data no aggregate covers still folds: a lone raw + // signature satisfies `aggregate_mixed`'s "at least one raw signature" + // precondition, and the resulting entry is a real finality vote. + let job = resolve_fold_job(hashed(7), &buffered(&[3]), &[], &validators(4)) + .expect("raw-only fold is valid"); + assert!(job.children.is_empty()); + assert_eq!(job.raw_ids, vec![3]); + } + + #[test] + fn fold_skips_signers_outside_the_registry() { + // A signature from an index the registry does not have cannot be resolved + // to a pubkey; dropping it beats passing a short pubkey list to the prover. + let job = resolve_fold_job(hashed(7), &buffered(&[1, 99]), &[], &validators(4)) + .expect("validator 1 is resolvable"); + assert_eq!(job.raw_ids, vec![1]); + } + + /// Distinguishable data for the same slot, so a committee split produces + /// several buffered `data_root`s. + fn split_data(slot: u64, target_marker: u8) -> AttestationData { + AttestationData { + slot, + head: Checkpoint::default(), + target: Checkpoint { + root: H256([target_marker; 32]), + slot, + }, + source: Checkpoint::default(), + } + } + + #[test] + fn snapshot_emits_one_job_for_the_thickest_committee_split() { + use ethlambda_storage::backend::InMemoryBackend; + use ethlambda_types::state::State; + use std::sync::Arc; + + let state = State::from_genesis(1000, validators(8)); + let store = Store::from_anchor_state(Arc::new(InMemoryBackend::new()), state); + + // A committee split: two signers on one data, three on another. + let thin = split_data(7, 0xAA); + let thick = split_data(7, 0xBB); + for (vid, data) in [(0u64, &thin), (1, &thin)] { + store.insert_heartbeat_vote(vid, data.clone()); + store.insert_heartbeat_signature( + 7, + HashedAttestationData::new(data.clone()).root(), + vid, + dummy_sig(), + ); + } + for (vid, data) in [(2u64, &thick), (3, &thick), (4, &thick)] { + store.insert_heartbeat_vote(vid, data.clone()); + store.insert_heartbeat_signature( + 7, + HashedAttestationData::new(data.clone()).root(), + vid, + dummy_sig(), + ); + } + + let snapshot = heartbeat_aggregation_snapshot(&store, 7).expect("a job is foldable"); + assert_eq!(snapshot.jobs.len(), 1, "exactly one job, always"); + assert_eq!( + snapshot.jobs[0].raw_ids, + vec![2, 3, 4], + "the thickest committee coverage wins, matching Tier::Heartbeat's ordering" + ); + assert_eq!(snapshot.groups_considered, 2); + } + + #[test] + fn snapshot_is_none_when_nothing_is_buffered() { + use ethlambda_storage::backend::InMemoryBackend; + use ethlambda_types::state::State; + use std::sync::Arc; + + let state = State::from_genesis(1000, validators(8)); + let store = Store::from_anchor_state(Arc::new(InMemoryBackend::new()), state); + + // No heartbeat signatures at all: the caller falls back to a subnet job. + assert!(heartbeat_aggregation_snapshot(&store, 7).is_none()); + + // A vote whose signature buffer entry is for a different slot is not + // foldable at this slot either. + store.insert_heartbeat_vote(0, split_data(6, 0xAA)); + assert!(heartbeat_aggregation_snapshot(&store, 7).is_none()); + } +} diff --git a/crates/blockchain/src/lib.rs b/crates/blockchain/src/lib.rs index 7faa1163..dfb03209 100644 --- a/crates/blockchain/src/lib.rs +++ b/crates/blockchain/src/lib.rs @@ -3,7 +3,9 @@ use std::time::{Duration, Instant, SystemTime}; use ethlambda_crypto::signature::{ValidatorPublicKey, ValidatorSignature}; use ethlambda_network_api::{BlockChainToP2PRef, BlockSource, InitP2P}; -use ethlambda_state_transition::is_proposer; +use ethlambda_state_transition::{ + effective_heartbeat_committee_size, is_heartbeat_committee_member, is_proposer, +}; use ethlambda_storage::{ALL_TABLES, Store}; use ethlambda_types::{ ShortRoot, @@ -16,8 +18,9 @@ use ethlambda_types::{ use crate::aggregation::{ AGGREGATION_DEADLINE, AggregateProduced, AggregationDeadline, AggregationDone, AggregationSession, EARLY_AGGREGATION_WINDOW, EarlyAggregationCheck, MAX_AGGREGATION_JOBS, - PRIOR_WORKER_JOIN_TIMEOUT, run_aggregation_worker, + PRIOR_WORKER_JOIN_TIMEOUT, SlotOrdering, run_aggregation_worker, }; +use crate::heartbeat_fold::heartbeat_aggregation_snapshot; use crate::key_manager::ValidatorKeyPair; use crate::sync_status::SyncStatusTracker; use spawned_concurrency::actor; @@ -38,6 +41,7 @@ pub mod block_builder; pub(crate) mod coverage; pub mod events; pub(crate) mod fork_choice_tree; +pub mod heartbeat_fold; pub mod key_manager; pub mod metrics; pub mod reaggregate; @@ -68,29 +72,42 @@ pub struct BlockChainConfig { pub proposer_config: ProposerConfig, } -/// Milliseconds per interval (800ms ticks). -pub const MILLISECONDS_PER_INTERVAL: u64 = 800; -/// Number of intervals per slot (5 intervals of 800ms = 4 seconds). -pub const INTERVALS_PER_SLOT: u64 = 5; -/// Milliseconds in a slot (derived from interval duration and count). -pub const MILLISECONDS_PER_SLOT: u64 = MILLISECONDS_PER_INTERVAL * INTERVALS_PER_SLOT; +// The interval grid lives in `ethlambda-types` because `ethlambda-storage` also +// derives slots from `store.time()` and must not carry a second copy of a +// consensus-critical constant. pub use ethlambda_types::block::MAX_ATTESTATIONS_DATA; +pub use ethlambda_types::constants::{ + INTERVALS_PER_SLOT, MILLISECONDS_PER_INTERVAL, MILLISECONDS_PER_SLOT, RLMD_LOOKBACK_LIMIT, +}; pub use sync_status::SyncStatusController; /// Future-slot tolerance for gossip attestations, expressed in intervals. /// /// Bounds the clock skew the time check is willing to absorb when admitting a -/// vote whose slot has not yet started locally. One interval is roughly 800 ms, +/// vote whose slot has not yet started locally. One interval is 1000 ms, /// the lean analogue of mainnet's `MAXIMUM_GOSSIP_CLOCK_DISPARITY`. /// /// See: leanSpec PR #682. pub const GOSSIP_DISPARITY_INTERVALS: u64 = 1; +/// The four ticks inside a slot. +/// +/// Declared in wall-clock order; the discriminant is the interval index within +/// the slot. The 5-interval grid's `Aggregation` tick is gone: committee +/// aggregation is now anchored to the interval-2 boundary rather than occupying +/// a tick of its own, so its duties fold into [`Self::SafeTargetUpdate`]. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum SlotInterval { + /// 0 ms. The slot's block arrives; import merges committee bits into the + /// heartbeat vote store and promotes new payloads to known. BlockPublication, + /// 1000 ms. Every validator attests to its subnet; committee members also + /// republish that same signature to the global heartbeat topic. AttestationProduction, - Aggregation, + /// 2000 ms. `safe_target` is recomputed from current-slot heartbeat votes; + /// committee aggregates begin publishing. SafeTargetUpdate, + /// 3000 ms. `lagging_head` then `fast_head`; promote payloads, log the tree, + /// and build + publish the next slot's block aligned to its slot boundary. EndOfSlot, } @@ -103,10 +120,9 @@ impl SlotInterval { match intervals_since_genesis % INTERVALS_PER_SLOT { 0 => Self::BlockPublication, 1 => Self::AttestationProduction, - 2 => Self::Aggregation, - 3 => Self::SafeTargetUpdate, - 4 => Self::EndOfSlot, - _ => unreachable!("slots only have 5 intervals"), + 2 => Self::SafeTargetUpdate, + 3 => Self::EndOfSlot, + _ => unreachable!("slots only have {INTERVALS_PER_SLOT} intervals"), } } @@ -117,9 +133,8 @@ impl SlotInterval { let interval = match self { Self::BlockPublication => 0, Self::AttestationProduction => 1, - Self::Aggregation => 2, - Self::SafeTargetUpdate => 3, - Self::EndOfSlot => 4, + Self::SafeTargetUpdate => 2, + Self::EndOfSlot => 3, }; slot * MILLISECONDS_PER_SLOT + interval * MILLISECONDS_PER_INTERVAL } @@ -167,6 +182,11 @@ impl BlockChain { let genesis_time = store.config().genesis_time; let mut key_manager = key_manager::KeyManager::new(validator_keys); + // Rebuild the in-memory heartbeat vote set from stored blocks before the + // first tick, so fork choice is not blind for a full RLMD window after a + // restart. + store::replay_heartbeat_votes(&store); + // Catch XMSS keys up to the current slot before the first tick // store.time() doesn't work here: after an offline gap it lags wall-clock by // exactly the gap we need to catch up through @@ -262,7 +282,7 @@ pub struct BlockChainServer { proposer_config: ProposerConfig, /// Pre-merge `new_payloads` snapshot for the attestation aggregate coverage - /// report. Captured at the end-of-slot promote (interval 4), read at the + /// report. Captured at the last promote of the slot (interval 3), read at the /// next slot boundary. Owned solely by the actor and only touched from the /// single-threaded message loop, so no synchronization is needed. /// Observability-only. @@ -330,16 +350,16 @@ impl BlockChainServer { let is_aggregator = self.aggregator.is_enabled(); metrics::set_is_aggregator(is_aggregator); - // ==== interval 4 (pre-tick) ==== + // ==== interval 3 (pre-tick) ==== - // Snapshot the pre-merge `new_payloads` set at the end-of-slot promote - // (interval 4), so the post-block report for this round sees its + // Snapshot the pre-merge `new_payloads` set at the last promote of the + // slot (interval 3), so the post-block report for this round sees its // "timely" cohort just before it is promoted out of `new_payloads`. // - // Only interval 4 — not the proposer's interval-0 promote. By interval 0 + // Only interval 3 — not the proposer's interval-0 promote. By interval 0 // the round's votes have already been promoted at the previous slot's - // interval 4; `new_payloads` then holds only stragglers, and snapshotting - // them here would overwrite the good interval-4 snapshot the report still + // interval 3; `new_payloads` then holds only stragglers, and snapshotting + // them here would overwrite the good interval-3 snapshot the report still // needs (those stragglers surface in the `late` section instead). Skip // empty snapshots so a missed round keeps the last set we saw. Pure // observability. @@ -365,15 +385,15 @@ impl BlockChainServer { // at tick time), so it doubles as the wall-clock slot for the gate. pre_tick.diff_and_emit(&self.store, &self.events, slot); - // Per-interval duties for this tick. Intervals 0 (block publish) and 3 - // (safe-target update) are driven inside `store::on_tick` above, so they - // carry only a note below. + // Per-interval duties for this tick. Interval 0 (block publish) and the + // fork-choice half of interval 2 / interval 3 are driven inside + // `store::on_tick` above, so they carry only a note below. match interval { // ==== interval 0 ==== // // No actor work at interval 0. The block is published here // conceptually (at the slot boundary), but the build+publish code - // path runs at interval 4 of the previous slot — where it also + // path runs at interval 3 of the previous slot — where it also // advances the store to this slot's interval 0 before building (see // `propose_block`). The real interval-0 tick is then skipped by the // idempotency guard above, since the store clock is already here. @@ -407,7 +427,13 @@ impl BlockChainServer { // Schedule the early-aggregation window check. This tick is // one interval before T2, so the timer fires right as the // window opens at T2 - EARLY_AGGREGATION_WINDOW. - if is_aggregator { + // + // Scheduled for the next slot's proposer as well as for + // aggregators. Without it a non-aggregator proposer would have no + // early-start opportunity at all: its heartbeat votes arrive + // during interval 1, before the window opens, so the per-insert + // checks all fall outside it and nothing would re-check. + if is_aggregator || self.proposes_next_slot(slot) { send_after( Duration::from_millis(MILLISECONDS_PER_INTERVAL) - EARLY_AGGREGATION_WINDOW, ctx.clone(), @@ -417,8 +443,16 @@ impl BlockChainServer { } // ==== interval 2 ==== - SlotInterval::Aggregation => { - if is_aggregator { + // + // The safe-target update itself is handled inside `store::on_tick`. + // Committee aggregation is anchored to this boundary: the session + // publishes from here, so its proofs are ready before the interval-3 + // build snapshots them. + SlotInterval::SafeTargetUpdate => { + // Two reasons to run a session: the committee-aggregator role, or + // proposing the next slot (which needs this slot's heartbeat votes + // folded into a packable type-1 before the interval-3 build). + if is_aggregator || self.proposes_next_slot(slot) { // The early trigger may have already started this slot's // session (running or finished) — it IS the slot's session, // so don't start a second one. @@ -436,18 +470,14 @@ impl BlockChainServer { // ==== interval 3 ==== // - // Safe-target update is handled inside `store::on_tick`. - SlotInterval::SafeTargetUpdate => {} - - // ==== interval 4 ==== - // // Build and publish the NEXT slot's block here, one interval early, // so the heavy leanVM work happens during this otherwise-idle // interval. `propose_block` blocks the actor for the build and aligns // publication to the slot boundary. Doing the whole proposal here — // rather than stashing it for the interval-0 tick — keeps it robust: // `on_tick` skips the interval-0 tick whenever this build overruns - // its interval. + // its interval. The head update itself runs inside `store::on_tick` + // above, so the build sees this slot's `fast_head` as its parent. SlotInterval::EndOfSlot => { let next_slot = slot + 1; let next_proposer = self @@ -460,15 +490,26 @@ impl BlockChainServer { } } - // Update safe target slot metric (updated by store.on_tick at interval 3) + // Update safe target slot metric (updated by store.on_tick at interval 2) metrics::update_safe_target_slot(self.store.safe_target_slot()); - // Update head slot metric (head may change when attestations are promoted at intervals 0/4) + // Update head slot metrics (head may change when attestations are promoted at intervals 0/3) metrics::update_head_slot(self.store.head_slot()); + metrics::update_lagging_head_slot(self.store.lagging_head_slot()); // Advance XMSS keys for next slot so the signing paths don't have to self.key_manager.advance_keys_to((slot + 1) as u32); } + /// Whether one of our validators proposes the slot after `slot`, and duties + /// are not suppressed by the sync gate. + /// + /// At slot `S` the block for `S+1` is what carries slot-`S` committee bits, so + /// "the next slot's proposer" is exactly the node that needs this slot's + /// heartbeat votes folded. + fn proposes_next_slot(&self, slot: u64) -> bool { + self.sync_status.duties_allowed() && self.get_our_proposer(slot + 1).is_some() + } + /// Kick off a committee-signature aggregation session: /// 1. If a prior session is still running (pathological), warn and join it. /// 2. Snapshot the aggregation inputs from the store, capped at a single job @@ -501,20 +542,41 @@ impl BlockChainServer { coverage::emit_agg_start_new_coverage(&self.store, self.attestation_committee_count); - // Limit ourselves to a single round of aggregation if we propose next round. - // This buys us time to build the block before the next slot's interval-0 tick. - let next_proposer = self - .get_our_proposer(slot + 1) - .filter(|_| self.sync_status.duties_allowed()); - let max_jobs = if next_proposer.is_some() { - 1 + // Proposing next round means one job, not `MAX_AGGREGATION_JOBS`: it buys + // time to build the block before the next slot's interval-0 tick. That one + // job is spent on the heartbeat committee's votes rather than on a subnet + // group, because the view-merge payload is what the block we are about to + // build most needs and the proposer's scarce resource is leanVM time before + // its interval-3 build. When nothing is foldable (no buffered heartbeat + // signature, or a single existing type-1 already covers every signer) we + // fall back to an ordinary single subnet job, still preferring this slot's + // groups since the block wants this slot's committee covered. + // + // Not proposing means the committee's votes are worth no queue jump: this + // node folds nothing, and every peer already has those votes raw off the + // global heartbeat topic. `TierOnly` therefore aggregates a current-slot + // group only when it wins on consensus value (finalizes, justifies, or + // adds coverage nothing else does), leaving the jobs for the finality + // progress only an aggregator can produce. + let snapshot = if self.proposes_next_slot(slot) { + heartbeat_aggregation_snapshot(&self.store, slot).or_else(|| { + aggregation::snapshot_aggregation_inputs( + &self.store, + slot, + 1, + SlotOrdering::CurrentSlotFirst, + ) + }) } else { - MAX_AGGREGATION_JOBS + aggregation::snapshot_aggregation_inputs( + &self.store, + slot, + MAX_AGGREGATION_JOBS, + SlotOrdering::TierOnly, + ) }; - - let Some(snapshot) = aggregation::snapshot_aggregation_inputs(&self.store, slot, max_jobs) - else { - // No current-slot gossip sigs — nothing to aggregate this slot. + let Some(snapshot) = snapshot else { + // Nothing to aggregate this slot. return; }; @@ -572,19 +634,26 @@ impl BlockChainServer { /// Early-aggregation trigger: start the slot's session ahead of the /// interval-2 tick when, inside the window `[T2 - EARLY_AGGREGATION_WINDOW, T2)`, - /// a single attestation-data group already holds 2/3 of the signatures - /// expected from this node's aggregation subnets. Called after every - /// stored current-slot gossip signature and once at the window opening via - /// [`EarlyAggregationCheck`]. Fires at most once per slot: the started - /// session stays in `current_aggregation` (running or finished) until the - /// next session replaces it. The latch has one hole: if the snapshot - /// yields no jobs (possible only when no signer's pubkey resolves, i.e. a - /// corrupted validator registry), no session is installed and the check - /// retries on later inserts — each retry is a no-op session attempt. + /// enough evidence is already in. There are two independent reasons to fire, + /// whichever comes first: + /// + /// - **Next slot's proposer**: this slot's heartbeat votes have reached the + /// safe-target threshold, `ceil(3K'/4)` of the committee. Same threshold the + /// safe target itself uses, and for the same reason: it is the point at which + /// the committee has spoken clearly enough to act on. Starting here is what + /// gives the fold room to finish before the interval-3 build. + /// - **Aggregator**: a single attestation-data group already holds 2/3 of the + /// signatures expected from this node's subscribed subnets. + /// + /// Called after every stored current-slot gossip signature, after every + /// accepted heartbeat vote, and once at the window opening via + /// [`EarlyAggregationCheck`]. Fires at most once per slot: the started session + /// stays in `current_aggregation` (running or finished) until the next session + /// replaces it. The latch has one hole: if the snapshot yields no jobs + /// (possible only when no signer's pubkey resolves, i.e. a corrupted validator + /// registry), no session is installed and the check retries on later inserts — + /// each retry is a no-op session attempt. async fn maybe_start_early_aggregation(&mut self, ctx: &Context) { - if !self.aggregator.is_enabled() { - return; - } // Only fire inside the early-aggregation window // `[T2 - EARLY_AGGREGATION_WINDOW, T2)`, where T2 is the current // slot's interval-2 boundary; the slot is derived from the wall clock. @@ -598,6 +667,10 @@ impl BlockChainServer { if ms_into_slot < t2_offset - window_ms || ms_into_slot >= t2_offset { return; } + // Everything below is denominated in this slot, including the proposer + // check: reading it from `store.time()` instead could disagree by one at a + // boundary, and then we would test the wrong slot's proposer against this + // slot's votes. let slot = ms_since_genesis / MILLISECONDS_PER_SLOT; if self .current_aggregation @@ -606,6 +679,18 @@ impl BlockChainServer { { return; } + // Heartbeat trigger. Checked first because it is the cheaper test and, + // for the proposer, the one that matters: the committee publishes at + // interval 1, so this is typically satisfied well before any subnet + // aggregate could be. + if self.proposes_next_slot(slot) && self.heartbeat_threshold_met(slot) { + self.start_aggregation_session(slot, ctx).await; + return; + } + if !self.aggregator.is_enabled() { + return; + } + let max_group = self.store.max_gossip_group_count_for_slot(slot); // Trigger once the largest current-slot group holds two-thirds of the // votes we expect it to collect, rounded up. Groups are keyed by @@ -644,6 +729,36 @@ impl BlockChainServer { self.start_aggregation_session(slot, ctx).await; } + /// Whether `slot`'s heartbeat votes have reached the safe-target threshold, + /// `ceil(3K'/4)` distinct committee voters. + /// + /// Denominated in the effective committee size `K' = min(K, n)`, never the raw + /// configured `K`: at `K > n` a raw denominator would demand more votes than + /// there are validators to cast them and the threshold could never be met. + fn heartbeat_threshold_met(&self, slot: u64) -> bool { + let num_validators = self.store.head_state().validators.len() as u64; + let committee_size = effective_heartbeat_committee_size( + self.store.heartbeat_committee_size(), + num_validators, + ); + if committee_size == 0 { + return false; + } + let threshold = (3 * committee_size).div_ceil(4); + let voters = self.store.heartbeat_voter_count_at(slot) as u64; + if voters < threshold { + return false; + } + info!( + %slot, + voters, + threshold, + committee_size, + "Heartbeat safe-target threshold met; starting proposer aggregation early" + ); + true + } + /// Returns the validator ID if any of our validators is the proposer for this slot. fn get_our_proposer(&self, slot: u64) -> Option { let head_state = self.store.head_state(); @@ -661,6 +776,12 @@ impl BlockChainServer { // Produce attestation data once for all validators let attestation_data = store::produce_attestation_data(&self.store, slot); + // Committee membership is denominated in the registry at the vote's own + // slot; the head state's registry is that registry, since the vote is for + // the current slot. + let num_validators = self.store.head_state().validators.len() as u64; + let heartbeat_committee_size = self.store.heartbeat_committee_size(); + // For each registered validator, produce and publish attestation for validator_id in self.key_manager.validator_ids() { // Sign the attestation @@ -692,6 +813,34 @@ impl BlockChainServer { }); } + // Committee members republish the *same* signature to the global + // heartbeat topic, so no extra XMSS epoch is consumed. Self-deliver + // it too: gossipsub does not echo to the sender, and unlike subnet + // aggregation this store is not gated on the aggregator role, so + // every node needs its own validators' votes locally. + if is_heartbeat_committee_member( + validator_id, + slot, + num_validators, + heartbeat_committee_size, + ) { + let _ = store::on_gossip_heartbeat_attestation( + &mut self.store, + &signed_attestation, + ) + .inspect_err(|err| { + warn!(%slot, %validator_id, %err, "Self-delivery of heartbeat vote failed") + }); + + if let Some(ref p2p) = self.p2p { + let _ = p2p + .publish_heartbeat_attestation(signed_attestation.clone()) + .inspect_err(|err| { + error!(%slot, %validator_id, %err, "Failed to publish heartbeat vote") + }); + } + } + // Publish to gossip network if let Some(ref p2p) = self.p2p { let _ = p2p.publish_attestation(signed_attestation).inspect_err( @@ -704,7 +853,7 @@ impl BlockChainServer { /// Build the target slot's block and publish it, one interval early. /// - /// Runs at the previous slot's interval 4, blocking the actor for the build + /// Runs at the previous slot's interval 3, blocking the actor for the build /// (the expensive part is the leanVM single-message → multi-message /// aggregate merge). It first /// advances the store to the target slot's interval 0 (accepting @@ -723,7 +872,7 @@ impl BlockChainServer { // Build the block. `produce_block_with_signatures` advances the store to // this slot's interval 0 (accepting attestations) before building — one - // interval ahead of the interval-4 tick we are running in — so the block + // interval ahead of the interval-3 tick we are running in — so the block // is built on the interval-0 state rather than the previous slot's end // state. Building early is safe because we publish below (nothing is // stashed for a later tick), and the real interval-0 tick is then skipped @@ -1366,7 +1515,7 @@ impl BlockChainServer { // --- Manual Handler impls for network-api messages --- use ethlambda_network_api::p2p_to_block_chain::{ - NewAggregatedAttestation, NewAttestation, NewBlock, + NewAggregatedAttestation, NewAttestation, NewBlock, NewHeartbeatAttestation, }; impl Handler for BlockChainServer { @@ -1420,6 +1569,20 @@ impl Handler for BlockChainServer { } } +impl Handler for BlockChainServer { + async fn handle(&mut self, msg: NewHeartbeatAttestation, ctx: &Context) { + let accepted = store::on_gossip_heartbeat_attestation(&mut self.store, &msg.attestation) + .inspect_err(|err| trace!(%err, "Rejected heartbeat attestation")) + .is_ok(); + // A heartbeat vote moves the next-slot proposer's early-start threshold, so + // re-check after each accepted one. Only current-slot votes are admitted at + // all, so no slot guard is needed here. + if accepted { + self.maybe_start_early_aggregation(ctx).await; + } + } +} + impl Handler for BlockChainServer { async fn handle(&mut self, msg: NewAggregatedAttestation, _ctx: &Context) { let arrival_ms = unix_now_ms(); diff --git a/crates/blockchain/src/metrics.rs b/crates/blockchain/src/metrics.rs index 41b70bd7..e51978b2 100644 --- a/crates/blockchain/src/metrics.rs +++ b/crates/blockchain/src/metrics.rs @@ -431,8 +431,12 @@ static LEAN_TICK_INTERVAL_DURATION_SECONDS: std::sync::LazyLock = register_histogram!( "lean_tick_interval_duration_seconds", "Elapsed time between clock ticks in seconds", + // Clustered tightly just above the interval width so ordinary jitter + // is still resolvable: the old buckets were centred on 0.8 s and + // would collapse every healthy sample into one bucket now that an + // interval is 1000 ms. vec![ - 0.4, 0.6, 0.75, 0.8, 0.805, 0.81, 0.815, 0.82, 0.825, 0.85, 0.9, 1.0, 1.2, 1.6 + 0.5, 0.75, 0.9, 1.0, 1.005, 1.01, 1.015, 1.02, 1.025, 1.05, 1.1, 1.25, 1.5, 2.0 ] ) .unwrap() @@ -690,8 +694,9 @@ pub fn observe_gossip_attestation_arrival(arrival_ms: u64, genesis_ms: u64, data } /// Observe a gossip aggregate's arrival against the most recent -/// [`crate::SlotInterval::Aggregation`] boundary at or before `arrival_ms`. -/// Zero point: that boundary — not the aggregate's own `data.slot`. +/// [`crate::SlotInterval::SafeTargetUpdate`] boundary at or before `arrival_ms`: +/// the interval-2 boundary aggregation sessions are anchored to. +/// Zero point: that boundary, not the aggregate's own `data.slot`. /// /// Deliberately takes no slot argument. An aggregate published at interval 2 /// of slot N can carry `data.slot < N` (the stale-group catch-up path in @@ -699,8 +704,11 @@ pub fn observe_gossip_attestation_arrival(arrival_ms: u64, genesis_ms: u64, data /// with large values that are not a health problem. Assuming the latest /// aggregation interval bounds the value to one slot. pub fn observe_gossip_aggregation_arrival(arrival_ms: u64, genesis_ms: u64) { - let delta_ms = - latest_interval_delta_ms(arrival_ms, genesis_ms, crate::SlotInterval::Aggregation); + let delta_ms = latest_interval_delta_ms( + arrival_ms, + genesis_ms, + crate::SlotInterval::SafeTargetUpdate, + ); LEAN_GOSSIP_AGGREGATION_ARRIVAL_DELAY_SECONDS .observe(Duration::from_millis(delta_ms.unsigned_abs()).as_secs_f64()); LEAN_GOSSIP_AGGREGATION_ARRIVAL_TOTAL @@ -913,6 +921,166 @@ pub fn update_safe_target_slot(slot: u64) { LEAN_SAFE_TARGET_SLOT.set(slot.try_into().unwrap()); } +// --- Heartbeat / two-tier fork choice --- + +/// Set the lagging-head gauge: the RLMD-window tree base. +/// +/// A frozen lagging head under a moving fast head is the RLMD-window failure +/// signature, so this is the panel to pair with `lean_fast_head_slot`. +pub fn update_lagging_head_slot(slot: u64) { + static LEAN_LAGGING_HEAD_SLOT: std::sync::LazyLock = std::sync::LazyLock::new(|| { + register_int_gauge!( + "lean_lagging_head_slot", + "Lagging head slot (RLMD window base)" + ) + .unwrap() + }); + LEAN_LAGGING_HEAD_SLOT.set(slot.try_into().unwrap()); +} + +/// Effective committee size `K' = min(K, n)`, so a config disagreement across a +/// network is visible without reading configs. +pub fn update_heartbeat_committee_size(size: u64) { + static LEAN_HEARTBEAT_COMMITTEE_SIZE: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_gauge!( + "lean_heartbeat_committee_size", + "Effective heartbeat committee size (min of configured size and validator count)" + ) + .unwrap() + }); + LEAN_HEARTBEAT_COMMITTEE_SIZE.set(size.try_into().unwrap()); +} + +/// Distinct committee voters seen for a slot. Read against `ceil(3K'/4)` this is +/// the safe-target stall predictor. +pub fn update_heartbeat_committee_participation(voters: u64) { + static LEAN_HEARTBEAT_COMMITTEE_PARTICIPATION: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_gauge!( + "lean_heartbeat_committee_participation", + "Distinct committee voters holding a heartbeat vote for the last slot" + ) + .unwrap() + }); + LEAN_HEARTBEAT_COMMITTEE_PARTICIPATION.set(voters.try_into().unwrap()); +} + +/// Count a heartbeat vote by where it came from. +/// +/// Paired with `lean_fast_head_window_slots` this tells you whether view-merge is +/// actually merging: `source="gossip"` winning means gossip beat the block. +pub fn inc_heartbeat_votes_received(source: HeartbeatVoteSource) { + static LEAN_HEARTBEAT_VOTES_RECEIVED_TOTAL: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_counter_vec!( + "lean_heartbeat_votes_received_total", + "Heartbeat votes recorded, by source", + &["source"] + ) + .unwrap() + }); + LEAN_HEARTBEAT_VOTES_RECEIVED_TOTAL + .with_label_values(&[source.as_str()]) + .inc(); +} + +/// Where a heartbeat vote entered the store. +#[derive(Clone, Copy, Debug)] +pub enum HeartbeatVoteSource { + /// The global heartbeat gossip topic, in-slot. + Gossip, + /// Extracted from an imported block's body. + Block, +} + +impl HeartbeatVoteSource { + fn as_str(self) -> &'static str { + match self { + Self::Gossip => "gossip", + Self::Block => "block", + } + } +} + +/// Committee bits found in a block's slot-`S-1` entries. Recovers the +/// observability a dedicated body field would have given. +pub fn observe_heartbeat_votes_in_block(votes: usize) { + static LEAN_HEARTBEAT_VOTES_IN_BLOCK: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_histogram!( + "lean_heartbeat_votes_in_block", + "Committee bits extracted from an imported block", + vec![0.0, 1.0, 2.0, 4.0, 8.0, 12.0, 16.0, 24.0, 32.0, 64.0] + ) + .unwrap() + }); + LEAN_HEARTBEAT_VOTES_IN_BLOCK.observe(votes as f64); +} + +/// Distinct `Tier::Heartbeat` entries packed into a block. +/// +/// The cap-pressure signal: at `MAX_ATTESTATIONS_DATA` committee votes are being +/// truncated and `Tier::Finalize` is being starved of entry slots. Note it is the +/// *entry* count that matters here, not the bit count. +pub fn observe_heartbeat_entries_in_block(entries: usize) { + static LEAN_HEARTBEAT_ENTRIES_IN_BLOCK: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_histogram!( + "lean_heartbeat_entries_in_block", + "Distinct heartbeat-tier attestation entries packed into a block", + vec![0.0, 1.0, 2.0, 3.0, 4.0, 8.0, 16.0, 32.0, 64.0] + ) + .unwrap() + }); + LEAN_HEARTBEAT_ENTRIES_IN_BLOCK.observe(entries as f64); +} + +/// How far the fast head's expanding fallback had to walk back. Greater than 1 +/// means slots were missed. +pub fn observe_fast_head_window_slots(slots: u64) { + static LEAN_FAST_HEAD_WINDOW_SLOTS: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_histogram!( + "lean_fast_head_window_slots", + "Slots the fast head's expanding vote window had to walk back", + vec![0.0, 1.0, 2.0, 3.0, 4.0, 6.0, 8.0] + ) + .unwrap() + }); + LEAN_FAST_HEAD_WINDOW_SLOTS.observe(slots as f64); +} + +/// Raw-signature aggregation cost inside the heartbeat fold, kept separate from +/// the committee session's timing. This is what tells you whether a chosen `K` +/// still fits inside interval 1. +pub fn time_heartbeat_fold() -> TimingGuard { + static LEAN_HEARTBEAT_FOLD_TIME_SECONDS: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_histogram!( + "lean_heartbeat_fold_time_seconds", + "Time spent folding raw heartbeat signatures into a type-1 aggregate", + vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0] + ) + .unwrap() + }); + TimingGuard::new(&LEAN_HEARTBEAT_FOLD_TIME_SECONDS) +} + +/// Count a fold that was skipped because a single existing type-1 already covers +/// every buffered signer — the free path. +pub fn inc_heartbeat_fold_skipped() { + static LEAN_HEARTBEAT_FOLD_SKIPPED_TOTAL: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + register_int_counter!( + "lean_heartbeat_fold_skipped_total", + "Heartbeat folds skipped because an existing type-1 already covers every signer" + ) + .unwrap() + }); + LEAN_HEARTBEAT_FOLD_SKIPPED_TOTAL.inc(); +} + pub fn set_node_info(name: &str, version: &str) { LEAN_NODE_INFO.with_label_values(&[name, version]).set(1); } diff --git a/crates/blockchain/src/spec_test_runner.rs b/crates/blockchain/src/spec_test_runner.rs index c3ab67af..f2ae58bb 100644 --- a/crates/blockchain/src/spec_test_runner.rs +++ b/crates/blockchain/src/spec_test_runner.rs @@ -98,6 +98,12 @@ pub fn rejection_reason(err: &StoreError) -> Option { | StoreError::SignatureAggregationFailed(_) | StoreError::MissingTargetState(_) | StoreError::SlotOutOfRange(_) => return None, + + // Placeholder: leanSpec names no rejection reason for the heartbeat tier + // yet, so these report as unclassified until it does. Revisit once the + // spec fixtures cover heartbeat vote validation. + StoreError::HeartbeatVoteOutOfSlot { .. } + | StoreError::NotHeartbeatCommitteeMember { .. } => return None, }; Some(reason) } diff --git a/crates/blockchain/src/store.rs b/crates/blockchain/src/store.rs index 5f2a4657..3bbe3506 100644 --- a/crates/blockchain/src/store.rs +++ b/crates/blockchain/src/store.rs @@ -1,7 +1,10 @@ use std::collections::{HashMap, HashSet}; use ethlambda_crypto::signature::{ValidatorPublicKey, ValidatorSignature}; -use ethlambda_state_transition::{is_proposer, slot_is_justifiable_after}; +use ethlambda_state_transition::{ + effective_heartbeat_committee_size, is_heartbeat_committee_member, is_proposer, + slot_is_justifiable_after, +}; use ethlambda_storage::{ForkCheckpoints, Store}; use ethlambda_types::{ ShortRoot, @@ -9,7 +12,7 @@ use ethlambda_types::{ Attestation, AttestationData, HashedAttestationData, SignedAggregatedAttestation, SignedAttestation, validator_indices, }, - block::{Block, BlockHeader, SignedBlock, SingleMessageAggregate}, + block::{Block, BlockBody, BlockHeader, SignedBlock, SingleMessageAggregate}, checkpoint::Checkpoint, primitives::{H256, HashTreeRoot as _}, state::{HISTORICAL_ROOTS_LIMIT, State}, @@ -18,13 +21,141 @@ use tracing::{info, trace, warn}; use crate::{ GOSSIP_DISPARITY_INTERVALS, INTERVALS_PER_SLOT, MAX_ATTESTATIONS_DATA, - MILLISECONDS_PER_INTERVAL, MILLISECONDS_PER_SLOT, SlotInterval, + MILLISECONDS_PER_INTERVAL, MILLISECONDS_PER_SLOT, RLMD_LOOKBACK_LIMIT, SlotInterval, block_builder::{PostBlockCheckpoints, ProposerConfig, build_block}, metrics, }; const JUSTIFICATION_LOOKBACK_SLOTS: u64 = 3; +/// Extract the heartbeat committee votes a block carries for its predecessor slot. +/// +/// A block for slot `T` carries slot-`T-1` committee bits: the view-merge payload +/// the next fast head is computed from. The gate is `data.slot == T - 1` +/// *strictly* — no window, no fallback to an earlier slot when slots were +/// skipped. That makes this an exact mirror of the packer's `Tier::Heartbeat` +/// gate, so "which slot's committee bits does this block carry" has exactly one +/// answer and never needs reconciling. Degrading on a missed slot is the fast +/// head's job, via its expanding fallback, not the packer's or this function's. +/// +/// This is a plain read of the body: no proofs, no filters, no cap. In particular +/// it must **not** reuse [`crate::reaggregate`]'s candidate selection, whose +/// filters (drop attestations whose target is already justified, skip +/// participants already covered locally, truncate to a per-block maximum) are +/// right for deciding what is worth a SNARK split and wrong for fork choice: the +/// fast head needs the `head` field of every committee vote regardless of whether +/// its target is already justified. +/// +/// Kept standalone over `(body, slot, n, committee_size)` so the restart replay +/// path can run the identical extraction over blocks read back from the database. +pub fn extract_heartbeat_votes( + body: &BlockBody, + block_slot: u64, + num_validators: u64, + committee_size: u64, +) -> Vec<(u64, AttestationData)> { + // Slot 0 has no predecessor, so a genesis-adjacent block carries nothing. + let Some(cttee_slot) = block_slot.checked_sub(1) else { + return Vec::new(); + }; + + let mut votes = Vec::new(); + for entry in body.attestations.iter() { + if entry.data.slot != cttee_slot { + continue; + } + for validator_id in validator_indices(&entry.aggregation_bits) { + if is_heartbeat_committee_member( + validator_id, + cttee_slot, + num_validators, + committee_size, + ) { + votes.push((validator_id, entry.data.clone())); + } + } + } + votes +} + +/// Rebuild the heartbeat vote set from the last [`RLMD_LOOKBACK_LIMIT`] blocks +/// on disk. +/// +/// `votes_per_slot`, `known_payloads` and the signature buffers are all +/// in-memory, so without this a restart blinds fork choice: `fast_head` finds +/// nothing, expands to the full window, and collapses to `lagging_head`, which +/// has no heartbeat evidence of its own either. Replaying the window makes a +/// restart deterministic instead of "recovers after a few slots". +/// +/// Cheap: at most `RLMD_LOOKBACK_LIMIT` blocks, each contributing at most `K'` +/// votes, and no new storage. Runs the identical extraction the live import path +/// uses, which is why [`extract_heartbeat_votes`] is factored out. +pub fn replay_heartbeat_votes(store: &Store) { + let mut root = store.head().expect("head block exists"); + let committee_size = store.heartbeat_committee_size(); + let mut replayed = 0usize; + + for _ in 0..RLMD_LOOKBACK_LIMIT { + let Some(block) = store.get_block(&root).expect("block read should succeed") else { + break; + }; + // The anchor block has no parent to walk to, and a slot-0 block carries + // no predecessor committee. + if block.slot == 0 { + break; + } + + // Registry as of the committee's own slot: the parent state is the + // closest state at or before `block.slot - 1`. Fall back to the head + // state when the parent predates the anchor (checkpoint-synced node). + let num_validators = store + .get_state(&block.parent_root) + .expect("state read should succeed") + .map_or_else( + || store.head_state().validators.len() as u64, + |state| state.validators.len() as u64, + ); + + for (validator_id, data) in + extract_heartbeat_votes(&block.body, block.slot, num_validators, committee_size) + { + store.insert_heartbeat_vote(validator_id, data); + replayed += 1; + } + + root = block.parent_root; + } + + if replayed > 0 { + info!( + replayed, + window_slots = RLMD_LOOKBACK_LIMIT, + "Replayed heartbeat votes from stored blocks" + ); + } +} + +/// Merge a block's heartbeat votes into the store's vote set. +/// +/// A union, not a replacement: `insert_heartbeat_vote` is last-write-wins per +/// `(slot, validator)`, so the block's copy overwrites a gossip copy for the same +/// validator while validators present only via gossip keep theirs. Convergence +/// comes from the proposer's set normally *dominating* — it had a full interval to +/// collect — not from discarding local votes. +fn merge_heartbeat_votes(store: &Store, body: &BlockBody, block_slot: u64, num_validators: u64) { + let votes = extract_heartbeat_votes( + body, + block_slot, + num_validators, + store.heartbeat_committee_size(), + ); + metrics::observe_heartbeat_votes_in_block(votes.len()); + for (validator_id, data) in votes { + store.insert_heartbeat_vote(validator_id, data); + metrics::inc_heartbeat_votes_received(metrics::HeartbeatVoteSource::Block); + } +} + /// Intermediate fork-choice data produced by [`update_head`], carried so the /// fork choice tree can be rendered without recomputing LMD GHOST. pub struct HeadUpdate { @@ -58,7 +189,21 @@ fn accept_new_attestations(store: &mut Store) -> HeadUpdate { update_head(store) } -/// Update the head based on the fork choice rule. +/// Update the fast head (the store head) via GHOST-Eph over the previous slot's +/// heartbeat committee votes. +/// +/// Rooted at [`update_lagging_head`]'s output rather than at the justified +/// checkpoint, so the two-tier structure is explicit: the lagging head is the +/// tree base the full validator set backs, and the fast head is the fast-moving +/// tip chosen within it. +/// +/// `min_score = 0` is easy to misread. The GHOST walk descends while the current +/// head has *any* child, picking `max_by_key(weight, root)`, so a freshly +/// proposed block with zero votes still becomes head as long as it is the only +/// child. The previous slot's committee votes do not *authorise* extension; they +/// only break ties between competing branches. That is exactly what view-merge +/// protects: without it, two nodes seeing different gossip could break the same +/// tie differently. /// /// Returns the block set, block weights, and new head computed during the /// update so callers can render the fork choice tree without recomputing it. @@ -66,18 +211,19 @@ pub fn update_head(store: &mut Store) -> HeadUpdate { let blocks = store .get_live_chain() .expect("get_live_chain should succeed"); - let attestations = store.extract_latest_known_attestations(); let old_head = store.head().expect("head block exists"); - let latest_justified_root = store - .latest_justified() - .expect("latest justified checkpoint exists") - .root; - let (new_head, weights) = ethlambda_fork_choice::compute_lmd_ghost_head( - latest_justified_root, - &blocks, - &attestations, - 0, - ); + let base = store.lagging_head().expect("lagging head exists"); + + // GHOST-Eph: the previous slot's committee votes, which the current slot's + // block just delivered. The window widens on an empty slot (see + // `heartbeat_votes_expanding_back`); after RLMD_LOOKBACK_LIMIT slots of + // nothing it is empty and the fast head collapses to `base`. + let (attestations, window_slots) = + store.heartbeat_votes_expanding_back(store.current_slot(), RLMD_LOOKBACK_LIMIT); + metrics::observe_fast_head_window_slots(window_slots); + + let (new_head, weights) = + ethlambda_fork_choice::compute_lmd_ghost_head(base, &blocks, &attestations, 0); if let Some(depth) = reorg_depth(old_head, new_head, store) { metrics::inc_fork_choice_reorgs(); metrics::observe_fork_choice_reorg_depth(depth); @@ -136,29 +282,75 @@ pub fn update_head(store: &mut Store) -> HeadUpdate { } } +/// Update the lagging head: the RLMD-window tree base the fast head sits on. +/// +/// Recency-latest-message-driven, so it reads the latest message per validator +/// over the half-open window `[S - N, S)` — excluding the current slot, +/// including `S - 1`. The fast head's slot-`S-1` votes are therefore a subset of +/// this window; that overlap is intended. +/// +/// Both vote pools are merged. `min_score = ceil(2n/3)` is denominated in the +/// full validator set, so the heartbeat pool alone could never reach it: the +/// aggregate pool is what carries full-set weight, and dropping it would pin the +/// lagging head to `latest_justified` forever. +fn update_lagging_head(store: &mut Store) { + let current_slot = store.current_slot(); + let window = current_slot.saturating_sub(RLMD_LOOKBACK_LIMIT)..current_slot; + let attestations = store.latest_message_per_validator(window); + + let num_validators = store.head_state().validators.len() as u64; + let min_target_score = (2 * num_validators).div_ceil(3); + + let blocks = store + .get_live_chain() + .expect("get_live_chain should succeed"); + let (lagging_head, _weights) = ethlambda_fork_choice::compute_lmd_ghost_head( + store + .latest_justified() + .expect("latest justified checkpoint exists") + .root, + &blocks, + &attestations, + min_target_score, + ); + store + .set_lagging_head(lagging_head) + .expect("set_lagging_head should succeed"); +} + /// Update the safe target for attestation. /// -/// Safe target is an *availability* signal, not a durable-knowledge signal: -/// only the "new" pool is considered. Migration from "new" to "known" runs at -/// interval 4, strictly after this computation at interval 3. 3sf-mini chose -/// that ordering deliberately so safe target sees only freshly received votes -/// from the current slot and ignores what was carried over from earlier slots -/// (block-included attestations, previously migrated gossip, self-attestations). -/// Counting "known" would let a node keep advancing its safe target on stale -/// evidence even when live participation has collapsed: exactly the failure -/// mode safe target is supposed to prevent. See leanSpec PR #680. +/// Safe target is an *availability* signal, not a durable-knowledge signal, and +/// under the heartbeat design it is backed by the committee rather than the full +/// validator set. Two properties differ deliberately from the other two +/// fork-choice values: +/// +/// - **Current slot only.** No RLMD window, no expanding fallback. Its whole +/// purpose is to say "this branch is backed *right now*", so evidence from +/// slot `S-1` or earlier is not merely unhelpful, it is wrong: it would hold +/// the clamp open on a branch the current committee has stopped voting for. +/// Block import writes slot-`S-1` votes into the same store, so restricting to +/// the current slot is also what keeps the block-merged copies out. +/// - **3/4 of the committee, not 2/3.** At `K' = 16` the formulas separate +/// (`ceil(2K'/3)` is 11, `ceil(3K'/4)` is 12). The clamp gates what every +/// validator is allowed to target, so it asks for a strict supermajority. +/// +/// When no branch clears the threshold, `compute_lmd_ghost_head` returns +/// `start_root`, so the safe target falls back to `latest_justified` and the +/// clamp degrades to "target the justified checkpoint" rather than doing +/// something surprising. See leanSpec PR #680. fn update_safe_target(store: &mut Store) { - let head_state = store - .get_state(&store.head().unwrap()) - .expect("head state exists"); - let num_validators = head_state.unwrap().validators.len() as u64; + let attestations = store.heartbeat_votes_at(store.current_slot()); - let min_target_score = (num_validators * 2).div_ceil(3); + let num_validators = store.head_state().validators.len() as u64; + let committee_size = + effective_heartbeat_committee_size(store.heartbeat_committee_size(), num_validators); + metrics::update_heartbeat_committee_size(committee_size); + let min_target_score = (3 * committee_size).div_ceil(4); let blocks = store .get_live_chain() .expect("get_live_chain should succeed"); - let attestations = store.extract_latest_new_attestations(); let (safe_target, _weights) = ethlambda_fork_choice::compute_lmd_ghost_head( store .latest_justified() @@ -326,7 +518,8 @@ fn validate_attestation_data(store: &Store, data: &AttestationData) -> Result<() /// Process a tick event. /// /// `store.time()` represents interval-count-since-genesis: each increment is one -/// 800ms interval. Slot and interval-within-slot are derived as: +/// [`MILLISECONDS_PER_INTERVAL`] interval. Slot and interval-within-slot are +/// derived as: /// slot = store.time() / INTERVALS_PER_SLOT /// interval = store.time() % INTERVALS_PER_SLOT pub fn on_tick(store: &mut Store, timestamp_ms: u64, has_proposal: bool) { @@ -358,10 +551,10 @@ pub fn on_tick(store: &mut Store, timestamp_ms: u64, has_proposal: bool) { let should_signal_proposal = has_proposal && is_final_tick; // NOTE: here we assume on_tick never skips intervals. - // Interval 2 (committee-signature aggregation) is no longer handled here: - // the blockchain actor orchestrates the aggregation worker directly so - // the actor's message loop stays unblocked during the expensive XMSS - // proofs. See `BlockChainServer::start_aggregation_session` in `lib.rs`. + // Committee-signature aggregation is not handled here: the blockchain + // actor orchestrates the aggregation worker directly so the actor's + // message loop stays unblocked during the expensive XMSS proofs. See + // `BlockChainServer::start_aggregation_session` in `lib.rs`. match interval { SlotInterval::BlockPublication => { // Start of slot - process attestations if proposal exists @@ -372,15 +565,15 @@ pub fn on_tick(store: &mut Store, timestamp_ms: u64, has_proposal: bool) { SlotInterval::AttestationProduction => { // Vote propagation — no action } - SlotInterval::Aggregation => { - // Aggregation is driven by the actor (off-thread); nothing to do here. - } SlotInterval::SafeTargetUpdate => { // Update safe target for validators update_safe_target(store); } SlotInterval::EndOfSlot => { - // End of slot - accept accumulated attestations and log tree + // Recompute the tree base from the RLMD window, then the fast + // head on top of it, before promoting this slot's payloads and + // logging the resulting tree. + update_lagging_head(store); let update = accept_new_attestations(store); log_fork_choice_tree(store, &update); } @@ -465,6 +658,115 @@ pub fn on_gossip_attestation( Ok(()) } +/// Process a vote received on the global heartbeat topic. +/// +/// Validation, in order: +/// +/// 1. The existing topology / time / availability checks on the data. +/// 2. XMSS verification against the validator's attestation pubkey in the state +/// at `data.slot`. +/// 3. Committee membership at `data.slot`. Without this the topic is an +/// unmetered global attestation firehose. +/// 4. `data.slot == current_slot` (within the gossip disparity margin). +/// Heartbeat votes are only useful in-slot; older ones arrive via blocks. +/// +/// On acceptance both the `AttestationData` and the raw signature are stored +/// **unconditionally** — not gated on `is_aggregator`. A proposer that is not a +/// committee aggregator still has to fold these signatures into its block, and it +/// is at most `K'` signatures per slot. +pub fn on_gossip_heartbeat_attestation( + store: &mut Store, + signed_attestation: &SignedAttestation, +) -> Result<(), StoreError> { + let validator_id = signed_attestation.validator_id; + let data = signed_attestation.data.clone(); + + validate_attestation_data(store, &data).inspect_err(|_| metrics::inc_attestations_invalid())?; + + // Heartbeat votes are in-slot only. Reject anything not for the current slot, + // allowing the same one-interval skew margin the data time check uses. + let current_slot = store.current_slot(); + let slot_start_interval = data.slot.saturating_mul(INTERVALS_PER_SLOT); + let too_old = data.slot < current_slot; + let too_new = slot_start_interval > store.time().unwrap() + GOSSIP_DISPARITY_INTERVALS; + if too_old || too_new { + metrics::inc_attestations_invalid(); + return Err(StoreError::HeartbeatVoteOutOfSlot { + attestation_slot: data.slot, + current_slot, + }); + } + + let hashed = HashedAttestationData::new(data.clone()); + let data_root = hashed.root(); + + // The state at the vote's own slot decides both the pubkey and the committee, + // so both reads agree by construction. `target.root`'s state is the closest + // available state at or before `data.slot`; only empty slots can lie between, + // so the registry is identical. + let target_state = store + .get_state(&data.target.root) + .expect("target state exists") + .ok_or(StoreError::MissingTargetState(data.target.root))?; + let num_validators = target_state.validators.len() as u64; + if validator_id >= num_validators { + return Err(StoreError::AttesterIndexOutOfRange { + validator_index: validator_id, + num_validators, + }); + } + + if !is_heartbeat_committee_member( + validator_id, + data.slot, + num_validators, + store.heartbeat_committee_size(), + ) { + metrics::inc_attestations_invalid(); + return Err(StoreError::NotHeartbeatCommitteeMember { + validator_id, + slot: data.slot, + }); + } + + let validator_pubkey = ValidatorPublicKey::from_bytes( + &target_state.validators[validator_id as usize].attestation_pubkey, + ) + .map_err(|_| StoreError::PubkeyDecodingFailed(validator_id))?; + + let slot_u32: u32 = data.slot.try_into().expect("slot exceeds u32"); + let signature = ValidatorSignature::from_bytes(&signed_attestation.signature) + .map_err(|_| StoreError::SignatureDecodingFailed)?; + let is_valid = { + let _timing = metrics::time_pq_sig_attestation_verification(); + signature.is_valid(&validator_pubkey, slot_u32, &data_root) + }; + if !is_valid { + metrics::inc_pq_sig_attestation_signatures_invalid(); + return Err(StoreError::SignatureVerificationFailed); + } + metrics::inc_pq_sig_attestation_signatures_valid(); + + store.insert_heartbeat_vote(validator_id, data.clone()); + store.insert_heartbeat_signature(data.slot, data_root, validator_id, signature); + metrics::inc_heartbeat_votes_received(metrics::HeartbeatVoteSource::Gossip); + metrics::update_heartbeat_committee_participation( + store.heartbeat_voter_count_at(data.slot) as u64 + ); + + info!( + slot = data.slot, + validator = validator_id, + target_slot = data.target.slot, + target_root = %ShortRoot(&data.target.root.0), + source_slot = data.source.slot, + source_root = %ShortRoot(&data.source.root.0), + "Heartbeat attestation processed" + ); + + Ok(()) +} + /// Process a gossiped aggregated attestation from the aggregation subnet. /// /// Aggregated attestations arrive from committee aggregators and contain a proof @@ -673,6 +975,12 @@ fn on_block_core( let block = signed_block.message.clone(); + // Registry size as of the committee's own slot. The parent state is the + // closest state at or before `block.slot - 1`, and only empty slots can lie + // between them, so the registry cannot have changed in the gap. Captured + // before `parent_state` is consumed by the state transition. + let cttee_num_validators = parent_state.validators.len() as u64; + // Execute state transition function to compute post-block state let state_transition_start = std::time::Instant::now(); let mut post_state = parent_state; @@ -713,6 +1021,12 @@ fn on_block_core( // arriving inside a block are counted by // `lean_state_transition_attestations_processed_total` instead. + // View-merge: fold the block's slot-(T-1) committee bits into the heartbeat + // vote set before fork choice reads it. After the STF (so a rejected block + // never contributes) and before `update_head` (so this slot's fast head sees + // the payload the proposer just delivered). + merge_heartbeat_votes(store, &block.body, slot, cttee_num_validators); + // Update forkchoice head based on new block and attestations update_head(store); @@ -930,6 +1244,12 @@ pub fn produce_block_with_signatures( } // Get known aggregated payloads: data_root -> (AttestationData, Vec) + // + // The heartbeat-folded aggregate needs no special handling here: the fold runs + // as an ordinary interval-2 aggregation job, so its result was inserted into + // `new_payloads` at the interval-2 boundary and promoted to `known_payloads` + // by the promote above. It reaches the builder as one candidate among many, + // and `Tier::Heartbeat` is what makes it win. let aggregated_payloads = store.known_aggregated_payloads(); let known_block_roots = store.get_block_roots().unwrap(); @@ -1081,6 +1401,17 @@ pub enum StoreError { store_time: u64, }, + #[error( + "heartbeat vote for slot {attestation_slot} is not for the current slot {current_slot}" + )] + HeartbeatVoteOutOfSlot { + attestation_slot: u64, + current_slot: u64, + }, + + #[error("validator {validator_id} is not in the heartbeat committee for slot {slot}")] + NotHeartbeatCommitteeMember { validator_id: u64, slot: u64 }, + #[error("Aggregated signature verification failed: {0}")] AggregateVerificationFailed(ethlambda_crypto::VerificationError), @@ -1312,6 +1643,141 @@ mod tests { bits } + // ============ Heartbeat Vote Extraction Tests ============ + + /// `n = 8, K = 4`, so the committee for slot 7 is `{7, 0, 1, 2}` and the + /// committee for slot 6 is `{6, 7, 0, 1}`. + const HB_VALIDATORS: u64 = 8; + const HB_COMMITTEE: u64 = 4; + + fn hb_data(slot: u64, target_marker: u8) -> AttestationData { + AttestationData { + slot, + head: Checkpoint::default(), + target: Checkpoint { + root: H256([target_marker; 32]), + slot, + }, + source: Checkpoint::default(), + } + } + + /// A body carrying one entry per `(slot, signers, target_marker)` triple. + fn hb_body(entries: &[(u64, &[usize], u8)]) -> BlockBody { + let attestations: Vec = entries + .iter() + .map(|(slot, signers, marker)| AggregatedAttestation { + aggregation_bits: make_bits(signers), + data: hb_data(*slot, *marker), + }) + .collect(); + BlockBody { + attestations: AggregatedAttestations::try_from(attestations).unwrap(), + } + } + + #[test] + fn extraction_keeps_committee_members_and_drops_outsiders() { + // Block slot 8 -> committee slot 7 -> committee {7, 0, 1, 2}. + // Signers 3, 4, 5 are outsiders. + let body = hb_body(&[(7, &[0, 1, 3, 4, 7], 0xAA)]); + let votes = extract_heartbeat_votes(&body, 8, HB_VALIDATORS, HB_COMMITTEE); + + let mut extracted: Vec = votes.iter().map(|(vid, _)| *vid).collect(); + extracted.sort_unstable(); + assert_eq!(extracted, vec![0, 1, 7], "outsider bits must be ignored"); + } + + #[test] + fn extraction_gate_is_strictly_the_previous_slot() { + // Entries for slot 6 and slot 8 must both be ignored by a slot-8 block: + // only slot 7 is the committee slot. This is the exact mirror of the + // packer's `Tier::Heartbeat` gate. + let body = hb_body(&[(6, &[6, 7, 0], 0xAA), (7, &[7, 0], 0xBB), (8, &[0], 0xCC)]); + let votes = extract_heartbeat_votes(&body, 8, HB_VALIDATORS, HB_COMMITTEE); + + assert_eq!(votes.len(), 2); + assert!( + votes.iter().all(|(_, data)| data.slot == 7), + "only the committee slot's entries are heartbeat votes" + ); + } + + #[test] + fn extraction_survives_a_skipped_slot_without_widening() { + // A block at slot 12 built after slots 8..11 were skipped still gates on + // exactly 11, carrying nothing rather than reaching back for evidence. + // Degrading on a missed slot is the fast head's job, not the packer's. + let body = hb_body(&[(7, &[7, 0, 1], 0xAA)]); + assert!( + extract_heartbeat_votes(&body, 12, HB_VALIDATORS, HB_COMMITTEE).is_empty(), + "no window and no fallback" + ); + } + + #[test] + fn extraction_ignores_genesis_adjacent_blocks() { + let body = hb_body(&[(0, &[0], 0xAA)]); + assert!(extract_heartbeat_votes(&body, 0, HB_VALIDATORS, HB_COMMITTEE).is_empty()); + } + + #[test] + fn extraction_does_not_apply_the_reaggregate_filters() { + // `reaggregate::select_candidates` drops attestations whose target is + // already justified, skips participants already covered locally, and + // truncates to a per-block maximum. Those filters are right for deciding + // what is worth a SNARK split and wrong here: the fast head needs the + // `head` field of every committee vote regardless. Extraction is a plain + // read of the body, so a target at slot 0 (trivially justified) and a + // large entry count both come through untouched. + let body = hb_body(&[(7, &[7, 0, 1, 2], 0)]); + let votes = extract_heartbeat_votes(&body, 8, HB_VALIDATORS, HB_COMMITTEE); + assert_eq!( + votes.len(), + 4, + "an already-justified target must still be extracted" + ); + + // Many distinct datas for the same committee slot: no cap is applied. + let many: Vec<(u64, &[usize], u8)> = (0..32).map(|i| (7, &[7, 0][..], i as u8)).collect(); + let body = hb_body(&many); + let votes = extract_heartbeat_votes(&body, 8, HB_VALIDATORS, HB_COMMITTEE); + assert_eq!(votes.len(), 64, "no truncation on the fork-choice path"); + } + + #[test] + fn block_votes_union_with_gossip_and_win_on_conflict() { + use ethlambda_storage::backend::InMemoryBackend; + use std::sync::Arc; + + let genesis_state = State::from_genesis(1000, vec![]); + let store = Store::from_anchor_state(Arc::new(InMemoryBackend::new()), genesis_state); + + // Gossip delivered validator 0 (conflicting data) and validator 1 (which + // the block will not mention at all). + store.insert_heartbeat_vote(0, hb_data(7, 0x11)); + store.insert_heartbeat_vote(1, hb_data(7, 0x11)); + + // The block carries validators 0 and 7 with different data. + let body = hb_body(&[(7, &[0, 7], 0x22)]); + merge_heartbeat_votes(&store, &body, 8, HB_VALIDATORS); + + let votes = store.heartbeat_votes_at(7); + assert_eq!(votes.len(), 3, "union, not replacement"); + assert_eq!( + votes[&0].target.root, + H256([0x22; 32]), + "block wins the conflict" + ); + assert_eq!( + votes[&1].target.root, + H256([0x11; 32]), + "a gossip-only validator keeps its vote, so a byzantine proposer \ + cannot blank the fast head's evidence" + ); + assert_eq!(votes[&7].target.root, H256([0x22; 32])); + } + #[test] fn on_block_rejects_duplicate_attestation_data() { use ethlambda_storage::backend::InMemoryBackend; diff --git a/crates/blockchain/state_transition/src/lib.rs b/crates/blockchain/state_transition/src/lib.rs index d53b089b..a1088868 100644 --- a/crates/blockchain/state_transition/src/lib.rs +++ b/crates/blockchain/state_transition/src/lib.rs @@ -245,6 +245,57 @@ pub fn is_proposer(validator_index: u64, slot: u64, num_validators: u64) -> bool current_proposer(slot, num_validators) == Some(validator_index) } +/// Effective heartbeat committee size: the configured size clamped to the +/// registry, `K' = min(K, n)`. +/// +/// Load-bearing rather than defensive. The committee is built by walking +/// `K` steps round-robin from the proposer, so once `K > n` each validator is +/// picked `ceil(K / n)` times and the *set* is just the whole registry. A +/// threshold denominated in the raw `K` — `ceil(3K/4)` for the safe target — +/// would then demand more votes than there are validators to cast them, and the +/// safe target would pin to `latest_justified` forever. At the default `K = 16` +/// that is not hypothetical: local devnets and the spec fixtures routinely run +/// `n <= 16`. +/// +/// Every heartbeat threshold must be denominated in this value, never in `K`. +pub fn effective_heartbeat_committee_size(committee_size: u64, num_validators: u64) -> u64 { + committee_size.min(num_validators) +} + +/// Check if a validator is part of the heartbeat committee for a given slot. +/// +/// The committee for `slot` is the proposer plus the next `K' - 1` validators, +/// round-robin: +/// +/// ```text +/// proposer(slot) = slot % n +/// committee = { (p + i) mod n : 0 <= i < K' }, p = proposer(slot) +/// ``` +/// +/// Membership is consumed both by heartbeat gossip admission and by the +/// fork-choice extraction that runs on every node over every block, so two +/// nodes disagreeing here disagree about which bits of a block are heartbeat +/// votes. `num_validators` must therefore always come from the state at the +/// vote's *own* slot, never from the head state. +pub fn is_heartbeat_committee_member( + validator_index: u64, + slot: u64, + num_validators: u64, + committee_size: u64, +) -> bool { + let Some(proposer) = current_proposer(slot, num_validators) else { + return false; + }; + if validator_index >= num_validators { + return false; + } + // Distance from the proposer walking forward round-robin. Membership is + // that distance falling inside the effective committee width, which is the + // closed form of the `(p + i) mod n` set-builder above. + let offset = (validator_index + num_validators - proposer) % num_validators; + offset < effective_heartbeat_committee_size(committee_size, num_validators) +} + /// Apply attestations and update justification/finalization /// according to the Lean Consensus 3SF-mini rules. fn process_attestations( @@ -655,6 +706,78 @@ mod tests { }; use libssz_types::SszList; + /// Collect the heartbeat committee for `slot` by asking the predicate about + /// every validator, so the tests exercise exactly what callers call. + fn committee(slot: u64, num_validators: u64, committee_size: u64) -> Vec { + (0..num_validators) + .filter(|vid| is_heartbeat_committee_member(*vid, slot, num_validators, committee_size)) + .collect() + } + + #[test] + fn committee_is_proposer_plus_next_k_minus_one() { + // n = 64, K = 16, slot 100 -> proposer 36, committee {36..=51}. + assert_eq!(committee(100, 64, 16), (36..=51).collect::>()); + // Slot 101 shifts by exactly one. + assert_eq!(committee(101, 64, 16), (37..=52).collect::>()); + } + + #[test] + fn committee_wraps_around_the_registry() { + // Proposer 60 with K = 8 wraps past the end: {60..=63} ∪ {0..=3}. + let mut expected: Vec = (60..=63).collect(); + expected.extend(0..=3); + expected.sort_unstable(); + assert_eq!(committee(60, 64, 8), expected); + } + + #[test] + fn committee_at_or_above_registry_size_is_everyone() { + // K == n and K > n both collapse to the whole registry, for every slot. + for slot in 0..20 { + assert_eq!(committee(slot, 16, 16), (0..16).collect::>()); + assert_eq!(committee(slot, 16, 64), (0..16).collect::>()); + assert_eq!(committee(slot, 8, 16), (0..8).collect::>()); + } + } + + #[test] + fn committee_handles_degenerate_registries() { + // Single validator: always the whole committee. + assert_eq!(committee(0, 1, 16), vec![0]); + assert_eq!(committee(7, 1, 16), vec![0]); + // Empty registry: nobody, and no division by zero. + assert!(!is_heartbeat_committee_member(0, 5, 0, 16)); + // Out-of-range index is never a member. + assert!(!is_heartbeat_committee_member(64, 0, 64, 16)); + } + + #[test] + fn effective_committee_size_keeps_thresholds_reachable() { + // The regression this guards: a threshold denominated in the raw K would + // demand more votes than there are validators, pinning the safe target to + // latest_justified forever. + for num_validators in [1u64, 4, 8, 16, 64] { + for configured in [1u64, 4, 16, 64, 4096] { + let effective = effective_heartbeat_committee_size(configured, num_validators); + assert!( + effective <= num_validators, + "K'={effective} exceeds n={num_validators}" + ); + let threshold = (3 * effective).div_ceil(4); + assert!( + threshold <= num_validators, + "ceil(3K'/4)={threshold} unreachable at n={num_validators}" + ); + // And it really is the size of the set the predicate yields. + assert_eq!( + committee(0, num_validators, configured).len() as u64, + effective + ); + } + } + } + fn make_validators(n: usize) -> Vec { (0..n) .map(|i| Validator { diff --git a/crates/common/types/src/constants.rs b/crates/common/types/src/constants.rs index 3066b344..debf475a 100644 --- a/crates/common/types/src/constants.rs +++ b/crates/common/types/src/constants.rs @@ -1,5 +1,28 @@ //! Protocol constants shared across crates. +/// Milliseconds per interval (1000ms ticks). +pub const MILLISECONDS_PER_INTERVAL: u64 = 1000; + +/// Number of intervals per slot (4 intervals of 1000ms = 4 seconds). +pub const INTERVALS_PER_SLOT: u64 = 4; + +/// Milliseconds in a slot (derived from interval duration and count). +/// +/// Deliberately unchanged at 4000 ms across the 5 -> 4 interval switch: the XMSS +/// epoch is the slot, so key lifetime in wall-clock terms is untouched, existing +/// `GENESIS_TIME` configs stay valid, and slot numbers still line up with other +/// clients even while the interval grid inside the slot diverges. +pub const MILLISECONDS_PER_SLOT: u64 = MILLISECONDS_PER_INTERVAL * INTERVALS_PER_SLOT; + +/// How many slots of vote memory recency-latest-message-driven (RLMD) fork +/// choice keeps: the window is the half-open range `[S - N, S)`. +/// +/// 8 slots is 32 s at a 4-second slot. It doubles as the cap on how far the fast +/// head's expanding fallback will walk back looking for evidence, and as the +/// retention bound for the heartbeat vote store — nothing ever reads further +/// back than this, so retaining more during a finality stall is unbounded waste. +pub const RLMD_LOOKBACK_LIMIT: u64 = 8; + /// Fork digest embedded in every gossipsub topic string, as lowercase hex /// without a `0x` prefix. /// @@ -8,3 +31,25 @@ /// eventually be derived from the fork version and genesis validators root. // TODO: derive dynamically once the spec defines fork identification. pub const FORK_DIGEST: &str = "12345678"; + +/// Heartbeat committee size applied when the genesis config omits +/// `HEARTBEAT_COMMITTEE_SIZE`. +/// +/// 16 rather than a smaller committee because every heartbeat threshold is a +/// *fraction* of the committee: the tolerated number of absent members is what +/// scales with the size. At 16 the safe target survives 4 absences, against 1 +/// at a committee of 4. +/// +/// Lives here, not next to `is_heartbeat_committee_member`, only because +/// `ethlambda-storage` needs it to seed its metadata key and does not depend on +/// `ethlambda-state-transition`. It is a plain `u64` and binds nothing on the +/// wire. +pub const DEFAULT_HEARTBEAT_COMMITTEE_SIZE: u64 = 16; + +/// Largest `HEARTBEAT_COMMITTEE_SIZE` accepted from a genesis config. +/// +/// Any value at or above the validator count behaves identically (the effective +/// size is clamped to the registry), so this is a typo guard rather than a +/// protocol bound. It matches `MAX_ATTESTATIONS_DATA` because that is the entry +/// budget a wide committee split competes for. +pub const MAX_HEARTBEAT_COMMITTEE_SIZE: u64 = 4096; diff --git a/crates/common/types/src/genesis.rs b/crates/common/types/src/genesis.rs index 239de125..96037886 100644 --- a/crates/common/types/src/genesis.rs +++ b/crates/common/types/src/genesis.rs @@ -1,6 +1,9 @@ use serde::Deserialize; -use crate::state::{State, Validator, ValidatorPubkeyBytes}; +use crate::{ + constants::{DEFAULT_HEARTBEAT_COMMITTEE_SIZE, MAX_HEARTBEAT_COMMITTEE_SIZE}, + state::{State, Validator, ValidatorPubkeyBytes}, +}; /// Ways a state can fail to belong to the configured genesis. /// @@ -34,10 +37,53 @@ pub struct GenesisValidatorEntry { pub struct GenesisConfig { #[serde(rename = "GENESIS_TIME")] pub genesis_time: u64, + /// Heartbeat committee size, network-wide. + /// + /// Read from the genesis config rather than a CLI flag on purpose: + /// committee membership decides which bits of an imported block are + /// heartbeat votes, so two nodes disagreeing about it is a fork-choice + /// divergence. A per-node flag makes that a one-node misconfiguration; a + /// genesis value makes it impossible within a network. + /// + /// Optional so existing configs keep loading at + /// [`DEFAULT_HEARTBEAT_COMMITTEE_SIZE`]. + #[serde( + rename = "HEARTBEAT_COMMITTEE_SIZE", + default = "default_heartbeat_committee_size", + deserialize_with = "deser_heartbeat_committee_size" + )] + pub heartbeat_committee_size: u64, #[serde(rename = "GENESIS_VALIDATORS")] pub genesis_validators: Vec, } +fn default_heartbeat_committee_size() -> u64 { + DEFAULT_HEARTBEAT_COMMITTEE_SIZE +} + +/// Reject `0` (an empty committee makes every heartbeat threshold vacuous) and +/// absurd sizes at load time, rather than letting them surface as a silently +/// stalled safe target. +fn deser_heartbeat_committee_size<'de, D>(d: D) -> Result +where + D: serde::Deserializer<'de>, +{ + use serde::de::Error; + + let size = u64::deserialize(d)?; + if size == 0 { + return Err(D::Error::custom( + "HEARTBEAT_COMMITTEE_SIZE must be at least 1", + )); + } + if size > MAX_HEARTBEAT_COMMITTEE_SIZE { + return Err(D::Error::custom(format!( + "HEARTBEAT_COMMITTEE_SIZE is {size} (maximum {MAX_HEARTBEAT_COMMITTEE_SIZE})" + ))); + } + Ok(size) +} + impl GenesisConfig { pub fn validators(&self) -> Vec { self.genesis_validators @@ -193,6 +239,68 @@ GENESIS_VALIDATORS: ); } + /// A minimal config with one validator, so the heartbeat tests can vary only + /// the `HEARTBEAT_COMMITTEE_SIZE` line. + fn config_yaml_with(heartbeat_line: &str) -> String { + format!( + r#"GENESIS_TIME: 1770407233 +{heartbeat_line} +GENESIS_VALIDATORS: + - attestation_pubkey: "{ATT_PUBKEY_A}" + proposal_pubkey: "{PROP_PUBKEY_A}" +"# + ) + } + + #[test] + fn heartbeat_committee_size_defaults_when_absent() { + let config: GenesisConfig = serde_yaml_ng::from_str(&config_yaml_with("")).unwrap(); + assert_eq!( + config.heartbeat_committee_size, + DEFAULT_HEARTBEAT_COMMITTEE_SIZE + ); + // And the pre-existing fixture, which has no such key, still loads. + let legacy: GenesisConfig = serde_yaml_ng::from_str(TEST_CONFIG_YAML).unwrap(); + assert_eq!( + legacy.heartbeat_committee_size, + DEFAULT_HEARTBEAT_COMMITTEE_SIZE + ); + } + + #[test] + fn heartbeat_committee_size_is_read_when_present() { + let config: GenesisConfig = + serde_yaml_ng::from_str(&config_yaml_with("HEARTBEAT_COMMITTEE_SIZE: 32")).unwrap(); + assert_eq!(config.heartbeat_committee_size, 32); + } + + #[test] + fn heartbeat_committee_size_rejects_zero() { + // An empty committee makes every heartbeat threshold vacuous, so this must + // fail at load rather than surface as a silently stalled safe target. + let err = serde_yaml_ng::from_str::(&config_yaml_with( + "HEARTBEAT_COMMITTEE_SIZE: 0", + )) + .expect_err("zero committee size must be rejected"); + assert!( + err.to_string().contains("at least 1"), + "unexpected error: {err}" + ); + } + + #[test] + fn heartbeat_committee_size_rejects_absurd_values() { + let too_big = MAX_HEARTBEAT_COMMITTEE_SIZE + 1; + let err = serde_yaml_ng::from_str::(&config_yaml_with(&format!( + "HEARTBEAT_COMMITTEE_SIZE: {too_big}" + ))) + .expect_err("oversized committee size must be rejected"); + assert!( + err.to_string().contains("maximum"), + "unexpected error: {err}" + ); + } + #[test] fn state_from_genesis_uses_defaults() { let validators = vec![Validator { diff --git a/crates/net/api/src/lib.rs b/crates/net/api/src/lib.rs index d6ec647d..4f6d48ce 100644 --- a/crates/net/api/src/lib.rs +++ b/crates/net/api/src/lib.rs @@ -13,6 +13,10 @@ use spawned_concurrency::protocol; pub trait BlockChainToP2P: Send + Sync { fn publish_block(&self, block: SignedBlock) -> Result<(), ActorError>; fn publish_attestation(&self, attestation: SignedAttestation) -> Result<(), ActorError>; + fn publish_heartbeat_attestation( + &self, + attestation: SignedAttestation, + ) -> Result<(), ActorError>; fn publish_aggregated_attestation( &self, attestation: SignedAggregatedAttestation, @@ -39,6 +43,13 @@ pub enum BlockSource { pub trait P2PToBlockChain: Send + Sync { fn new_block(&self, block: SignedBlock, source: BlockSource) -> Result<(), ActorError>; fn new_attestation(&self, attestation: SignedAttestation) -> Result<(), ActorError>; + /// A vote received on the global heartbeat topic. + /// + /// Separate from [`Self::new_attestation`] because heartbeat votes take a + /// different admission path: committee membership and an in-slot recency + /// check on top of the usual validation, and unconditional storage of both + /// the vote and its raw signature regardless of the aggregator role. + fn new_heartbeat_attestation(&self, attestation: SignedAttestation) -> Result<(), ActorError>; fn new_aggregated_attestation( &self, attestation: SignedAggregatedAttestation, diff --git a/crates/net/p2p/src/gossipsub/handler.rs b/crates/net/p2p/src/gossipsub/handler.rs index 95ac7081..ea0f276c 100644 --- a/crates/net/p2p/src/gossipsub/handler.rs +++ b/crates/net/p2p/src/gossipsub/handler.rs @@ -13,7 +13,7 @@ use super::{ encoding::{compress_message, decompress_message}, messages::{ AGGREGATION_TOPIC_KIND, ATTESTATION_SUBNET_TOPIC_PREFIX, BLOCK_TOPIC_KIND, - attestation_subnet_topic, + HEARTBEAT_TOPIC_KIND, attestation_subnet_topic, }, }; use crate::{P2PServer, metrics}; @@ -129,12 +129,72 @@ pub async fn handle_gossipsub_message(server: &mut P2PServer, event: Event) { .inspect_err(|err| error!(%err, "Failed to forward attestation to blockchain")); } } + Some(kind) if kind == HEARTBEAT_TOPIC_KIND => { + info!(kind = "heartbeat", peer_count, "P2P message received"); + let compressed_len = message.data.len(); + let Ok(uncompressed_data) = decompress_message(&message.data) + .inspect_err(|err| error!(%err, "Failed to decompress gossipped heartbeat")) + else { + return; + }; + metrics::observe_gossip_attestation_size(uncompressed_data.len(), compressed_len); + + let Ok(signed_attestation) = SignedAttestation::from_ssz_bytes(&uncompressed_data) + .inspect_err(|err| error!(?err, "Failed to decode gossipped heartbeat")) + else { + return; + }; + let slot = signed_attestation.data.slot; + let validator = signed_attestation.validator_id; + info!( + %slot, + validator, + head_root = %ShortRoot(&signed_attestation.data.head.root.0), + target_slot = signed_attestation.data.target.slot, + target_root = %ShortRoot(&signed_attestation.data.target.root.0), + source_slot = signed_attestation.data.source.slot, + source_root = %ShortRoot(&signed_attestation.data.source.root.0), + "Received heartbeat attestation from gossip" + ); + if let Some(ref blockchain) = server.blockchain { + let _ = blockchain + .new_heartbeat_attestation(signed_attestation) + .inspect_err(|err| error!(%err, "Failed to forward heartbeat to blockchain")); + } + } _ => { trace!("Received message on unknown topic: {}", message.topic); } } } +/// Republish a committee member's attestation to the global heartbeat topic. +/// +/// The topic is a latency shortcut, nothing more: it gets the committee's raw +/// signatures to the next proposer within one interval, without waiting for a +/// subnet aggregate to be built and propagated. This publishes the *same* +/// signature the validator already sent to its subnet, so no extra XMSS epoch is +/// consumed and no second signing operation is needed. +pub async fn publish_heartbeat_attestation(server: &mut P2PServer, attestation: SignedAttestation) { + let slot = attestation.data.slot; + let validator = attestation.validator_id; + + let ssz_bytes = attestation.to_ssz(); + let compressed = compress_message(&ssz_bytes); + metrics::observe_gossip_attestation_size(ssz_bytes.len(), compressed.len()); + + server + .swarm_handle + .publish(server.heartbeat_topic.clone(), compressed); + info!( + %slot, + validator, + target_slot = attestation.data.target.slot, + target_root = %ShortRoot(&attestation.data.target.root.0), + "Published heartbeat attestation to gossipsub" + ); +} + pub async fn publish_attestation(server: &mut P2PServer, attestation: SignedAttestation) { let slot = attestation.data.slot; let validator = attestation.validator_id; diff --git a/crates/net/p2p/src/gossipsub/messages.rs b/crates/net/p2p/src/gossipsub/messages.rs index 11664750..993a93dd 100644 --- a/crates/net/p2p/src/gossipsub/messages.rs +++ b/crates/net/p2p/src/gossipsub/messages.rs @@ -2,6 +2,10 @@ pub use ethlambda_types::constants::FORK_DIGEST; /// Topic kind for block gossip pub const BLOCK_TOPIC_KIND: &str = "block"; + +/// Topic kind for heartbeat gossip +pub const HEARTBEAT_TOPIC_KIND: &str = "heartbeat"; + /// Topic kind prefix for per-committee attestation subnets. /// /// Full topic format: `/leanconsensus/{FORK_DIGEST}/attestation_{subnet_id}/ssz_snappy` @@ -31,3 +35,10 @@ pub fn attestation_subnet_topic(subnet_id: u64) -> libp2p::gossipsub::IdentTopic "/leanconsensus/{FORK_DIGEST}/{ATTESTATION_SUBNET_TOPIC_PREFIX}_{subnet_id}/ssz_snappy" )) } + +/// Build a heartbeat gossipsub topic. +pub fn heartbeat_topic() -> libp2p::gossipsub::IdentTopic { + libp2p::gossipsub::IdentTopic::new(format!( + "/leanconsensus/{FORK_DIGEST}/{HEARTBEAT_TOPIC_KIND}/ssz_snappy" + )) +} diff --git a/crates/net/p2p/src/gossipsub/mod.rs b/crates/net/p2p/src/gossipsub/mod.rs index b50ea4fd..2ad00233 100644 --- a/crates/net/p2p/src/gossipsub/mod.rs +++ b/crates/net/p2p/src/gossipsub/mod.rs @@ -5,5 +5,6 @@ mod messages; pub use encoding::decompress_message; pub use handler::{ handle_gossipsub_message, publish_aggregated_attestation, publish_attestation, publish_block, + publish_heartbeat_attestation, }; -pub use messages::{aggregation_topic, attestation_subnet_topic, block_topic}; +pub use messages::{aggregation_topic, attestation_subnet_topic, block_topic, heartbeat_topic}; diff --git a/crates/net/p2p/src/lib.rs b/crates/net/p2p/src/lib.rs index 4726ce25..34e4068f 100644 --- a/crates/net/p2p/src/lib.rs +++ b/crates/net/p2p/src/lib.rs @@ -9,6 +9,7 @@ use ethlambda_network_api::{ InitBlockChain, P2PToBlockChainRef, block_chain_to_p2p::{ FetchBlock, PublishAggregatedAttestation, PublishAttestation, PublishBlock, + PublishHeartbeatAttestation, }, }; use ethlambda_storage::Store; @@ -37,8 +38,9 @@ use tracing::{info, trace, warn}; use crate::{ gossipsub::{ - aggregation_topic, attestation_subnet_topic, block_topic, publish_aggregated_attestation, - publish_attestation, publish_block, + aggregation_topic, attestation_subnet_topic, block_topic, heartbeat_topic, + publish_aggregated_attestation, publish_attestation, publish_block, + publish_heartbeat_attestation, }, req_resp::{ BLOCKS_BY_RANGE_PROTOCOL_V1, BLOCKS_BY_ROOT_PROTOCOL_V1, Codec, @@ -212,6 +214,7 @@ pub struct BuiltSwarm { pub(crate) attestation_topics: HashMap, pub(crate) attestation_committee_count: u64, pub(crate) block_topic: libp2p::gossipsub::IdentTopic, + pub(crate) heartbeat_topic: libp2p::gossipsub::IdentTopic, pub(crate) aggregation_topic: libp2p::gossipsub::IdentTopic, pub(crate) bootnode_addrs: HashMap, } @@ -327,6 +330,14 @@ pub fn build_swarm( .subscribe(&block_topic) .unwrap(); + // Subscribe to heartbeat topic (all nodes) + let heartbeat_topic = heartbeat_topic(); + swarm + .behaviour_mut() + .gossipsub + .subscribe(&heartbeat_topic) + .unwrap(); + // Subscribe to aggregation topic (all validators) let aggregation_topic = aggregation_topic(); swarm @@ -361,6 +372,7 @@ pub fn build_swarm( attestation_topics, attestation_committee_count: config.attestation_committee_count, block_topic, + heartbeat_topic, aggregation_topic, bootnode_addrs, }) @@ -386,6 +398,7 @@ impl P2P { attestation_topics: built.attestation_topics, attestation_committee_count: built.attestation_committee_count, block_topic: built.block_topic, + heartbeat_topic: built.heartbeat_topic, aggregation_topic: built.aggregation_topic, connected_peers: HashSet::new(), pending_root_requests: HashMap::new(), @@ -422,6 +435,7 @@ pub struct P2PServer { pub(crate) attestation_topics: HashMap, pub(crate) attestation_committee_count: u64, pub(crate) block_topic: libp2p::gossipsub::IdentTopic, + pub(crate) heartbeat_topic: libp2p::gossipsub::IdentTopic, pub(crate) aggregation_topic: libp2p::gossipsub::IdentTopic, pub(crate) connected_peers: HashSet, @@ -516,6 +530,12 @@ impl Handler for P2PServer { } } +impl Handler for P2PServer { + async fn handle(&mut self, msg: PublishHeartbeatAttestation, _ctx: &Context) { + publish_heartbeat_attestation(self, msg.attestation).await; + } +} + impl Handler for P2PServer { async fn handle(&mut self, msg: PublishAggregatedAttestation, _ctx: &Context) { publish_aggregated_attestation(self, msg.attestation).await; diff --git a/crates/net/rpc/src/spec.rs b/crates/net/rpc/src/spec.rs index 01c593aa..bf31a517 100644 --- a/crates/net/rpc/src/spec.rs +++ b/crates/net/rpc/src/spec.rs @@ -1,4 +1,4 @@ -use axum::{Router, response::IntoResponse, routing::get}; +use axum::{Router, extract::State as AxumState, response::IntoResponse, routing::get}; use ethlambda_blockchain::{INTERVALS_PER_SLOT, MILLISECONDS_PER_INTERVAL, MILLISECONDS_PER_SLOT}; use ethlambda_storage::Store; use ethlambda_types::{constants::FORK_DIGEST, state::HISTORICAL_ROOTS_LIMIT}; @@ -16,16 +16,24 @@ struct SpecResponse { ms_per_interval: u64, #[serde(rename = "HISTORICAL_ROOTS_LIMIT")] historical_roots_limit: u64, + /// The network's configured heartbeat committee size, `K`. + /// + /// Served next to the interval constants so a devnet can be checked for + /// agreement without reading configs: a mismatched `K` is a fork-choice + /// divergence that produces no error, only disagreement. + #[serde(rename = "HEARTBEAT_COMMITTEE_SIZE")] + heartbeat_committee_size: u64, #[serde(rename = "FORK_DIGEST")] fork_digest: &'static str, } -async fn get_spec() -> impl IntoResponse { +async fn get_spec(AxumState(store): AxumState) -> impl IntoResponse { json_response(SpecResponse { ms_per_slot: MILLISECONDS_PER_SLOT, intervals_per_slot: INTERVALS_PER_SLOT, ms_per_interval: MILLISECONDS_PER_INTERVAL, historical_roots_limit: HISTORICAL_ROOTS_LIMIT as u64, + heartbeat_committee_size: store.heartbeat_committee_size(), fork_digest: FORK_DIGEST, }) } @@ -46,6 +54,7 @@ mod tests { INTERVALS_PER_SLOT, MILLISECONDS_PER_INTERVAL, MILLISECONDS_PER_SLOT, }; use ethlambda_storage::{Store, backend::InMemoryBackend}; + use ethlambda_types::constants::DEFAULT_HEARTBEAT_COMMITTEE_SIZE; use ethlambda_types::state::HISTORICAL_ROOTS_LIMIT; use http_body_util::BodyExt; use std::sync::Arc; @@ -74,6 +83,10 @@ mod tests { json["HISTORICAL_ROOTS_LIMIT"], HISTORICAL_ROOTS_LIMIT as u64 ); + assert_eq!( + json["HEARTBEAT_COMMITTEE_SIZE"], + DEFAULT_HEARTBEAT_COMMITTEE_SIZE + ); assert_eq!(json["FORK_DIGEST"], FORK_DIGEST); } } diff --git a/crates/storage/src/store.rs b/crates/storage/src/store.rs index 626cbf2e..7660a93a 100644 --- a/crates/storage/src/store.rs +++ b/crates/storage/src/store.rs @@ -14,6 +14,7 @@ use ethlambda_types::{ Block, BlockBody, BlockHeader, MultiMessageAggregate, SignedBlock, SingleMessageAggregate, }, checkpoint::Checkpoint, + constants::{DEFAULT_HEARTBEAT_COMMITTEE_SIZE, INTERVALS_PER_SLOT, RLMD_LOOKBACK_LIMIT}, genesis::GenesisConfig, primitives::{H256, HashTreeRoot as _}, state::{ChainConfig, State, anchor_pair_is_consistent}, @@ -22,7 +23,7 @@ use libssz::{SszDecode, SszEncode}; use crate::state_diff::StateDiff; use thiserror::Error; -use tracing::{error, info}; +use tracing::{error, info, warn}; /// Errors returned by [`Store::get_forkchoice_store`]. #[derive(Debug, Error)] @@ -85,6 +86,20 @@ const KEY_CONFIG: &[u8] = b"config"; const KEY_HEAD: &[u8] = b"head"; /// Key for "safe_target" field of the Store. Its value has type [`H256`] and it's SSZ-encoded. const KEY_SAFE_TARGET: &[u8] = b"safe_target"; +/// Key for "lagging_head" field of the Store. Its value has type [`H256`] and it's SSZ-encoded. +/// +/// The RLMD-window head that the fast head is computed on top of. Persisted +/// alongside `safe_target` so a restart starts from a real tree base rather than +/// from the justified checkpoint. +const KEY_LAGGING_HEAD: &[u8] = b"lagging_head"; +/// Key for the network's heartbeat committee size. Its value has type [`u64`] and +/// it's SSZ-encoded. +/// +/// Seeded from the genesis config at first boot and thereafter authoritative: a +/// restart must not silently adopt a different `K` from an edited config file, +/// because committee membership decides which bits of an imported block are +/// heartbeat votes. +const KEY_HEARTBEAT_COMMITTEE_SIZE: &[u8] = b"heartbeat_committee_size"; /// Key for "latest_justified" field of the Store. Its value has type [`Checkpoint`] and it's SSZ-encoded. const KEY_LATEST_JUSTIFIED: &[u8] = b"latest_justified"; /// Key for "latest_finalized" field of the Store. Its value has type [`Checkpoint`] and it's SSZ-encoded. @@ -128,6 +143,15 @@ const NEW_PAYLOAD_CAP: usize = 64; /// Each XMSS signature is ~3KB, so worst-case memory is ~6 MB. const GOSSIP_SIGNATURE_CAP: usize = 2048; +/// Hard cap for the heartbeat signature buffer, in individual signatures. +/// +/// Sized against what the heartbeat fold can ever consume: at most one committee +/// per slot within the retention window, so `RLMD_LOOKBACK_LIMIT * +/// MAX_HEARTBEAT_COMMITTEE_SIZE` bounds it even at the largest configurable +/// committee. Slot-scoped pruning does the real work; this only bounds a +/// pathological flood of distinct `AttestationData` before pruning next runs. +const HEARTBEAT_SIGNATURE_CAP: usize = 1024; + /// An entry in the payload buffer: attestation data + set of proofs. #[derive(Clone)] struct PayloadEntry { @@ -513,6 +537,78 @@ impl GossipSignatureBuffer { } } +/// Raw heartbeat XMSS signatures awaiting the proposer's fold, keyed by slot then +/// `data_root` then validator. +/// +/// Separate from [`GossipSignatureBuffer`] on purpose: these are stored by every +/// node (not just aggregators), they are keyed by slot so pruning is a window +/// operation rather than a scan, and they are consumed by the heartbeat fold +/// session rather than by committee aggregation. +#[derive(Default)] +struct HeartbeatSignatureBuffer { + /// slot -> data_root -> validator -> signature. `BTreeMap` on the validator + /// level because XMSS aggregation requires ascending signer order. + by_slot: BTreeMap>>, + total_signatures: usize, +} + +impl HeartbeatSignatureBuffer { + /// Insert one raw heartbeat signature. Last-write-wins per + /// `(slot, data_root, validator)`. + fn insert( + &mut self, + slot: u64, + data_root: H256, + validator_id: u64, + signature: ValidatorSignature, + ) { + let is_new = self + .by_slot + .entry(slot) + .or_default() + .entry(data_root) + .or_default() + .insert(validator_id, signature) + .is_none(); + if is_new { + self.total_signatures += 1; + } + // Bound a pathological flood of distinct data by dropping whole oldest + // slots: the fold only ever reads the current slot, so the oldest slot is + // always the least useful thing to evict. + while self.total_signatures > HEARTBEAT_SIGNATURE_CAP { + let Some((&oldest, _)) = self.by_slot.iter().next() else { + break; + }; + self.remove_slot(oldest); + } + } + + fn remove_slot(&mut self, slot: u64) { + if let Some(by_root) = self.by_slot.remove(&slot) { + self.total_signatures -= by_root.values().map(BTreeMap::len).sum::(); + } + } + + /// All buffered signatures for `slot`, grouped by `data_root`. + fn at_slot(&self, slot: u64) -> HashMap> { + self.by_slot.get(&slot).cloned().unwrap_or_default() + } + + /// Drop every slot strictly below `retain_from`. Returns slots pruned. + fn prune_below(&mut self, retain_from: u64) -> usize { + let stale: Vec = self + .by_slot + .range(..retain_from) + .map(|(slot, _)| *slot) + .collect(); + for slot in &stale { + self.remove_slot(*slot); + } + stale.len() + } +} + /// Encode a LiveChain key (slot, root) to bytes. /// Layout: slot (8 bytes big-endian) || root (32 bytes) /// Big-endian ensures lexicographic ordering matches numeric ordering. @@ -565,6 +661,16 @@ pub struct Store { known_payloads: Arc>, /// In-memory gossip signatures, consumed at interval 2 aggregation. gossip_signatures: Arc>, + /// Heartbeat votes by slot, then validator. + /// + /// Written from two sources: heartbeat gossip (current slot) and block import + /// (the block's slot minus one). Read by all three fork-choice values. The + /// per-validator insert is last-write-wins, which makes block import a + /// *union* with whatever gossip already delivered rather than a replacement — + /// a lazy or byzantine proposer cannot blank the fast head's evidence. + votes_per_slot: Arc>>>, + /// Raw heartbeat XMSS signatures awaiting the proposer's fold. + heartbeat_signatures: Arc>, /// LRU memoization of states by block root, shared across `Store` clones. /// Avoids reconstructing recent states from diffs on every read. state_cache: Arc>>, @@ -655,6 +761,8 @@ impl Store { gossip_signatures: Arc::new(Mutex::new(GossipSignatureBuffer::new( GOSSIP_SIGNATURE_CAP, ))), + votes_per_slot: Arc::new(Mutex::new(BTreeMap::new())), + heartbeat_signatures: Arc::new(Mutex::new(HeartbeatSignatureBuffer::default())), state_cache: new_state_cache(), }; @@ -725,6 +833,15 @@ impl Store { (KEY_CONFIG.to_vec(), anchor_state.config.to_ssz()), (KEY_HEAD.to_vec(), anchor_block_root.to_ssz()), (KEY_SAFE_TARGET.to_vec(), anchor_block_root.to_ssz()), + // The anchor is the only block there is, so it is trivially the + // RLMD-window head as well. + (KEY_LAGGING_HEAD.to_vec(), anchor_block_root.to_ssz()), + // Seeded at the default; the node overwrites it once from the + // genesis config via `reconcile_heartbeat_committee_size`. + ( + KEY_HEARTBEAT_COMMITTEE_SIZE.to_vec(), + DEFAULT_HEARTBEAT_COMMITTEE_SIZE.to_ssz(), + ), (KEY_LATEST_JUSTIFIED.to_vec(), anchor_checkpoint.to_ssz()), (KEY_LATEST_FINALIZED.to_vec(), anchor_checkpoint.to_ssz()), ]; @@ -789,12 +906,25 @@ impl Store { gossip_signatures: Arc::new(Mutex::new(GossipSignatureBuffer::new( GOSSIP_SIGNATURE_CAP, ))), + votes_per_slot: Arc::new(Mutex::new(BTreeMap::new())), + heartbeat_signatures: Arc::new(Mutex::new(HeartbeatSignatureBuffer::default())), state_cache: new_state_cache(), }) } // ============ Metadata Helpers ============ + /// Read a metadata key that may be absent. + /// + /// Unlike [`Self::get_metadata`], a missing key is `None` rather than a + /// panic, so keys introduced after a database was written can carry a + /// default instead of forcing a resync. + fn get_optional_metadata(&self, key: &[u8]) -> Option { + let view = self.backend.begin_read().expect("read view"); + let bytes = view.get(Table::Metadata, key).expect("get")?; + Some(T::from_ssz_bytes(&bytes).expect("valid encoding")) + } + fn get_metadata(&self, key: &[u8]) -> Result { let view = self.backend.begin_read().expect("read view"); let bytes = view @@ -817,13 +947,18 @@ impl Store { /// Returns the current store time in interval counts since genesis. /// - /// Each increment represents one 800ms interval. Derive slot/interval as: + /// Each increment represents one interval. Derive slot/interval as: /// slot = time() / INTERVALS_PER_SLOT /// interval = time() % INTERVALS_PER_SLOT pub fn time(&self) -> Result { self.get_metadata(KEY_TIME) } + /// The current slot, derived from the store clock. + pub fn current_slot(&self) -> u64 { + self.time().expect("store time exists") / INTERVALS_PER_SLOT + } + /// Sets the current store time. pub fn set_time(&mut self, time: u64) -> Result<(), Error> { self.set_metadata(KEY_TIME, &time) @@ -839,6 +974,41 @@ impl Store { &self.config } + /// The network's heartbeat committee size, `K`. + /// + /// Falls back to [`DEFAULT_HEARTBEAT_COMMITTEE_SIZE`] for databases written + /// before the key existed, so an in-place upgrade keeps loading. + /// + /// Callers wanting a threshold denominator want the *effective* size + /// `min(K, n)` instead — see + /// `ethlambda_state_transition::effective_heartbeat_committee_size`. + pub fn heartbeat_committee_size(&self) -> u64 { + self.get_optional_metadata(KEY_HEARTBEAT_COMMITTEE_SIZE) + .unwrap_or(DEFAULT_HEARTBEAT_COMMITTEE_SIZE) + } + + /// Adopt the genesis config's `HEARTBEAT_COMMITTEE_SIZE` on first boot. + /// + /// The persisted value wins on every later boot: committee membership feeds + /// the fork-choice extraction that runs over every imported block, so a + /// restart must not quietly change which bits of a block count as heartbeat + /// votes. A mismatch against the config file is logged rather than applied. + pub fn reconcile_heartbeat_committee_size(&mut self, configured: u64) -> Result<(), Error> { + match self.get_optional_metadata::(KEY_HEARTBEAT_COMMITTEE_SIZE) { + Some(persisted) if persisted == configured => Ok(()), + Some(persisted) => { + warn!( + persisted, + configured, + "HEARTBEAT_COMMITTEE_SIZE in the config file differs from the persisted value; \ + keeping the persisted one. Wipe the datadir to adopt the new size." + ); + Ok(()) + } + None => self.set_metadata(KEY_HEARTBEAT_COMMITTEE_SIZE, &configured), + } + } + // ============ Head ============ /// Returns the current head block root. @@ -858,6 +1028,25 @@ impl Store { self.set_metadata(KEY_SAFE_TARGET, &safe_target) } + // ============ Lagging Head ============ + + /// Returns the lagging head: the RLMD-window head the fast head builds on. + /// + /// Falls back to the current head for databases written before the key + /// existed, which is the same value `update_lagging_head` would converge to + /// on the next interval-3 tick. + pub fn lagging_head(&self) -> Result { + match self.get_optional_metadata(KEY_LAGGING_HEAD) { + Some(root) => Ok(root), + None => self.head(), + } + } + + /// Sets the lagging head block root. + pub fn set_lagging_head(&mut self, lagging_head: H256) -> Result<(), Error> { + self.set_metadata(KEY_LAGGING_HEAD, &lagging_head) + } + // ============ Checkpoints ============ /// Returns the latest justified checkpoint. @@ -920,10 +1109,18 @@ impl Store { let pruned_payloads = self.prune_stale_aggregated_payloads(finalized.slot); - if pruned_chain > 0 || pruned_sigs > 0 || pruned_payloads > 0 { + // Window-bounded, not finalized-bounded: see `prune_heartbeat_votes`. + let current_slot = self.current_slot(); + let pruned_heartbeat = self.prune_heartbeat_votes(finalized.slot, current_slot); + + if pruned_chain > 0 || pruned_sigs > 0 || pruned_payloads > 0 || pruned_heartbeat > 0 { info!( finalized_slot = finalized.slot, - pruned_chain, pruned_sigs, pruned_payloads, "Pruned finalized data" + pruned_chain, + pruned_sigs, + pruned_payloads, + pruned_heartbeat, + "Pruned finalized data" ); } } @@ -1451,6 +1648,167 @@ impl Store { Ok(()) } + // ============ Heartbeat Votes ============ + + /// Record one heartbeat vote. + /// + /// Last-write-wins per `(data.slot, validator)`. Block import therefore + /// overwrites a gossip copy for the same validator while validators known + /// only from gossip keep theirs: `V <- V union V_block`. A conflict means the + /// validator signed two different datas for one slot, i.e. it equivocated; + /// block-wins is then a deterministic tie-break, not a judgement. + pub fn insert_heartbeat_vote(&self, validator_id: u64, data: AttestationData) { + self.votes_per_slot + .lock() + .unwrap() + .entry(data.slot) + .or_default() + .insert(validator_id, data); + } + + /// Buffer one raw heartbeat XMSS signature for the proposer's fold. + pub fn insert_heartbeat_signature( + &self, + slot: u64, + data_root: H256, + validator_id: u64, + signature: ValidatorSignature, + ) { + self.heartbeat_signatures + .lock() + .unwrap() + .insert(slot, data_root, validator_id, signature); + } + + /// Buffered heartbeat signatures for `slot`, grouped by `data_root`. + pub fn heartbeat_signatures_at( + &self, + slot: u64, + ) -> HashMap> { + self.heartbeat_signatures.lock().unwrap().at_slot(slot) + } + + /// Heartbeat votes recorded for exactly `slot`. + /// + /// The safe target reads this and nothing wider on purpose: its whole job is + /// to say "this branch is backed *right now*", so evidence from an earlier + /// slot would keep the clamp open on a branch the current committee has + /// stopped voting for. + pub fn heartbeat_votes_at(&self, slot: u64) -> HashMap { + self.votes_per_slot + .lock() + .unwrap() + .get(&slot) + .cloned() + .unwrap_or_default() + } + + /// Heartbeat votes for slot `current_slot - 1`, widening the window one slot + /// at a time while it yields nothing, up to `cap` slots back. + /// + /// This is what makes an empty slot, an offline proposer, or a silent + /// committee degrade gracefully instead of blanking the fast head: the window + /// widens until it finds evidence, and after `cap` slots of nothing it + /// returns empty so the fast head collapses to its base. + /// + /// Also returns how many slots back the walk had to go (0 when nothing was + /// found at all), for the `lean_fast_head_window_slots` histogram. + pub fn heartbeat_votes_expanding_back( + &self, + current_slot: u64, + cap: u64, + ) -> (HashMap, u64) { + let votes_per_slot = self.votes_per_slot.lock().unwrap(); + for back in 1..=cap { + let Some(slot) = current_slot.checked_sub(back) else { + break; + }; + if let Some(votes) = votes_per_slot.get(&slot).filter(|v| !v.is_empty()) { + return (votes.clone(), back); + } + } + (HashMap::new(), 0) + } + + /// Latest message per validator over the half-open slot range, merging the + /// heartbeat votes with the known (fork-choice-active) aggregate pool. + /// + /// Explicitly latest-per-validator rather than relying on ascending map + /// iteration to overwrite: a higher `data.slot` always wins, and on an equal + /// slot the heartbeat copy wins because it is the block-merged, canonical + /// one. The known-payload half is not optional — consecutive committees + /// overlap in `K' - 1` members, so the heartbeat pool alone covers only + /// `min(N + K' - 1, n)` distinct validators and can never reach a + /// `ceil(2n/3)` threshold at realistic `n`. + pub fn latest_message_per_validator( + &self, + range: std::ops::Range, + ) -> HashMap { + let mut latest: HashMap = HashMap::new(); + + // Aggregate pool first, so an equal-slot heartbeat vote overwrites it. + for (validator, data) in self.extract_latest_known_attestations() { + if range.contains(&data.slot) { + Self::keep_later_vote(&mut latest, validator, data, false); + } + } + for (_, votes) in self.votes_per_slot.lock().unwrap().range(range.clone()) { + for (validator, data) in votes { + Self::keep_later_vote(&mut latest, *validator, data.clone(), true); + } + } + latest + } + + /// Keep `candidate` for `validator` if it is later than what is held, or if + /// it ties on slot and comes from the heartbeat pool. + fn keep_later_vote( + latest: &mut HashMap, + validator: u64, + candidate: AttestationData, + from_heartbeat: bool, + ) { + match latest.get(&validator) { + Some(held) + if held.slot > candidate.slot + || (held.slot == candidate.slot && !from_heartbeat) => {} + _ => { + latest.insert(validator, candidate); + } + } + } + + /// Distinct validators holding a heartbeat vote for `slot`. + pub fn heartbeat_voter_count_at(&self, slot: u64) -> usize { + self.votes_per_slot + .lock() + .unwrap() + .get(&slot) + .map_or(0, HashMap::len) + } + + /// Drop heartbeat votes and signatures below `max(finalized_slot, + /// current_slot - RLMD_LOOKBACK_LIMIT)`. + /// + /// Retaining only down to the finalized boundary is unbounded during a + /// finality stall, and RLMD never reads further back than the window, so the + /// window is the correct retention bound. Returns the number of slots pruned + /// from the vote store. + pub fn prune_heartbeat_votes(&mut self, finalized_slot: u64, current_slot: u64) -> usize { + let retain_from = finalized_slot.max(current_slot.saturating_sub(RLMD_LOOKBACK_LIMIT)); + let pruned = { + let mut votes_per_slot = self.votes_per_slot.lock().unwrap(); + let before = votes_per_slot.len(); + votes_per_slot.retain(|slot, _| *slot >= retain_from); + before - votes_per_slot.len() + }; + self.heartbeat_signatures + .lock() + .unwrap() + .prune_below(retain_from); + pruned + } + // ============ Attestation Extraction ============ /// Extract per-validator latest attestations from known (fork-choice-active) payloads. @@ -1692,6 +2050,17 @@ impl Store { .slot } + /// Returns the slot of the current lagging head. + /// + /// A frozen lagging head under a moving fast head is the RLMD-window failure + /// signature, so this is exported as its own gauge. + pub fn lagging_head_slot(&self) -> u64 { + self.get_block_header(&self.lagging_head().expect("lagging head exists")) + .expect("lagging head exists") + .unwrap() + .slot + } + /// Returns a clone of the head state. pub fn head_state(&self) -> State { self.get_state(&self.head().expect("head block exists")) @@ -1744,6 +2113,7 @@ fn write_signed_block( mod tests { use super::*; use crate::backend::InMemoryBackend; + use ethlambda_types::constants::DEFAULT_HEARTBEAT_COMMITTEE_SIZE; use ethlambda_types::genesis::{GenesisMismatch, GenesisValidatorEntry}; /// Validator at `index` whose two pubkeys are filled with `seed`, so @@ -1761,6 +2131,7 @@ mod tests { fn genesis_config(genesis_time: u64, validators: &[Validator]) -> GenesisConfig { GenesisConfig { genesis_time, + heartbeat_committee_size: DEFAULT_HEARTBEAT_COMMITTEE_SIZE, genesis_validators: validators .iter() .map(|v| GenesisValidatorEntry { @@ -1871,6 +2242,8 @@ mod tests { gossip_signatures: Arc::new(Mutex::new(GossipSignatureBuffer::new( GOSSIP_SIGNATURE_CAP, ))), + votes_per_slot: Arc::new(Mutex::new(BTreeMap::new())), + heartbeat_signatures: Arc::new(Mutex::new(HeartbeatSignatureBuffer::default())), state_cache: new_state_cache(), } } @@ -1886,6 +2259,8 @@ mod tests { gossip_signatures: Arc::new(Mutex::new(GossipSignatureBuffer::new( GOSSIP_SIGNATURE_CAP, ))), + votes_per_slot: Arc::new(Mutex::new(BTreeMap::new())), + heartbeat_signatures: Arc::new(Mutex::new(HeartbeatSignatureBuffer::default())), state_cache: new_state_cache(), } } @@ -2220,6 +2595,203 @@ mod tests { } } + // ============ Heartbeat Vote Store Tests ============ + + /// Attestation data distinguishable by both slot and target root, so tests can + /// tell *which* vote survived a merge rather than only how many did. + fn heartbeat_data(slot: u64, target_marker: u8) -> AttestationData { + AttestationData { + slot, + head: Checkpoint::default(), + target: Checkpoint { + root: H256([target_marker; 32]), + slot, + }, + source: Checkpoint::default(), + } + } + + #[test] + fn heartbeat_votes_are_last_write_wins_per_validator() { + let store = Store::test_store(); + store.insert_heartbeat_vote(7, heartbeat_data(5, 0xAA)); + // Same (slot, validator) again: the later write wins. This is what makes + // block import a union over gossip rather than a replacement. + store.insert_heartbeat_vote(7, heartbeat_data(5, 0xBB)); + // A different validator in the same slot is untouched. + store.insert_heartbeat_vote(8, heartbeat_data(5, 0xCC)); + + let votes = store.heartbeat_votes_at(5); + assert_eq!(votes.len(), 2); + assert_eq!(votes[&7].target.root, H256([0xBB; 32])); + assert_eq!(votes[&8].target.root, H256([0xCC; 32])); + } + + #[test] + fn heartbeat_votes_at_is_exact_slot_only() { + let store = Store::test_store(); + store.insert_heartbeat_vote(1, heartbeat_data(4, 0x11)); + store.insert_heartbeat_vote(2, heartbeat_data(5, 0x22)); + + // The safe target reads this: an earlier slot's evidence must not leak in. + assert_eq!(store.heartbeat_votes_at(5).len(), 1); + assert!(store.heartbeat_votes_at(6).is_empty()); + } + + #[test] + fn expanding_window_finds_the_nearest_populated_slot() { + let store = Store::test_store(); + // Nothing at slot 9 (= S-1 for S=10) or 8; evidence sits at slot 7. + store.insert_heartbeat_vote(3, heartbeat_data(7, 0x33)); + + let (votes, back) = store.heartbeat_votes_expanding_back(10, RLMD_LOOKBACK_LIMIT); + assert_eq!(votes.len(), 1); + assert_eq!(back, 3, "should report how far it walked"); + } + + #[test] + fn expanding_window_prefers_the_previous_slot() { + let store = Store::test_store(); + store.insert_heartbeat_vote(3, heartbeat_data(7, 0x33)); + store.insert_heartbeat_vote(4, heartbeat_data(9, 0x44)); + + let (votes, back) = store.heartbeat_votes_expanding_back(10, RLMD_LOOKBACK_LIMIT); + assert_eq!(back, 1); + assert_eq!(votes.len(), 1); + assert!(votes.contains_key(&4)); + } + + #[test] + fn expanding_window_respects_the_cap_and_gives_up_empty() { + let store = Store::test_store(); + // Evidence exists, but further back than the cap allows. + store.insert_heartbeat_vote(3, heartbeat_data(1, 0x33)); + + let (votes, back) = store.heartbeat_votes_expanding_back(10, 3); + assert!( + votes.is_empty(), + "beyond the cap the fast head must collapse to its base" + ); + assert_eq!(back, 0); + + // And it does not underflow near genesis. + let (votes, back) = store.heartbeat_votes_expanding_back(0, RLMD_LOOKBACK_LIMIT); + assert!(votes.is_empty()); + assert_eq!(back, 0); + } + + #[test] + fn latest_message_per_validator_is_half_open_and_latest_wins() { + let store = Store::test_store(); + // Below the window, inside it (twice for one validator), and at the + // excluded upper bound. + store.insert_heartbeat_vote(1, heartbeat_data(1, 0x01)); + store.insert_heartbeat_vote(2, heartbeat_data(3, 0x03)); + store.insert_heartbeat_vote(2, heartbeat_data(5, 0x05)); + store.insert_heartbeat_vote(3, heartbeat_data(8, 0x08)); + + let latest = store.latest_message_per_validator(2..8); + assert!(!latest.contains_key(&1), "slot 1 is below the window start"); + assert!( + !latest.contains_key(&3), + "slot 8 is the excluded upper bound" + ); + // Validator 2 voted at both 3 and 5 inside the window; the later wins. + assert_eq!(latest[&2].slot, 5); + } + + #[test] + fn latest_message_per_validator_prefers_heartbeat_on_an_equal_slot() { + let mut store = Store::test_store(); + // Same validator, same slot, different target: once via the aggregate pool + // and once via heartbeat. The heartbeat copy is the block-merged canonical + // one, so it must win the tie. + let aggregate_vote = heartbeat_data(4, 0xEE); + let mut bits = AggregationBits::with_length(6).unwrap(); + bits.set(5, true).unwrap(); + store.insert_known_aggregated_payload( + HashedAttestationData::new(aggregate_vote), + SingleMessageAggregate::empty(bits), + ); + store.insert_heartbeat_vote(5, heartbeat_data(4, 0xFF)); + + let latest = store.latest_message_per_validator(0..8); + assert_eq!(latest[&5].target.root, H256([0xFF; 32])); + } + + #[test] + fn latest_message_per_validator_includes_the_aggregate_pool() { + let mut store = Store::test_store(); + // The known-payload half is not optional: without it a ceil(2n/3) + // threshold is unreachable from committee votes alone. + let mut bits = AggregationBits::with_length(4).unwrap(); + bits.set(0, true).unwrap(); + bits.set(3, true).unwrap(); + store.insert_known_aggregated_payload( + HashedAttestationData::new(heartbeat_data(4, 0xEE)), + SingleMessageAggregate::empty(bits), + ); + + let latest = store.latest_message_per_validator(0..8); + assert_eq!(latest.len(), 2); + assert!(latest.contains_key(&0) && latest.contains_key(&3)); + } + + #[test] + fn prune_heartbeat_votes_uses_the_window_not_just_finalization() { + let mut store = Store::test_store(); + for slot in 0..=20 { + store.insert_heartbeat_vote(1, heartbeat_data(slot, 0x01)); + } + + // Finality is stalled at 0, so a finalized-only bound would retain + // everything. The window bound must still prune. + store.prune_heartbeat_votes(0, 20); + for slot in 0..20 - RLMD_LOOKBACK_LIMIT { + assert!( + store.heartbeat_votes_at(slot).is_empty(), + "slot {slot} should be outside the retention window" + ); + } + assert!(!store.heartbeat_votes_at(20).is_empty()); + assert!( + !store + .heartbeat_votes_at(20 - RLMD_LOOKBACK_LIMIT) + .is_empty() + ); + } + + #[test] + fn prune_heartbeat_votes_keeps_everything_above_finalization() { + let mut store = Store::test_store(); + for slot in 10..=14 { + store.insert_heartbeat_vote(1, heartbeat_data(slot, 0x01)); + } + // current_slot - RLMD_LOOKBACK_LIMIT would be below 10 here, so the + // finalized bound is the tighter one and wins. + store.prune_heartbeat_votes(12, 14); + assert!(store.heartbeat_votes_at(11).is_empty()); + assert!(!store.heartbeat_votes_at(12).is_empty()); + } + + #[test] + fn heartbeat_committee_size_defaults_and_reconciles_once() { + let mut store = Store::test_store(); + // A store built without going through `init_store` has no key: the getter + // falls back rather than panicking, so pre-existing databases keep loading. + assert_eq!( + store.heartbeat_committee_size(), + DEFAULT_HEARTBEAT_COMMITTEE_SIZE + ); + + store.reconcile_heartbeat_committee_size(32).unwrap(); + assert_eq!(store.heartbeat_committee_size(), 32); + + // A later boot with a different config file must NOT change it. + store.reconcile_heartbeat_committee_size(8).unwrap(); + assert_eq!(store.heartbeat_committee_size(), 32); + } + #[test] fn payload_buffer_fifo_eviction() { let mut buf = PayloadBuffer::new(3); diff --git a/docs/metrics.md b/docs/metrics.md index 017dc9ea..21bb13e2 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -50,6 +50,7 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le | Name | Type | Usage | Sample collection event | Labels | Buckets | Supported | |--------|-------|-------|-------------------------|--------|---------|-----------| | `lean_head_slot` | Gauge | Latest slot of the lean chain | On get fork choice head | | | ✅ | +| `lean_lagging_head_slot` | Gauge | Lagging head slot: the RLMD-window tree base the fast head sits on | Each tick | | | ❌ | | `lean_current_slot` | Gauge | Current slot of the lean chain | On scrape | | | ✅(*) | | `lean_safe_target_slot` | Gauge | Safe target slot | On safe target update | | | ✅ | |`lean_fork_choice_block_processing_time_seconds`| Histogram | Time taken to process block | On fork choice process block | | 0.005, 0.01, 0.025, 0.05, 0.1, 1, 1.25, 1.5, 2, 4 | ✅ | @@ -58,13 +59,26 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le |`lean_attestation_validation_time_seconds`| Histogram | Time taken to validate attestation | On validate attestation | | 0.005, 0.01, 0.025, 0.05, 0.1, 1 | ✅ | | `lean_fork_choice_reorgs_total` | Counter | Total number of fork choice reorgs | On fork choice reorg | | | ✅ | | `lean_fork_choice_reorg_depth` | Histogram | Depth of fork choice reorgs (in blocks) | On fork choice reorg | | 1, 2, 3, 5, 7, 10, 20, 30, 50, 100 | ✅ | -| `lean_tick_interval_duration_seconds` | Histogram | Elapsed time between clock ticks in seconds | At the start of each tick interval | | 0.4, 0.6, 0.75, 0.8, 0.805, 0.81, 0.815, 0.82, 0.825, 0.85, 0.9, 1.0, 1.2, 1.6 | ✅ | +| `lean_tick_interval_duration_seconds` | Histogram | Elapsed time between clock ticks in seconds | At the start of each tick interval | | 0.5, 0.75, 0.9, 1.0, 1.005, 1.01, 1.015, 1.02, 1.025, 1.05, 1.1, 1.25, 1.5, 2.0 | ✅ | | `lean_gossip_signatures` | Gauge | Number of gossip signatures in fork-choice store | On gossip signatures update | | | ✅ | | `lean_latest_new_aggregated_payloads` | Gauge | Number of new aggregated payload items | On `latest_new_aggregated_payloads` update | | | ✅ | | `lean_latest_known_aggregated_payloads` | Gauge | Number of known aggregated payload items | On `latest_known_aggregated_payloads` update | | | ✅ | | `lean_committee_signatures_aggregation_time_seconds` | Histogram | Time taken to aggregate committee signatures | On committee signatures aggregation | | 0.05, 0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4 | ✅ | | `lean_node_sync_status` | Gauge | Node sync status | On node sync status change | status=idle,syncing,synced | | ✅ | +### Heartbeat / two-tier fork choice + +| Name | Type | Usage | Sample collection event | Labels | Buckets | Supported | +|--------|-------|-------|-------------------------|--------|---------|-----------| +| `lean_heartbeat_votes_received_total` | Counter | Heartbeat votes recorded, by where they arrived from. Read against `lean_fast_head_window_slots` this is the view-merge health signal: gossip winning means gossip beat the block | On heartbeat vote insert | source=gossip,block | | ❌ | +| `lean_heartbeat_votes_in_block` | Histogram | Committee bits extracted from an imported block. Recovers the observability a dedicated body field would have given | On block import | | 0, 1, 2, 4, 8, 12, 16, 24, 32, 64 | ❌ | +| `lean_heartbeat_entries_in_block` | Histogram | Distinct `Tier::Heartbeat` entries packed. At `MAX_ATTESTATIONS_DATA` committee votes are being truncated and `Tier::Finalize` is starved — note it is the *entry* count that matters, not the bit count | On block build | | 0, 1, 2, 3, 4, 8, 16, 32, 64 | ❌ | +| `lean_heartbeat_committee_participation` | Gauge | Distinct committee voters holding a vote for the last slot. Against `ceil(3K'/4)` this is the safe-target stall predictor | On heartbeat vote insert | | | ❌ | +| `lean_heartbeat_committee_size` | Gauge | Effective committee size `K' = min(K, n)`, so a config disagreement across a network is visible without reading configs | On safe target update | | | ❌ | +| `lean_fast_head_window_slots` | Histogram | Slots the fast head's expanding vote window had to walk back. Greater than 1 means slots were missed | On head update | | 0, 1, 2, 3, 4, 6, 8 | ❌ | +| `lean_heartbeat_fold_time_seconds` | Histogram | Raw-signature aggregation cost in the heartbeat fold, separate from the committee session. This is what tells you whether a chosen `K` still fits inside interval 1 | On heartbeat fold | | 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2 | ❌ | +| `lean_heartbeat_fold_skipped_total` | Counter | Folds skipped because a single existing type-1 already covers every buffered signer — the free path | On heartbeat fold snapshot | | | ❌ | + ## State Transition Metrics | Name | Type | Usage | Sample collection event | Labels | Buckets | Supported | diff --git a/docs/rpc.md b/docs/rpc.md index fb2802a9..04eeeb16 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -48,20 +48,27 @@ The handler emits a fixed, compact body (no whitespace): ### `GET /lean/v0/config/spec` -Protocol constants the node was built with. Keys mirror the leanSpec constant names: +Protocol constants the node runs with. Keys mirror the leanSpec constant names: ```json { "MILLISECONDS_PER_SLOT": 4000, - "INTERVALS_PER_SLOT": 5, - "MILLISECONDS_PER_INTERVAL": 800, + "INTERVALS_PER_SLOT": 4, + "MILLISECONDS_PER_INTERVAL": 1000, "HISTORICAL_ROOTS_LIMIT": 262144, + "HEARTBEAT_COMMITTEE_SIZE": 16, "FORK_DIGEST": "12345678" } ``` `FORK_DIGEST` is the 4-byte hex string (no `0x` prefix) embedded in gossipsub topic names. +`HEARTBEAT_COMMITTEE_SIZE` is the network's configured `K`, read from the persisted +store rather than from the config file on disk — the value is adopted once at first +boot and a later config edit is warned about, not applied. It is served here because +a mismatched `K` across a network is a fork-choice divergence that produces no +error, only disagreement, so this endpoint is how you check for agreement. + ### `GET /lean/v0/genesis` ```json