Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .test_patterns.yml
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,7 @@ tests:
# http://ci.aztec-labs.com/930ddc9ede87059f
# Pipelining race: slasher doesn't record the duplicate proposal offense before
# the test's wait timeout — same family as #23501.
- regex: "src/multi-node/slashing/duplicate_proposal.test.ts.*duplicate proposals"
- regex: "src/multi-node/slashing/equivocation_offenses.test.ts.*duplicate proposals"
error_regex: "TimeoutError: Timeout awaiting duplicate proposal offense"
owners:
- *palla
2 changes: 1 addition & 1 deletion yarn-project/end-to-end/src/multi-node/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ suffix marks files with more than one top-level `it`; CI splits each `it` into i
| `recovery/` | `MultiNodeTestContext` + wide-slot / block-production timing | The chain detects a bad/withheld/conflicting proposal and recovers: `proposal_failure_recovery.parallel` (all nodes prune and recover when a proposer fails to publish to L1), `pipeline_prune` (an uncheckpointed-blocks prune under pipelined MBPS, then recovery to a multi-block checkpoint), `equivocation_recovery` (an L1-confirmed checkpoint overrides a gossip-only equivocating proposal, the chain heals, and observers record the offense). |
| `invalid-attestations/` | `MultiNodeTestContext` (slasher on) | Invalid checkpoints are detected, invalidated on L1, and the chain progresses: `invalidate_block.parallel`, a six-validator suite injecting insufficient/fake/high-s/unrecoverable/shuffled attestations and withheld blobs. |
| `high-availability/` | `MultiNodeTestContext` + `setupHaPairs` | HA-pair sync and handoff between nodes that share validator keys: `ha_sync` (a peer that did not build a block syncs to the proposed chain tip over P2P), `ha_checkpoint_handoff` (a peer records and takes over a pipelined checkpoint when its partner proposes the previous slot). |
| `slashing/` | `MultiNodeTestContext` + `SLASHER_ENABLED_MULTI_VALIDATOR_OPTS` (`setup.ts`) | Offense detection and the sentinel that drives it. Equivocation (`duplicate_proposal`, `duplicate_attestation`), broadcast of an invalid block or checkpoint proposal (`broadcasted_invalid_block_proposal_slash`, `broadcasted_invalid_checkpoint_proposal_slash.parallel`), attesting to an invalid checkpoint (`attested_invalid_proposal.parallel`), data withholding (`data_withholding_slash`), inactivity (`inactivity_slash`, `inactivity_slash_with_consecutive_epochs`, sharing the `InactivityTest` fixture in `inactivity_setup.ts`), the slasher veto path (`slash_veto_demo`), and sentinel observability (`validators_sentinel.parallel`, `multiple_validators_sentinel.parallel`, `sentinel_status_slash.parallel`). `setup.ts` holds the offense-detection config and the shared slashing waiters (`awaitCommitteeExists`, `awaitOffenseDetected`, `advanceToEpochBeforeProposer`, `findUpcomingProposerSlot`, `awaitCommitteeKicked`, `awaitProposalExecution`). |
| `slashing/` | `MultiNodeTestContext` + `SLASHER_ENABLED_MULTI_VALIDATOR_OPTS` (`setup.ts`) | Offense detection and the sentinel that drives it. Equivocation — both duplicate-proposal and duplicate-attestation — (`equivocation_offenses`), broadcast of an invalid block or checkpoint proposal (`broadcasted_invalid_block_proposal_slash`, `broadcasted_invalid_checkpoint_proposal_slash.parallel`), attesting to an invalid checkpoint (`attested_invalid_proposal.parallel`), data withholding (`data_withholding_slash`), inactivity (`inactivity_slash`, `inactivity_slash_with_consecutive_epochs`, sharing the `InactivityTest` fixture in `inactivity_setup.ts`), the slasher veto path (`slash_veto_demo`), and sentinel observability (`validators_sentinel.parallel`, `multiple_validators_sentinel.parallel`, `sentinel_status_slash.parallel`). `setup.ts` holds the two timing profiles (`baseSlashingOpts`, `SENTINEL_TIMING`), the shared offense lookup (`findSlashOffense`), and the slashing waiters (`awaitCommitteeExists`, `awaitOffenseDetected`, `advanceToEpochBeforeProposer`, `findUpcomingProposerSlot`, `awaitCommitteeKicked`, `awaitProposalExecution`). |
| `governance/` | `MultiNodeTestContext` + `MOCK_GOSSIP_MULTI_VALIDATOR_OPTS` + `GOVERNANCE_TIMING` (`setup.ts`) | A validator committee drives an L1 governance upgrade. `upgrade_governance_proposer` (the committee signals a new governance-proposer payload, reaches quorum, and the proposal executes) and `add_rollup` (a new rollup version is registered and the nodes migrate to it, exercising L1↔L2 bridging on both rollups). |

## Helper surface
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,16 @@ import {
SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
buildMockGossipValidators,
} from '../multi_node_test_context.js';
import { awaitCommitteeExists } from './setup.js';
import { SENTINEL_TIMING, awaitCommitteeExists, findSlashOffense } from './setup.js';

const TEST_TIMEOUT = 1_000_000;

jest.setTimeout(TEST_TIMEOUT);

const NUM_VALIDATORS = 3;
const COMMITTEE_SIZE = NUM_VALIDATORS;
const ETHEREUM_SLOT_DURATION = 4;
// Long 36s L2 slots (vs SENTINEL_TIMING's 8s): the bad proposer serializes an AVM-heavy 3-block
// checkpoint, which must fit inside a single slot for honest receivers to accept and attest to it.
const AZTEC_SLOT_DURATION = 36;
const BLOCK_DURATION_MS = 8_000;
const BLOCKS_PER_CHECKPOINT = 3;
Expand All @@ -44,16 +45,6 @@ const OFFENSE_DETECTION_TIMEOUT = AZTEC_SLOT_DURATION * 3;
const INVALID_BLOCK_REMOVAL_TIMEOUT = AZTEC_SLOT_DURATION * 3;

type BlockProposedEvent = Parameters<SequencerEvents['block-proposed']>[0];
type SlashOffense = Awaited<ReturnType<AztecNodeService['getSlashOffenses']>>[number];

function findSlashOffense(offenses: SlashOffense[], validator: EthAddress, offenseType: OffenseType, slot: SlotNumber) {
return offenses.find(
offense =>
offense.validator.equals(validator) &&
offense.offenseType === offenseType &&
offense.epochOrSlot === BigInt(slot),
);
}

// Validators are keyed from `getPrivateKeyFromIndex(i + 3)` (the `buildMockGossipValidators` convention),
// so the signer for validator `index` is derived from the same key its node signs proposals with.
Expand Down Expand Up @@ -160,11 +151,9 @@ describe('multi-node/slashing/attested_invalid_proposal', () => {
beforeEach(async () => {
test = await MultiNodeTestContext.setup({
...SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
anvilSlotsInAnEpoch: 4,
listenAddress: '127.0.0.1',
aztecEpochDuration: 2,
ethereumSlotDuration: ETHEREUM_SLOT_DURATION,
aztecSlotDuration: AZTEC_SLOT_DURATION,
...SENTINEL_TIMING,
sentinelEnabled: false, // reuse only the fast base timing; this test does not use the sentinel
aztecSlotDuration: AZTEC_SLOT_DURATION, // override SENTINEL_TIMING's 8s slot with 36s (see const above)
aztecTargetCommitteeSize: COMMITTEE_SIZE,
aztecProofSubmissionEpochs: 1024,
slashInactivityConsecutiveEpochThreshold: 32,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,15 @@ import {
SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
buildMockGossipValidators,
} from '../multi_node_test_context.js';
import { advanceToEpochBeforeProposer, awaitCommitteeExists } from './setup.js';
import { SENTINEL_TIMING, advanceToEpochBeforeProposer, awaitCommitteeExists } from './setup.js';

const NUM_VALIDATORS = 4;
const COMMITTEE_SIZE = NUM_VALIDATORS;
const ETHEREUM_SLOT_DURATION = 4;
const AZTEC_SLOT_DURATION = ETHEREUM_SLOT_DURATION * 2;
const BLOCK_DURATION_MS = 2000;
const AZTEC_SLOT_DURATION = SENTINEL_TIMING.aztecSlotDuration;

const slashingUnit = BigInt(1e18);
const slashingQuorum = 3;
const slashingRoundSize = 4;
const aztecEpochDuration = 2;

/**
* Test that slashing occurs when a validator broadcasts an invalid block proposal.
Expand All @@ -48,19 +45,16 @@ describe('multi-node/slashing/broadcasted_invalid_block_proposal_slash', () => {
beforeEach(async () => {
test = await MultiNodeTestContext.setup({
...SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
anvilSlotsInAnEpoch: 4,
listenAddress: '127.0.0.1',
aztecEpochDuration,
ethereumSlotDuration: ETHEREUM_SLOT_DURATION,
aztecSlotDuration: AZTEC_SLOT_DURATION,
blockDurationMs: BLOCK_DURATION_MS,
...SENTINEL_TIMING,
sentinelEnabled: false, // reuse only the fast 8s-slot timing; this test does not use the sentinel
blockDurationMs: 2000,
aztecTargetCommitteeSize: COMMITTEE_SIZE,
inboxLag: 2,
aztecProofSubmissionEpochs: 1024, // effectively do not reorg
slashInactivityConsecutiveEpochThreshold: 32, // effectively do not slash for inactivity
minTxsPerBlock: 0, // always be building
slashingQuorum,
slashingRoundSizeInEpochs: slashingRoundSize / aztecEpochDuration,
slashingRoundSizeInEpochs: slashingRoundSize / SENTINEL_TIMING.aztecEpochDuration,
slashAmountSmall: slashingUnit,
slashAmountMedium: slashingUnit * 2n,
slashAmountLarge: slashingUnit * 3n,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type { AztecNodeService } from '@aztec/aztec-node';
import type { TestAztecNodeService } from '@aztec/aztec-node/test';
import type { EthAddress } from '@aztec/aztec.js/addresses';
import { Fr } from '@aztec/aztec.js/fields';
import { BlockNumber, EpochNumber, IndexWithinCheckpoint, SlotNumber } from '@aztec/foundation/branded-types';
import { Buffer32 } from '@aztec/foundation/buffer';
Expand All @@ -24,25 +25,26 @@ import {
SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
buildMockGossipValidators,
} from '../multi_node_test_context.js';
import { advanceToEpochBeforeProposer, awaitCommitteeExists } from './setup.js';
import {
SENTINEL_TIMING,
type SlashOffense,
advanceToEpochBeforeProposer,
awaitCommitteeExists,
findSlashOffense,
} from './setup.js';

const TEST_TIMEOUT = 1_000_000;

jest.setTimeout(TEST_TIMEOUT);

const NUM_VALIDATORS = 2;
const COMMITTEE_SIZE = NUM_VALIDATORS;
const ETHEREUM_SLOT_DURATION = 4;
const AZTEC_EPOCH_DURATION = 2;
const AZTEC_SLOT_DURATION = ETHEREUM_SLOT_DURATION * AZTEC_EPOCH_DURATION;
const BLOCK_DURATION_MS = 2000;
const AZTEC_SLOT_DURATION = SENTINEL_TIMING.aztecSlotDuration;
const SLASHING_QUORUM = 5;
const SLASHING_ROUND_SIZE = 8;
const TERMINAL_BLOCK_INDEX = IndexWithinCheckpoint(1);
const HIGHER_BLOCK_INDEX = IndexWithinCheckpoint(2);

type SlashOffense = Awaited<ReturnType<AztecNodeService['getSlashOffenses']>>[number];

// Validators are keyed from `getPrivateKeyFromIndex(i + 3)` (the `buildMockGossipValidators` convention),
// so the signer for validator `index` is derived from the same key its node signs proposals with.
function getAttesterSigner(validatorIndex: number) {
Expand All @@ -52,15 +54,10 @@ function getAttesterSigner(validatorIndex: number) {

function findBroadcastedInvalidCheckpointOffense(
offenses: SlashOffense[],
validator: string,
validator: EthAddress,
slot: SlotNumber,
): SlashOffense | undefined {
return offenses.find(
offense =>
offense.validator.toString() === validator &&
offense.offenseType === OffenseType.BROADCASTED_INVALID_CHECKPOINT_PROPOSAL &&
offense.epochOrSlot === BigInt(slot),
);
return findSlashOffense(offenses, validator, OffenseType.BROADCASTED_INVALID_CHECKPOINT_PROPOSAL, slot);
}

async function awaitBroadcastedInvalidCheckpointOffense({
Expand All @@ -69,7 +66,7 @@ async function awaitBroadcastedInvalidCheckpointOffense({
slot,
}: {
node: AztecNodeService;
validator: string;
validator: EthAddress;
slot: SlotNumber;
}) {
return await retryUntil(
Expand All @@ -88,14 +85,14 @@ async function awaitAnyBroadcastedInvalidCheckpointOffense({
validator,
}: {
nodes: AztecNodeService[];
validator: string;
validator: EthAddress;
}) {
return await retryUntil(
async () => {
const offenses = (await Promise.all(nodes.map(node => node.getSlashOffenses('all')))).flat();
const matchingOffenses = offenses.filter(
offense =>
offense.validator.toString() === validator &&
offense.validator.equals(validator) &&
offense.offenseType === OffenseType.BROADCASTED_INVALID_CHECKPOINT_PROPOSAL,
);
return matchingOffenses.length > 0 ? matchingOffenses : undefined;
Expand All @@ -112,7 +109,7 @@ async function expectNoBroadcastedInvalidCheckpointOffense({
slot,
}: {
node: AztecNodeService;
validator: string;
validator: EthAddress;
slot: SlotNumber;
}) {
// The watcher polls every second with this test's slot timing; wait long enough
Expand Down Expand Up @@ -237,18 +234,15 @@ describe('multi-node/slashing/broadcasted_invalid_checkpoint_proposal_slash', ()
beforeEach(async () => {
test = await MultiNodeTestContext.setup({
...SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
anvilSlotsInAnEpoch: 4,
listenAddress: '127.0.0.1',
aztecEpochDuration: AZTEC_EPOCH_DURATION,
ethereumSlotDuration: ETHEREUM_SLOT_DURATION,
aztecSlotDuration: AZTEC_SLOT_DURATION,
blockDurationMs: BLOCK_DURATION_MS,
...SENTINEL_TIMING,
sentinelEnabled: false, // reuse only the fast 8s-slot timing; this test does not use the sentinel
blockDurationMs: 2000,
aztecTargetCommitteeSize: COMMITTEE_SIZE,
aztecProofSubmissionEpochs: 1024,
minTxsPerBlock: 0,
inboxLag: 2,
slashingQuorum: SLASHING_QUORUM,
slashingRoundSizeInEpochs: SLASHING_ROUND_SIZE / AZTEC_EPOCH_DURATION,
slashingRoundSizeInEpochs: SLASHING_ROUND_SIZE / SENTINEL_TIMING.aztecEpochDuration,
slashAmountSmall: slashingUnit,
slashAmountMedium: slashingUnit * 2n,
slashAmountLarge: slashingUnit * 3n,
Expand Down Expand Up @@ -292,7 +286,7 @@ describe('multi-node/slashing/broadcasted_invalid_checkpoint_proposal_slash', ()
expect(currentSlot).toBeGreaterThan(2);

const signer = getAttesterSigner(0);
const validator = test.addressAt(0).toString();
const validator = test.addressAt(0);
const signatureContext: CoordinationSignatureContext = {
chainId: test.context.config.l1ChainId,
rollupAddress: test.context.deployL1ContractsValues.l1ContractAddresses.rollupAddress,
Expand Down Expand Up @@ -327,11 +321,11 @@ describe('multi-node/slashing/broadcasted_invalid_checkpoint_proposal_slash', ()
checkpointCount: 1,
});
expect(firstProposals.blockProposals.map(proposal => proposal.getSender()?.toString())).toEqual([
validator,
validator,
validator,
validator.toString(),
validator.toString(),
validator.toString(),
]);
expect(firstProposals.checkpointProposals[0].getSender()?.toString()).toEqual(validator);
expect(firstProposals.checkpointProposals[0].getSender()?.toString()).toEqual(validator.toString());

const firstOffense = await awaitBroadcastedInvalidCheckpointOffense({
node,
Expand Down Expand Up @@ -387,7 +381,7 @@ describe('multi-node/slashing/broadcasted_invalid_checkpoint_proposal_slash', ()

const offenses = await awaitAnyBroadcastedInvalidCheckpointOffense({
nodes: honestNodes,
validator: invalidProposer.toString(),
validator: invalidProposer,
});

test.logger.warn(`Collected broadcasted invalid checkpoint proposal offenses`, { offenses });
Expand Down Expand Up @@ -424,15 +418,15 @@ describe('multi-node/slashing/broadcasted_invalid_checkpoint_proposal_slash', ()
checkpointCount: 1,
});
expect(validProposals.blockProposals.map(proposal => proposal.getSender()?.toString())).toEqual([
validator,
validator,
validator.toString(),
validator.toString(),
]);
const terminalProposal = validProposals.blockProposals.find(
proposal => proposal.indexWithinCheckpoint === TERMINAL_BLOCK_INDEX,
);
expect(terminalProposal?.archive.toString()).toEqual(lateHigherBlockProposals.terminalBlock.archive.toString());
expect(terminalProposal?.getSender()?.toString()).toEqual(validator);
expect(validProposals.checkpointProposals[0].getSender()?.toString()).toEqual(validator);
expect(terminalProposal?.getSender()?.toString()).toEqual(validator.toString());
expect(validProposals.checkpointProposals[0].getSender()?.toString()).toEqual(validator.toString());
await expectNoBroadcastedInvalidCheckpointOffense({ node, validator, slot: targetSlot });

await node.getP2P().broadcastProposal(lateHigherBlockProposals.higherBlock);
Expand All @@ -444,9 +438,9 @@ describe('multi-node/slashing/broadcasted_invalid_checkpoint_proposal_slash', ()
checkpointCount: 1,
});
expect(invalidProposals.blockProposals.map(proposal => proposal.getSender()?.toString())).toEqual([
validator,
validator,
validator,
validator.toString(),
validator.toString(),
validator.toString(),
]);

const offense = await awaitBroadcastedInvalidCheckpointOffense({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,22 @@ import {
SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
buildMockGossipValidators,
} from '../multi_node_test_context.js';
import { awaitCommitteeExists, awaitOffenseDetected, getMinedSlot, submitTxsThroughNode } from './setup.js';
import {
SENTINEL_TIMING,
awaitCommitteeExists,
awaitOffenseDetected,
getMinedSlot,
submitTxsThroughNode,
} from './setup.js';

const TEST_TIMEOUT = 1_000_000;
jest.setTimeout(TEST_TIMEOUT);

const NUM_VALIDATORS = 4;
const COMMITTEE_SIZE = NUM_VALIDATORS;
const ETHEREUM_SLOT_DURATION = 4;
const AZTEC_SLOT_DURATION = ETHEREUM_SLOT_DURATION * 3;
// Longer 12s L2 slots (3 eth-slots) than SENTINEL_TIMING's default 8s, giving the data-withholding
// tolerance window room to settle before the watchers probe their mempools.
const AZTEC_SLOT_DURATION = SENTINEL_TIMING.ethereumSlotDuration * 3;
const TOLERANCE_SLOTS = 3;

/**
Expand Down Expand Up @@ -62,17 +69,15 @@ describe('multi-node/slashing/data_withholding_slash', () => {
// (~23% probability). Extending slashOffenseExpirationRounds gives us several rounds to
// hit quorum before the offense expires.
const slashingRoundSize = 4;
const aztecEpochDuration = 2;
const aztecEpochDuration = SENTINEL_TIMING.aztecEpochDuration;
const slashingAmount = slashingUnit * 3n;

beforeEach(async () => {
test = await MultiNodeTestContext.setup({
...SLASHER_ENABLED_MULTI_VALIDATOR_OPTS,
anvilSlotsInAnEpoch: 4,
listenAddress: '127.0.0.1',
aztecEpochDuration,
ethereumSlotDuration: ETHEREUM_SLOT_DURATION,
aztecSlotDuration: AZTEC_SLOT_DURATION,
...SENTINEL_TIMING,
sentinelEnabled: false, // reuse only the fast base timing; this test does not use the sentinel
aztecSlotDuration: AZTEC_SLOT_DURATION, // override SENTINEL_TIMING's 8s slot with 12s (see const above)
aztecTargetCommitteeSize: COMMITTEE_SIZE,
// Long proof submission window so the legacy L1-prune path is irrelevant.
aztecProofSubmissionEpochs: 1024,
Expand Down
Loading
Loading