feat(genesis-writer): add offline chain history population tool

[rickyrombo]

Summary

Adds cmd/genesis-writer, a one-time migration tool that populates a new Core chain with the full historical Audius state by writing synthetic CometBFT blocks directly to PostgreSQL and blockstore.db — no running network or consensus needed.

The Audius protocol is migrating from a Python-based Discovery Provider architecture to a Go-based consensus chain (CometBFT/Core). ~6 years of on-chain state (users, tracks, playlists, social graph, plays, comments, developer apps, etc.) currently lives in the DP's PostgreSQL database. Without this tool, either the chain starts empty and all historical data is lost, or every historical transaction must be replayed through live consensus — infeasible for millions of transactions.

The genesis writer solves this by reading every current, non-deleted entity from a source DP database, wrapping each in a ManageEntityLegacyMigration proto signed with the genesis migration keypair, packing them into structurally valid CometBFT blocks (with proper headers, commits, chain linkage, and signatures), and writing them to core_blocks/core_transactions/core_app_state. After writing, it primes CometBFT's state.db and blockstore.db so a single bootstrap node can start from the written height and immediately propose the next live block. Other nodes can then state-sync from it.

Architecture

Data flow:

DP Postgres (source) → genesis-writer → Core Postgres + blockstore.db (destination)
                           ↓
                    ManageEntityLegacyMigration transactions
                    wrapped in real CometBFT blocks

Key components:

• main.go — CLI with write subcommand, flag handling, managed postgres auto-start
• writer.go — Core engine: async block writer pipeline, block construction, signing, COPY-based bulk inserts, step-based resume
• cmt_state.go — CometBFT state.db/blockstore.db initialization, genesis.json patching
• batch.go — Generic concurrent batch processor (processBatched[T]) with NumCPU worker pool
• postgres.go — Optional self-managed local postgres instance for destination
• entities_*.go — One file per domain (users, tracks, playlists, social, plays, comments, developer apps, emails, tips, dashboard wallets)

Entity processing phases (dependency order):

1. Identity — users, associated wallets, dashboard wallets
2. Content — tracks, track downloads, playlists
3. Social — follows, saves, reposts, shares, subscriptions, muted users
4. Apps — developer apps, grants
5. Comments — comments, comment reactions
6. Emails — encrypted emails, email access grants
7. Activity — plays, tip reactions

Design Decisions & Tradeoffs

New proto type (ManageEntityLegacyMigration) vs reusing ManageEntityLegacy

Chose a separate proto message (structurally identical fields) so indexers get an unambiguous signal that a transaction came from genesis migration. This allows different validation rules: skip wallet-ownership checks, verify against the migration authority key instead of the Signer field. A flag on ManageEntityLegacy would be fragile and potentially spoofable in live transactions.

Direct DB writes vs consensus replay

Writing directly to postgres and blockstore is orders of magnitude faster than feeding transactions through actual consensus. The blocks are syntactically valid CometBFT blocks with proper chain linkage (prevBlockID, commits, app hashes), so a node can boot from this state. The tradeoff is that these blocks were never executed through ABCI — the app hashes are synthetic (SHA256(concat of tx bytes)) rather than ABCI-derived. This is acceptable because no node will ever re-execute genesis-range blocks.

Signer override semantics

All transactions are signed with a single migration key, then the Signer field is overwritten with the entity's real wallet address. The signature will NOT recover to the Signer value. Indexers must verify authority by recovering from the signature and checking against the genesis migration authority — Signer is an identity hint only. This is documented in the code and README.

No ABCI side-effects for migration transactions

finalizeManageEntityMigration in abci.go is a pass-through — it doesn't populate sound_recordings or management_keys like finalizeManageEntity does for live Track creates. The ETL must handle migration transactions end-to-end. This is correct because genesis data doesn't carry live CIDs.

Block time is synthetic

Block time starts at genesisTime and increments by 1 second per block, unrelated to when entities were actually created. Entity timestamps are preserved in transaction metadata for the ETL to use.

Concurrent entity processing

processBatched fans out to NumCPU goroutines for signing/marshaling, serialized by blockMu for block assembly. Transaction ordering within an entity type is non-deterministic, which is acceptable for genesis data.

Play data coverage gap

The DP plays table is pruned to ~400 days. aggregate_plays has lifetime totals but aggregate_monthly_plays overcounts by ~2x due to country-dimension duplication. The current PR writes only raw plays that exist. Historical play count reconciliation is a follow-up PR.

Rollout Strategy

PR Sequence

This PR is the first of three:

1. This PR — Genesis writer tool + ManageEntityLegacyMigration proto + ABCI pass-through handler
2. Follow-up PR — Play count reconciliation step (PlayCount/Reconcile entity type) to cover the ~400-day play data gap using aggregate_plays as ground truth
3. Follow-up PR — ETL support for ManageEntityMigration transaction type, so the indexer can consume genesis blocks and populate domain tables

This ordering is safe: the genesis writer writes blocks to the chain DB, and nothing reads ManageEntityMigration transactions until the ETL PR lands. The writer can be run against production data independently to validate output before the ETL handler is ready.

Network Migration

The genesis writer is one piece of a larger network cutover from the existing Core chain to a new chain bootstrapped with full historical state. The migration follows the same dual-write pattern used in previous network transitions (POA → Nethermind, Solana → Core), adapted for this chain-to-chain migration. No write-freeze or downtime is required.

Phase 1: Snapshot + Bootstrap

• Take a pg_dump of the production Discovery Provider at a known block height — the "cutoff block." The existing network continues operating normally; no downtime required.
• Run genesis-writer against the snapshot. This produces a fully-populated Core chain postgres, blockstore.db, state.db, and a patched genesis.json with genesis_migration_address and genesis_migration_end_height.
• Embed the output genesis.json into the source tree (pkg/core/config/genesis/prod.json) and build a new release binary.
• Boot the bootstrap node with the genesis-writer output as its data directory. It picks up at height N and begins proposing block N+1 through live CometBFT consensus as the sole initial validator.

Phase 2: Dual-Write + Catch-Up

• Deploy a relay change so it writes transactions to both the old chain and the new chain simultaneously. This ensures no writes are lost during the transition window.
• Concurrently, process all blocks from the cutoff block to the current block on the new chain — closing the gap between the snapshot point and "now."
• Once caught up, the new chain has complete state: genesis blocks (historical) + caught-up blocks (gap) + live dual-writes (ongoing).

Phase 3: Infrastructure Cutover

• Update rpc.audius.engineering and grpc.audius.engineering nodes to the new chain ID and genesis.json. These nodes state-sync from the bootstrap node to join the new chain.
• Once the infrastructure nodes are on the new chain, all consumers that read from these endpoints (ETL instances, external API clients, other nodes) automatically get the new chain's data.

Phase 4: Validator Migration

• Remaining validators update to the new binary, state-sync from the bootstrap node, and join the new chain's validator set.
• Once the validator set is fully migrated, decommission the old chain.

Key Invariants

• No downtime during genesis write — the existing network continues operating normally while the genesis writer runs offline
• No write-freeze during cutover — the dual-write relay ensures transactions land on both chains during the transition window
• Migration key is height-bounded — genesis_migration_end_height in genesis.json ensures the migration authority cannot mint new entities after the genesis range
• State sync eliminates replay — new nodes don't need the full genesis history; they state-sync to a recent height and catch up from there
• Idempotent resume — if the genesis writer crashes mid-run, --resume picks up from the last completed entity-type step

Test Plan

• Integration test (TestGenesisWriter) covering write → index → verify → consensus → state-sync lifecycle
• Docker Compose stack with source DP postgres, target Core postgres, eth-ganache, nginx ingress, two openaudio nodes
• Seed data covers all entity types with nullable field edge cases (9 tracks, 6 users, 4 playlists, 8 follows, 5 saves, 5 reposts, 8 plays)
• Verifies block linkage, transaction hashes, entity counts match source snapshot
• Production dry run against real DP snapshot
• ETL indexing verification against DP snapshot (blocked on ETL PR)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new cmd/genesis-writer tool to bootstrap a Core chain from an existing Discovery Provider Postgres DB by emitting synthetic CometBFT blocks directly into Core’s DB tables and writing CometBFT state/blockstore data so a node can start at the migrated height.

Changes:

• Introduces ManageEntityLegacyMigration as a new SignedTransaction variant and wires it into Core’s ABCI finalize path.
• Adds cmd/genesis-writer (writer, CometBFT state priming, managed local Postgres option) plus an end-to-end Docker Compose integration test + seed data.
• Updates dependencies to support the new CLI tool and Postgres/migrations usage.

Reviewed changes

Copilot reviewed 26 out of 28 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
proto/core/v1/types.proto	Adds ManageEntityLegacyMigration message + SignedTransaction oneof case.
pkg/core/server/manage_entity.go	Adds finalize handler for migration manage-entity txs.
pkg/core/server/abci.go	Improves snapshot offer handling and routes migration txs through finalizeTransaction.
go.mod	Adds direct deps for genesis-writer (cli, pq) and related indirects.
go.sum	Updates module checksums after dependency changes.
cmd/genesis-writer/main.go	CLI entrypoint, flag parsing, key resolution, managed-postgres startup.
cmd/genesis-writer/writer.go	Core migration writer: reads DP entities, builds/signs txs, builds blocks, writes to Core DB + blockstore.
cmd/genesis-writer/batch.go	Generic batched/concurrent entity processing helpers.
cmd/genesis-writer/cmt_state.go	Loads genesis + validator key, opens blockstore, bootstraps CometBFT state.db, writes updated genesis.json.
cmd/genesis-writer/postgres.go	Local managed Postgres cluster lifecycle for offline runs.
cmd/genesis-writer/entities_user.go	User + wallet-related entity extraction/serialization.
cmd/genesis-writer/entities_track.go	Track + track-download extraction/serialization.
cmd/genesis-writer/entities_playlist.go	Playlist extraction/serialization.
cmd/genesis-writer/entities_social.go	Social actions extraction/serialization (follows/saves/reposts/etc).
cmd/genesis-writer/entities_play.go	Play event extraction into TrackPlays txs.
cmd/genesis-writer/entities_developer_app.go	Developer app + grant extraction/serialization.
cmd/genesis-writer/entities_dashboard_wallet.go	Dashboard wallet user extraction/serialization.
cmd/genesis-writer/entities_comment.go	Comment + comment-reaction extraction/serialization.
cmd/genesis-writer/entities_email.go	Encrypted email + email access extraction/serialization.
cmd/genesis-writer/entities_tip.go	Tip reaction extraction/serialization.
cmd/genesis-writer/integration_test.go	Docker-based integration test validating round-trip + consensus advancement + state sync.
cmd/genesis-writer/docker-compose.yml	Integration-test stack (source DP DB, core DBs, ganache, ingress, nodes).
cmd/genesis-writer/README.md	Tool documentation, usage, and integration test instructions.
cmd/genesis-writer/Makefile	Convenience target to run integration test flow.
cmd/genesis-writer/testdata/source_init.sh	Initializes/creates the seeded source DB in Docker.
cmd/genesis-writer/testdata/seed.sql	Comprehensive DP seed dataset for integration tests.
cmd/genesis-writer/testdata/dp_seed.sql	Minimal DP seed to satisfy DP indexer assumptions.

Comments suppressed due to low confidence (1)

pkg/core/server/abci.go:597

• In the "new snapshot offered" branch, acceptedSnapshotHeight/Hash are cleared but execution continues into the hash-mismatch check. Since acceptedSnapshotHash is now nil, this will always reject the newly offered snapshot (even though we intended to accept it). Restructure this logic so that when height differs you either (a) immediately treat it as a fresh offer (skip the hash check) or (b) update acceptedSnapshotHeight/Hash to the new snapshot before validating further state.

	// If we've already accepted a snapshot, check if CometBFT is re-offering the
	// same one (resume) or a different one (previous snapshot failed verification).
	if s.acceptedSnapshotHeight != 0 {
		if req.Snapshot.Height != s.acceptedSnapshotHeight {
			// CometBFT is offering a different snapshot, which means the previously
			// accepted one failed (e.g. consensus params verification error). Clear
			// the old state so we can accept the new snapshot.
			s.logger.Info("clearing previous snapshot state: CometBFT offered a new snapshot",
				zap.Uint64("previous_height", s.acceptedSnapshotHeight),
				zap.Uint64("new_height", req.Snapshot.Height))
			s.acceptedSnapshotHeight = 0
			s.acceptedSnapshotHash = nil
		}
		// Check hash matches too
		if !bytes.Equal(req.Snapshot.Hash, s.acceptedSnapshotHash) {
			s.logger.Info("rejecting snapshot: hash mismatch",
				zap.Uint64("height", req.Snapshot.Height),
				zap.String("offered_hash", hex.EncodeToString(req.Snapshot.Hash)),
				zap.String("accepted_hash", hex.EncodeToString(s.acceptedSnapshotHash)))
			return &abcitypes.OfferSnapshotResponse{
				Result: abcitypes.OFFER_SNAPSHOT_RESULT_REJECT,
			}, nil

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

pkg/core/server/abci.go:598

• In the "accepted snapshot already set" branch, when a different snapshot height is offered you reset acceptedSnapshotHeight/Hash to allow accepting a new snapshot, but the function then continues into the hash-mismatch check and will reject because acceptedSnapshotHash is now nil. Restructure this logic so that when the offered height differs you clear state and then fall through to the "first snapshot" validation/accept path (skipping the hash check for the previous snapshot).

	if s.acceptedSnapshotHeight != 0 {
		if req.Snapshot.Height != s.acceptedSnapshotHeight {
			// CometBFT is offering a different snapshot, which means the previously
			// accepted one failed (e.g. consensus params verification error). Clear
			// the old state so we can accept the new snapshot.
			s.logger.Info("clearing previous snapshot state: CometBFT offered a new snapshot",
				zap.Uint64("previous_height", s.acceptedSnapshotHeight),
				zap.Uint64("new_height", req.Snapshot.Height))
			s.acceptedSnapshotHeight = 0
			s.acceptedSnapshotHash = nil
		}
		// Check hash matches too
		if !bytes.Equal(req.Snapshot.Hash, s.acceptedSnapshotHash) {
			s.logger.Info("rejecting snapshot: hash mismatch",
				zap.Uint64("height", req.Snapshot.Height),
				zap.String("offered_hash", hex.EncodeToString(req.Snapshot.Hash)),
				zap.String("accepted_hash", hex.EncodeToString(s.acceptedSnapshotHash)))
			return &abcitypes.OfferSnapshotResponse{
				Result: abcitypes.OFFER_SNAPSHOT_RESULT_REJECT,
			}, nil
		}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

pkg/core/server/abci.go:598

• If a different snapshot height is offered, the code clears acceptedSnapshotHeight/Hash but then immediately compares the offered hash against the now-nil acceptedSnapshotHash and will always reject. After clearing state, it should fall through to the “first snapshot” accept path (or reinitialize acceptedSnapshotHash) rather than performing the hash mismatch check on a cleared value.

	// If we've already accepted a snapshot, check if CometBFT is re-offering the
	// same one (resume) or a different one (previous snapshot failed verification).
	if s.acceptedSnapshotHeight != 0 {
		if req.Snapshot.Height != s.acceptedSnapshotHeight {
			// CometBFT is offering a different snapshot, which means the previously
			// accepted one failed (e.g. consensus params verification error). Clear
			// the old state so we can accept the new snapshot.
			s.logger.Info("clearing previous snapshot state: CometBFT offered a new snapshot",
				zap.Uint64("previous_height", s.acceptedSnapshotHeight),
				zap.Uint64("new_height", req.Snapshot.Height))
			s.acceptedSnapshotHeight = 0
			s.acceptedSnapshotHash = nil
		}
		// Check hash matches too
		if !bytes.Equal(req.Snapshot.Hash, s.acceptedSnapshotHash) {
			s.logger.Info("rejecting snapshot: hash mismatch",
				zap.Uint64("height", req.Snapshot.Height),
				zap.String("offered_hash", hex.EncodeToString(req.Snapshot.Hash)),
				zap.String("accepted_hash", hex.EncodeToString(s.acceptedSnapshotHash)))
			return &abcitypes.OfferSnapshotResponse{
				Result: abcitypes.OFFER_SNAPSHOT_RESULT_REJECT,
			}, nil
		}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 28 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

pkg/core/server/abci.go:598

• In the height-mismatch case you clear acceptedSnapshotHeight/acceptedSnapshotHash, but then still run the hash-mismatch check against a nil acceptedSnapshotHash, which will reject the newly offered snapshot. After clearing, this branch should fall through to the “First snapshot, validate and accept it” path (e.g., return ACCEPT after re-validating, or restructure with an else/goto so the hash check only runs when the snapshot height matches).

	if s.acceptedSnapshotHeight != 0 {
		if req.Snapshot.Height != s.acceptedSnapshotHeight {
			// CometBFT is offering a different snapshot, which means the previously
			// accepted one failed (e.g. consensus params verification error). Clear
			// the old state so we can accept the new snapshot.
			s.logger.Info("clearing previous snapshot state: CometBFT offered a new snapshot",
				zap.Uint64("previous_height", s.acceptedSnapshotHeight),
				zap.Uint64("new_height", req.Snapshot.Height))
			s.acceptedSnapshotHeight = 0
			s.acceptedSnapshotHash = nil
		}
		// Check hash matches too
		if !bytes.Equal(req.Snapshot.Hash, s.acceptedSnapshotHash) {
			s.logger.Info("rejecting snapshot: hash mismatch",
				zap.Uint64("height", req.Snapshot.Height),
				zap.String("offered_hash", hex.EncodeToString(req.Snapshot.Hash)),
				zap.String("accepted_hash", hex.EncodeToString(s.acceptedSnapshotHash)))
			return &abcitypes.OfferSnapshotResponse{
				Result: abcitypes.OFFER_SNAPSHOT_RESULT_REJECT,
			}, nil
		}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.