refactor(shadow-indexer): stop persisting canonical blocks - #4624
Open
jowparks wants to merge 5 commits into
Open
refactor(shadow-indexer): stop persisting canonical blocks#4624jowparks wants to merge 5 commits into
jowparks wants to merge 5 commits into
Conversation
…g canonical blocks The shadow indexer persisted a row for every committed block, so canonical rows dominated a table whose only consumer reads reorged-out blocks. Stop writing them, and re-key `shadow_blocks` from `PRIMARY KEY(number, hash)` to `PRIMARY KEY(number)`. Every remaining row is by definition a reorged-out shadow block, which makes `reorged_out` and the `hash` cursor tie-breaker dead weight; both are removed. Migration 0004 creates the new table and swaps it in, renaming the old one to `shadow_blocks_legacy` rather than dropping it so the change stays recoverable. The copy lists `created_at`/`updated_at` explicitly to stop `DEFAULT now()` from restamping retained rows and re-emitting stale stats. Same-height reorgs now collapse under `ON CONFLICT (number)`, so the reader's `blocks_inspected_total` becomes poll-timing dependent. `ShadowIndexerMetrics::reorged_blocks_total` counts them at write time to preserve an exact figure. The shadow-metrics readiness probe is widened to fail `/readyz` loudly on a stale schema. Migration 0004 only drops columns, so the columns the reader projects all exist pre-migration and a projection alone cannot tell the two schemas apart. Deploy shadow-indexer before shadow-metrics: the indexer applies the migration at startup. Co-authored-by: OpenCode <opencode-noreply@coinbase.com> Co-Authored-By: Claude <noreply@anthropic.com>
…nd empty-table cursor boot The end-to-end test asserted the old contract, waiting for canonical rows. It now proves their absence across the whole poll window, guarded by a liveness check that the writer applied its migrations, since an emptiness assertion alone would also pass if the extension never started. Two paths had no coverage at all. Migrating populated legacy data was untested because every existing test provisions a fresh schema from empty; the new test seeds a 0003-era table and asserts the surviving rows, including that timestamps are preserved bit-for-bit. `max_cursor` returning `None` was nearly unreachable while canonical rows existed and is now a normal first boot, so the bootstrap test covers genesis boot, rows arriving after it, and backlog skipping. Reader fixtures move to distinct block numbers: under the number-only key, rows sharing a number silently collapse and a test would assert against a smaller set than it intends. Co-authored-by: OpenCode <opencode-noreply@coinbase.com> Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
🟡 Heimdall Review Status
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Keying shadow_blocks by number alone was never required to stop persisting canonical blocks, and it forced three breaking schema changes: dropping reorged_out, narrowing the primary key, and dropping shadow_metrics_cursor.last_hash. Each one breaks an old binary against the new schema, which is why the change needed a strict indexer-before-metrics deploy order and a readiness check to refuse booting against the wrong half of the roll. Reverting the schema leaves the actual improvement intact: the ExEx no longer writes a row per committed block, it just marks every row it does write as reorged out. The reader's existing reorged_out filter keeps hiding canonical rows written by earlier builds, so old and new binaries interoperate in both directions and rollback is an image revert. Retaining the composite key also removes the same-height collapse, so blocks_inspected_total is exact again rather than a poll-timing- dependent lower bound. Co-authored-by: OpenCode <opencode-noreply@coinbase.com> Co-Authored-By: Claude <noreply@anthropic.com>
jowparks
changed the title
Metric emission belongs to shadow-metrics; the indexer had no metrics dependency before this branch and should not gain one for a single counter. The counter only existed to compensate for same-height reorgs collapsing under the number-only key, which is no longer in the branch. Co-authored-by: OpenCode <opencode-noreply@coinbase.com> Co-Authored-By: Claude <noreply@anthropic.com>
jowparks
changed the title
jowparks
marked this pull request as ready for review
August 21, 2026 23:26
The reader bootstrap paths it covered are unchanged by this branch, so the coverage belongs with a change to the reader rather than here. Co-authored-by: OpenCode <opencode-noreply@coinbase.com> Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Review SummaryNo issues found. Reviewed: The refactor cleanly removes canonical-block persistence from the shadow indexer ExEx. Key observations:
|
Contributor
Contributor
|
Caution This PR may regress performance. 2 benchmark(s) slower by more than 10% beyond the noise band: Benchmark results (advisory)Median time on the PR head versus the base branch, measured on the same host. Wall-clock, so a change is only flagged when it clears ±10% and the confidence intervals do not overlap. Only benchmarks past the ±10% threshold (plus new or dropped ones) are listed. This check never blocks a merge.
47 benchmark(s) within ±10% omitted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ExEx wrote a row for every committed block, but the shadow metrics reader only ever consumes reorged-out and reverted ones. It now persists just those.
No migration: rows are still written with
reorged_outset, so the reader's existing filter keeps hiding the canonical rows earlier builds left behind.shadow_blocksis empty until the first reorg, so the reader bootstraps at the genesis cursor instead of the table tip.PRIMARY KEY(number, hash)and the(updated_at, number, hash)cursor index stay as they are. Re-keying onnumberalone would collapse repeated reorgs at the same height into one row and undercount them.