Refactor dashboard PubSub consumers to Stream.fromQueue pipelines#95
Open
danielbdyer wants to merge 1 commit into
Open
Refactor dashboard PubSub consumers to Stream.fromQueue pipelines#95danielbdyer wants to merge 1 commit into
danielbdyer wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Queue.takeloops withStream.fromQueuepipelines to centralize consumer behaviour and reduce imperative loop code.PubSubas the canonical event source while making downstream consumers (SAB writer, WS broadcaster, journal writer) composable via streams.Description
lib/infrastructure/dashboard/pubsub-stream-consumer.tsexposingsubscribePubSubStreamConsumerwith options foreventKinds, batching (groupedWithin),onEvent/onBatchsinks, andonErrorhandling.Queue.takeintake loops inlib/infrastructure/dashboard/pipeline-event-bus.tswith the new consumer for the SharedArrayBuffer + string channel writer and for the WS broadcaster.lib/infrastructure/dashboard/journal-writer.tsto consume events via the shared stream consumer using batchedgroupedWithinsemantics and to perform batched enrich/append/index updates while preserving the original spaced flush cadence.tests/dashboard-stream-consumer.laws.spec.tsthat validate no event loss and FIFO ordering under burst load and that batching preserves spaced flush semantics.Testing
npm run test -- tests/dashboard-stream-consumer.laws.spec.ts tests/pubsub-backpressure.laws.spec.ts tests/flywheel-server-integration.laws.spec.tsand all tests passed (22 passed).dashboard-stream-consumertests exercised burst-load ordering andgroupedWithin-based batching behavior and succeeded.npm run typechecksurfaced unrelated missing generated module declarations (../generated/tesseract-knowledge/../lib/generated/tesseract-knowledge) and failed; this is pre-existing and not caused by these changes.Codex Task