Skip to content

CI: test/firestore.test.tsx flakes intermittently on both Node 22 and Node 24 #776

Description

@tyler-reitz

test/firestore.test.tsx fails intermittently in CI and passes on re-run with no code change. It has now fired five times in three days, every time on a PR that cannot plausibly cause it, including a lockfile-only dependency bump and a README-only docs change.

Every occurrence costs a full CI cycle and a manual re-run, and it trains reviewers to ignore red runs.

⚠️ Two corrections, 2026-08-05

This issue previously said the flake was Node 24 only and that the RESOURCE_EXHAUSTED error was ruled out. Both were wrong, and the earlier investigation comment is superseded on those two points.

  1. It is not Node-24-specific. On docs: replace the concurrent mode note with a Suspense section #778, a README-only diff, Test Node.js 22 failed twice consecutively while Test Node.js 24 passed in the same runs. The earlier "Node 22 passes every time" claim rested on a local sample of 6 runs, which was underpowered rather than clean.
  2. RESOURCE_EXHAUSTED is not ruled out. It is absent from every local reproduction, which is all the evidence supported. In CI it is present, including on the docs: replace the concurrent mode note with a Suspense section #778 failures. This raises the possibility that the local repro and the CI failure are not the same bug, which matters because the fix below was measured against the local one.

Signature

AssertionError: expected 'loading' to deeply equal 'success'
Expected: "success"
Received: "loading"

In CI, alongside it on the same run:

@firebase/firestore: Firestore (11.10.0): GrpcConnection RPC 'Listen' stream error.
Code: 8 Message: 8 RESOURCE_EXHAUSTED: Received message larger than max (795107700 vs 4194304)

A ~795 MB (and in an earlier run ~3.5 GB) claimed frame against a 4 MB cap is a gRPC framing desync: the reader is interpreting payload bytes as a length header. That is a @grpc/grpc-js-layer failure, not an application one.

Locally the failure looks different: waitFor hitting its 1000ms default, at 1058ms and 1142ms, plus one 120s hang, with no gRPC error at all.

Occurrences

Date PR What it touched Which Node
2026-08-05 #778 README.md only 22 (twice in a row; 24 passed)
2026-08-04 #774 package-lock.json only (postcss bump) 24
2026-08-03 #770 src/ import specifiers only 24
2026-08-03 (earlier run) no source changes 24

⚠️ The #778 pair is the first time it did not clear on a single re-run, and the first time Node 22 failed. Prior occurrences all passed on the first re-run.

The two tests that fail

Both are in test/firestore.test.tsx, and the same two every time:

  • useFirestoreDocData > returns undefined if document does not exist (~line 100)
  • useFirestoreDocOnce > works when the document does not exist, and does not update when it is created (~line 174)

They are the two tests that read a document that does not exist. That is the shared property, and it is more specific than "these tests are slow". A non-existent document cannot be answered from the local cache, so the hook stays in loading until the Listen stream confirms absence. That round trip is what exceeds waitFor's 1000ms budget, so these two fail first whenever the stream is slow or broken.

Suspected cause

The current suspect is @grpc/grpc-js 1.9.16, which @firebase/firestore pins via ~1.9.0. A framing desync is exactly the class of bug that lives at that layer, which is what the CI-side RESOURCE_EXHAUSTED points at.

Measured locally, on Node 24:

Arm Failures
Baseline (@grpc/grpc-js 1.9.16) 6 / 44 (13.6%)
overrides forcing 1.14.4 0 / 38

This is suggestive, not proven, and the caveats have grown:

  1. The pre-registered 30-runs-per-arm comparison came back p = 0.12, which is not significant. Pooling all batches gives p = 0.02, but that pooling was chosen after seeing the results.
  2. The measurement was against the local failure mode, which has no gRPC error. If the CI failure is a different bug, these numbers do not speak to it.
  3. The Node 22 result that made Node 24 look special has since been contradicted in CI.

Two constraints on any fix:

  1. Upgrading firebase does not help. Every published @firebase/firestore through 4.17.0 still pins ~1.9.0. An overrides entry is the only lever in this repo.
  2. An override makes CI test a @grpc/grpc-js that consumers do not get. If 1.9.16 is genuinely unwell, the bug still reaches anyone running reactfire server-side, and the real fix belongs upstream in firebase-js-sdk. An override here buys a quiet CI, not a fixed product.

Suggested scope

Apply the overrides bump as an uncontrolled fix, and say so plainly rather than claiming it is verified.

⚠️ Merging it means CI never runs 1.9.16 again, so no further baseline accumulates. It is a before-and-after with no control, not an experiment. The honest reading afterwards is:

  • flake stops → consistent with the theory, but not proof
  • flake continues → the theory is wrong, and that is genuinely informative

If it continues, the 120s hang is the next thread. Note that raising the waitFor timeout would not have saved that run, so "just bump the timeout" is not a sufficient fallback.

Note for reviewers

Until this is fixed: a red Test Node.js 22 or Test Node.js 24 on a PR touching no source is probably this. Re-run before investigating, and say so in the PR thread rather than leaving a red run unexplained. Do not conclude anything from a single run in either direction, since one pass does not clear a branch and one failure does not condemn it. As of #778, assume it may take more than one re-run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions