Skip to content

πŸ› legacy usage compound sparse unique index collapses to one document per organization, everΒ #3991

Description

@PierreBrisorgueil

What β€” BillingUsage's (organizationId, weekKey) unique+sparse index (used for meter-mode replay protection) is a compound index. MongoDB's sparse-index exclusion rule for compound indexes only skips a document when every indexed field is missing β€” since organizationId is always present, a legacy (non-meter) usage document (which never sets weekKey) still gets indexed, with weekKey treated as null. A second legacy document for the same organization β€” regardless of month β€” collides on {organizationId, weekKey: null} and is rejected as a duplicate key.

Impact β€” BillingUsageRepository.increment() catches the duplicate-key error and retries without upsert; the retry's filter ({organizationId, month}) matches nothing (the existing doc is for a different month), so it resolves to null with no error surfaced. Net effect: under meterMode: false (the default), a downstream consumer's legacy usage counters can only ever be recorded for the first month an organization is active β€” every subsequent month's increment() call silently no-ops and the write is lost.

Reproduction (against a real MongoDB, using the real repository functions):

await repo.increment(orgId, '2199-01', 'executions', 5); // creates the doc
await repo.increment(orgId, '2199-02', 'executions', 7); // returns null β€” nothing written

Confirmed: after both calls, exactly one document exists for the organization (the January one); February's counters are gone.

Discovered via β€” writing fresh-database boot tests for #3990 (a distinct, already-fixed bug: startMongoose() not awaiting index builds, and an invalid $exists:false partial filter on the other usage index). This defect is independent of both: it exists regardless of index-build timing or the partial-filter fix, and reproduces even with the index built correctly and awaited.

Suggested fix direction (not evaluated in depth β€” needs its own design pass): either (a) make the weekKey index a genuine partial index (partialFilterExpression: { weekKey: { $exists: true } } instead of sparse: true, mirroring the pattern used elsewhere in this codebase for "only index docs where field X is present") so legacy documents are excluded regardless of other fields' presence, or (b) reconsider the compound index's field order/shape. Needs a migration for already-deployed databases either way, plus a decision on whether the silent null return from increment() should instead throw/log when a duplicate-key retry matches nothing (currently indistinguishable from a legitimate replay no-op).

Out of scope for #3990 β€” filing separately per this repo's non-obvious-bug documentation convention.
Scope: validated 2026-07-28

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions