fix(mongoose): await model index builds at startup + repair the never-created billing month index

MIGRATIONS.md

@@ -4,6 +4,42 @@ Breaking changes and upgrade notes for downstream projects.
 
 ---
 
+## Boot awaits index builds; billing usage month-index partial-filter fix (2026-07-27)
+
+Fixes a silent index-creation failure: the legacy `(organizationId, month)` unique index on `billingusages` declared `partialFilterExpression: { weekKey: { $exists: false } }`, which MongoDB does not support (only `$eq`, `$exists: true`, `$gt`, `$gte`, `$lt`, `$lte`, `$type`, and top-level `$and` are allowed inside a partial filter). Mongoose autoIndex reported the creation failure on the model's unlistened `'index'` event, so the index **never existed on any deployed database** — the uniqueness guard ran on application code alone (a racy upsert). While fixing it, boot itself was hardened: it no longer treats `mongoose.connect()` resolving as "ready" (autoIndex builds run in the background).
+
+### What changed (this repo)
+
+- **`lib/services/mongoose.js`** — new `awaitIndexBuilds()`, called by `lib/app.js#startMongoose()` right after `connect()` and BEFORE `migrations.run()`. It awaits every registered model's `Model#init()` (mongoose already triggers this once on model compile; this just awaits the in-flight promise) and now SURFACES a rejection instead of it being swallowed on the unlistened `'index'` event — **this applies to every module**, not just billing: any schema that declares an unsupported/invalid index will now fail loudly at boot (or time out — see the config knob below) instead of silently never building.
+- **New config knob `db.awaitIndexBuilds`** (`config/defaults/development.config.js`, inherited by all envs) — bounds the wait so a big-collection index build can't stall readiness stack-wide on a rolling deploy: default `{ timeoutMs: 60000 }`. On timeout, boot **continues** in a degraded (pre-fix) state — the build keeps going in the background and a loud warning names the still-building model(s); an eventual build failure is still logged after the fact. Set to `false` to skip the wait entirely (restores the pre-#3990 fire-and-forget behavior).
+- **`modules/billing/models/billing.usage.model.mongoose.js`** — the index now filters on a new `legacyPeriod: Boolean` discriminator (`partialFilterExpression: { legacyPeriod: { $exists: true } }`) instead of the unsupported `weekKey` negative check. `legacyPeriod` is set only by the legacy (non-meter) write path (`BillingUsageRepository.increment`'s `$setOnInsert`) — meter-mode documents never carry it.
+- **New migration `modules/billing/migrations/20260727120000-fix-usage-month-index-partial-filter.js`** — the authoritative index creator on already-deployed databases. Ordering is deliberately boot-ordering-safe (boot now awaits index builds BEFORE migrations run, so the partial index above may already be LIVE AND EMPTY by the time this migration executes): **(a)** duplicate pre-check FIRST via a plain query (`weekKey: { $exists: false }`, not an index filter — zero writes), abort loud on any pre-existing duplicate `(organizationId, month)` pair; then, only if the index isn't already the exact target shape with nothing left to backfill (fast-path skip, the steady-state case): **(b)** drop the index if present (boot-built-empty or divergent); **(c)** backfill `legacyPeriod: true` onto legacy documents; **(d)** recreate the index — a duplicate-key error here (a still-serving old instance racing a write into the (b)-(d) window on a rolling deploy) is caught and re-thrown as the same actionable abort as (a), never a bare driver error. Idempotent on re-run.
+
+### Pre-deploy duplicate-data audit (downstreams using legacy/non-meter billing usage)
+
+Run this against your production `billingusages` collection **before** deploying this change. Any result means the migration will abort boot until you remediate (delete/merge the offending rows) — better to catch it ahead of time:
+
+```js
+db.billingusages.aggregate([
+  { $match: { weekKey: { $exists: false } } },
+  { $group: { _id: { organizationId: '$organizationId', month: '$month' }, count: { $sum: 1 }, ids: { $push: '$_id' } } },
+  { $match: { count: { $gt: 1 } } },
+]);
+```
+
+Downstreams running exclusively in meter mode (every `billingusages` document has `weekKey` set) will always get an empty result — this only applies to legacy (non-meter) usage tracking.
+
+### Action required for downstream projects (`/update-stack`)
+
+1. All changes are devkit-owned stack files → arrive via `/update-stack` (`--theirs`).
+2. **Run the duplicate-data audit above against prod before deploying.** If it returns any group, resolve the duplicates first — otherwise the migration aborts boot on next deploy (loud error naming the offending doc ids, zero writes performed).
+3. No action needed on the `db.awaitIndexBuilds` knob — default (`{ timeoutMs: 60000 }`) is safe for normal collection sizes. Only override it (in `config/defaults/{project}.config.js`) if you have an unusually large collection with a slow index build and want a longer/shorter timeout, or `false` to opt back into fire-and-forget autoIndex.
+4. Because index-build failures now surface loudly stack-wide (not just for billing), watch the first post-deploy boot log for any `Index builds still in flight` warning or a boot failure — it means some model's schema declares an index MongoDB rejects, previously silent.
+5. Migrations run at boot before `listen()`; the index swap + `legacyPeriod` backfill land automatically once the duplicate-data audit passes.
+6. **Rolling deploys only:** the very first successful run of this migration on a given database briefly drops the index while backfilling (old, still-serving instances writing into that window can trip a duplicate-key abort — self-healing, retried on next boot). For a strict no-window guarantee, run this specific deploy during a maintenance window or scale to a single instance first. Every later boot (including every other instance in the same rolling deploy once the database has converged) skips the window entirely.
+
+---
+
 ## Config: `docs.excludeModules` — doc-only module exclusion (2026-06-29)
 
 New opt-in `config.docs.excludeModules` (default `[]` → **no behavior change**). It drops a module's `doc/*.yml` (OpenAPI) + `doc/guides/*.md` (guide tree) from the public spec (`/api/spec.json`) and guide tree (`/api/public/docs`), **independent of module runtime activation** — so it works even for **core** modules (`core`/`auth`/`users`/`home`), which `filterByActivation` never filters.

config/defaults/development.config.js

@@ -42,6 +42,15 @@ const config = {
   db: {
     uri: 'mongodb://127.0.0.1:27017/NodeDev',
     debug: true,
+    // Bounds lib/services/mongoose.js#awaitIndexBuilds (#3990 follow-up): boot
+    // awaits every model's index build up to timeoutMs before continuing, so a
+    // unique-index idempotency guard can't be raced by an early write. On
+    // timeout, boot continues in a degraded (pre-#3990) state and logs a loud
+    // warning instead of hanging forever — set to `false` to disable the wait
+    // entirely (fire-and-forget autoIndex, pre-#3990 behavior).
+    awaitIndexBuilds: {
+      timeoutMs: 60000,
+    },
     options: {
       user: '',
       pass: '',

lib/app.js

@@ -17,6 +17,11 @@ const startMongoose = async () => {
   try {
     await mongooseService.loadModels();
     const connection = await mongooseService.connect();
+    // connect() resolving only means the socket is up — autoIndex builds still
+    // run in the background. Wait for every model's indexes to finish before
+    // this function (and therefore bootstrap/start) resolves, so nothing can
+    // write through a unique-index idempotency guard before it exists (#3990).
+    await mongooseService.awaitIndexBuilds();
     return connection;
   } catch (e) {
     throw new Error(e);

lib/services/mongoose.js

@@ -20,6 +20,12 @@ import logger from './logger.js';
  */
 const resolveDebug = (cfg = config) => Boolean(cfg?.db?.debug) && configHelper.isDevEnv();
 
+/**
+ * Default bound (ms) on {@link awaitIndexBuilds} when `config.db.awaitIndexBuilds`
+ * does not specify its own `timeoutMs` (#3990 follow-up).
+ */
+const DEFAULT_AWAIT_INDEX_BUILDS_TIMEOUT_MS = 60000;
+
 /**
  * Load all mongoose related models
  */
@@ -62,6 +68,95 @@ const connect = async () => {
   }
 };
 
+/**
+ * @desc Await index readiness for every currently registered model.
+ *  `mongoose.connect()` resolving does not mean indexes exist yet — autoIndex
+ *  builds run in the background, so callers must explicitly wait before
+ *  treating the app as ready. `Model#init()` resolves once that model's index
+ *  builds finish; mongoose already calls it once automatically when the model
+ *  is compiled (cached on `model.$init`), so this simply awaits builds that
+ *  are already in flight rather than triggering a second build.
+ *  It also SURFACES index-creation errors (e.g. an unsupported
+ *  `partialFilterExpression` operator) that a fire-and-forget autoIndex would
+ *  otherwise silently swallow on the model's unlistened `'index'` event —
+ *  callers should expect this call to reject when a schema declares an
+ *  invalid index (#3990).
+ *  Respects the effective `autoIndex` option (schema > connection > global):
+ *  when it resolves falsy, `Model#init()` still resolves — it just skips the
+ *  index build rather than forcing one.
+ *
+ *  Bounded by `config.db.awaitIndexBuilds` (#3990 follow-up — an unbounded
+ *  wait here stalls readiness stack-wide on a rolling deploy whenever a
+ *  big collection's index build takes a while):
+ *    - `false` skips the wait entirely — index builds still happen via
+ *      autoIndex, just fire-and-forget again (pre-#3990 behaviour).
+ *    - any other value (default: `{}`) waits up to `timeoutMs` (default
+ *      {@link DEFAULT_AWAIT_INDEX_BUILDS_TIMEOUT_MS}). On timeout, boot
+ *      CONTINUES — a loud warning names the model(s) still building, and the
+ *      in-flight build(s) keep going in the background; an eventual failure
+ *      (e.g. an invalid index declaration) is still logged, just after boot
+ *      already proceeded past this call.
+ *  A genuine rejection that happens BEFORE the timeout (the common case — an
+ *  invalid index declaration fails fast) still propagates and rejects this
+ *  call, exactly as before.
+ * @param {object} [cfg=config] - application configuration object
+ * @returns {Promise<void>} Resolves once every registered model's index builds
+ *  finish, the configured timeout elapses, or immediately when disabled.
+ */
+const awaitIndexBuilds = async (cfg = config) => {
+  const setting = cfg?.db?.awaitIndexBuilds;
+  if (setting === false) return;
+
+  // Number(...) (not Number.isFinite directly on the raw value) so a numeric
+  // STRING survives — a `db.awaitIndexBuilds.timeoutMs` override supplied via a
+  // Layer-4 DEVKIT_NODE_* env var always arrives as a string (config/index.js
+  // only coerces the literal 'true'/'false', nothing numeric); reject
+  // non-positive values so a malformed override falls back to the default
+  // instead of racing straight to a 0ms/negative timeout.
+  const configuredTimeoutMs = setting && typeof setting === 'object' ? Number(setting.timeoutMs) : NaN;
+  const timeoutMs = Number.isFinite(configuredTimeoutMs) && configuredTimeoutMs > 0 ? configuredTimeoutMs : DEFAULT_AWAIT_INDEX_BUILDS_TIMEOUT_MS;
+
+  const modelNames = mongoose.modelNames();
+  if (modelNames.length === 0) return;
+
+  const pending = new Set(modelNames);
+  const builds = Promise.all(
+    modelNames.map((name) =>
+      mongoose.model(name)
+        .init()
+        .then((result) => {
+          pending.delete(name);
+          return result;
+        }),
+    ),
+  );
+
+  let timer;
+  const timeoutPromise = new Promise((resolve) => {
+    timer = setTimeout(() => resolve('timeout'), timeoutMs);
+    timer.unref?.();
+  });
+
+  try {
+    const winner = await Promise.race([builds, timeoutPromise]);
+    if (winner === 'timeout') {
+      logger.warn(
+        chalk.red(
+          `Index builds still in flight after ${timeoutMs}ms — continuing boot in a DEGRADED state (writes may race a not-yet-built index) for model(s): ${[...pending].join(', ')}`,
+        ),
+      );
+      // Let the build(s) keep going in the background; surface an eventual
+      // failure loudly instead of silently swallowing it, even though boot
+      // already continued past this call.
+      builds.catch((err) => {
+        logger.error(chalk.red('Index build failed after boot continued past the awaitIndexBuilds timeout:'), err);
+      });
+    }
+  } finally {
+    clearTimeout(timer);
+  }
+};
+
 /**
  * Disconnect from the MongoDB server
  */
@@ -73,6 +168,7 @@ const disconnect = async () => {
 export default {
   loadModels,
   connect,
+  awaitIndexBuilds,
   disconnect,
   resolveDebug,
 };