feat(webapp): resolve which shard an environment mints run roots into (#4755)

## Summary

Adds the shard-selection stage of run-id minting.
`resolveMintShard(env)` returns which run-ops database an environment
mints its new run roots into: the active shard list, then a fleet-wide
override, then a per-environment or per-organization pin, then a
rendezvous hash of the environment id.

That half is inert. Nothing calls `resolveMintShard`, no deployment has
any of the new flags set, and an empty active list returns the current
answer without reading anything.

**The other half is not inert, and it is where review effort belongs.**
To stamp a grace window this needs a read-then-write under a lock, so it
rewrites the global feature-flag write path that `runOpsMintKind`
already depends on in production. See below.

## Placement

Resolution reads the active list from a global flag, applies the grace
window, and then picks:

- a fleet-wide override if one is set, which is how a cutover completes
without visiting each organization. `new` holds the whole fleet on the
current id format.
- otherwise a per-environment or per-organization pin. `new` holds one
organization back while the rest move, which is how a canary works.
- otherwise a rendezvous hash, so adding a shard moves only about
1/(N+1) of environments and removing one moves only its own.

Two hash details are load-bearing. Scores are 64-bit `sha256(envId \0
key)`, because a 32-bit score collides at our environment count and an
undetected tie would resolve by iteration order. The parsed key list is
sorted, because otherwise two deployments listing the same shards in a
different CSV order would place environments differently.

A pin or override naming a shard that has left the active list falls
through to the hash and reports once. Honouring it would leak the drain
the active list exists to perform, and throwing would fail triggers
whenever a pinned shard drains.

## Why the active list is a flag and not an environment variable

A deploy rolls for hours, so two pods hold two different environment
values at the same time. A list held in the environment therefore splits
the fleet for the length of the rollout, with new pods placing an
environment on one shard and old pods on another. A grace window
measured in seconds cannot cover that, and the same knob times the
existing mint-kind flip so it cannot simply be lengthened. An
environment variable also cannot record its own flip time, and an
operator cannot know a rollout's end in advance.

So the list, its grace stamp and the override are global flags, written
server-side against the control-plane clock under an advisory lock. This
branch adds no environment variables.

## The write path, which is live

Stamping generalises to any number of graced flag groups in one
transaction under one lock. That has three consequences a reviewer
should look at directly:

- It closes a real bug. `runOpsMintKind` is an editable control on the
global flags page, and that page previously wrote it with a bare upsert:
no lock, no stamp. An operator flipping mint kind through the UI got an
ungraced flip, so every pod crossed the cutover at a different moment.
Verified against a running instance, before and after.
- A graced group is all-or-nothing. Submitting its primary writes the
group with a fresh stamp; omitting it deletes the primary and its stamp
together, because a stamp left without its primary keeps being served
and would mint into a shard just removed.
- The advisory lock takes the previous id as well as the current one, in
a fixed order, so writers on an older release still serialise during a
rollout. The legacy id can be dropped one release after this ships.

This folds with #4751 rather than replacing it: its `unlockLockedFlags`
rule decides what the sweep may delete, and the graced groups keep their
stamp under the lock. Both sets of tests pass.

## Notes for review

Determinism is a property of the pure core for fixed inputs. The wrapper
supplies the clock, the same split `effectiveMintKind` already uses. A
failed read of the list falls back to the current id format rather than
guessing.

Six flags appear in the admin pages immediately. The two pins are
per-organization, so they render read-only on the global page. The list,
its stamp and the override are deployment-wide, so they render read-only
in the organization dialog.

Nothing bounds the active list against shards that actually exist. That
is safe while nothing mints, but the change that carries a shard key
into an id must land after the shard descriptors bound the list, or
bound it itself.

Author: d-cs

apps/webapp/app/components/admin/flagChangeList.ts

@@ -0,0 +1,56 @@
+import { derivedFlagsClearedWith } from "~/v3/featureFlags";
+
+export type FlagChange =
+  | { key: string; type: "added"; newVal: string }
+  | { key: string; type: "removed"; oldVal: string }
+  | { key: string; type: "changed"; oldVal: string; newVal: string };
+
+/**
+ * What a global flag save will do, for the confirm dialog.
+ *
+ * A graced primary that is unset also clears its stamps. Those keys are locked, so the caller
+ * filters them out of `initialValues` — the cascade therefore reads `storedValues`, which is the
+ * unfiltered set the loader returned. Reading `initialValues` finds nothing and understates the
+ * deletion, which is the defect this parameter exists to prevent.
+ */
+export function buildFlagChangeList(params: {
+  editableKeys: readonly string[];
+  lockedKeys: readonly string[];
+  initialValues: Record<string, unknown>;
+  storedValues: Record<string, unknown>;
+  newValues: Record<string, unknown>;
+}): FlagChange[] {
+  const { editableKeys, initialValues, storedValues, newValues } = params;
+
+  return editableKeys.flatMap<FlagChange>((key) => {
+    const wasSet = key in initialValues;
+    const isSet = key in newValues;
+    const oldVal = initialValues[key];
+    const newVal = newValues[key];
+
+    if (!wasSet && !isSet) return [];
+    if (wasSet && isSet && stableValue(oldVal) === stableValue(newVal)) return [];
+
+    if (!wasSet && isSet) {
+      return [{ key, type: "added", newVal: String(newVal) }];
+    }
+
+    if (wasSet && !isSet) {
+      // Only an unset clears the stamps. A change re-stamps instead.
+      const cascaded = derivedFlagsClearedWith(key)
+        .filter((derived) => derived in storedValues)
+        .map<FlagChange>((derived) => ({
+          key: derived,
+          type: "removed",
+          oldVal: String(storedValues[derived]),
+        }));
+      return [{ key, type: "removed", oldVal: String(oldVal) }, ...cascaded];
+    }
+
+    return [{ key, type: "changed", oldVal: String(oldVal), newVal: String(newVal) }];
+  });
+}
+
+function stableValue(value: unknown): string {
+  return JSON.stringify(value ?? null);
+}

apps/webapp/app/routes/admin.api.v1.feature-flags.ts

@@ -3,7 +3,12 @@ import { json } from "@remix-run/server-runtime";
 import { prisma } from "~/db.server";
 import { env } from "~/env.server";
 import { requireAdminApiRequest } from "~/services/personalAccessToken.server";
-import { applyGlobalMintKindFlip, makeSetMultipleFlags } from "~/v3/featureFlags.server";
+import {
+  applyGlobalGracedFlips,
+  makeSetMultipleFlags,
+  touchesGracedGroup,
+  withoutDerivedKeys,
+} from "~/v3/featureFlags.server";
 import { validatePartialFeatureFlags } from "~/v3/featureFlags";
 
 export async function action({ request }: ActionFunctionArgs) {
@@ -25,19 +30,16 @@ export async function action({ request }: ActionFunctionArgs) {
       );
     }
 
-    // Derived grace-stamp fields are computed server-side; never trust them from the body.
-    const {
-      runOpsMintKindPrev: _ignoredPrev,
-      runOpsMintKindFlippedAt: _ignoredFlippedAt,
-      ...requestedFlags
-    } = validationResult.data;
+    // Both the strip and the branch derive from the graced-group table, so adding a group needs
+    // no edit here. Naming the keys inline is how a new group ends up writing its stamp straight
+    // from the request body, with no lock.
+    const requestedFlags = withoutDerivedKeys(validationResult.data) as Partial<
+      typeof validationResult.data
+    >;
 
-    // A global mint-kind flip stamps its grace window under a lock (applyGlobalMintKindFlip);
-    // any other flag save writes directly.
-    const updatedFlags =
-      requestedFlags.runOpsMintKind !== undefined
-        ? await applyGlobalMintKindFlip(prisma, requestedFlags, env.RUN_OPS_MINT_FLIP_GRACE_MS)
-        : await makeSetMultipleFlags(prisma)(requestedFlags);
+    const updatedFlags = touchesGracedGroup(requestedFlags)
+      ? await applyGlobalGracedFlips(prisma, requestedFlags, env.RUN_OPS_MINT_FLIP_GRACE_MS)
+      : await makeSetMultipleFlags(prisma)(requestedFlags);
 
     return json({
       success: true,

apps/webapp/app/routes/admin.feature-flags.tsx

@@ -14,6 +14,7 @@ import {
   type FeatureFlagKey,
   type FlagControlType,
   getAllFlagControlTypes,
+  lockedFlagsInPayload,
   validatePartialFeatureFlags,
 } from "~/v3/featureFlags";
 import { flags as getGlobalFlags, replaceGlobalFeatureFlags } from "~/v3/featureFlags.server";
@@ -29,6 +30,7 @@ import {
   DialogFooter,
 } from "~/components/primitives/Dialog";
 import { cn } from "~/utils/cn";
+import { buildFlagChangeList } from "~/components/admin/flagChangeList";
 import {
   UNSET_VALUE,
   BooleanControl,
@@ -111,17 +113,12 @@ export const action = dashboardAction(
 
     const { isManagedCloud } = featuresForRequest(request);
 
-    // On managed cloud, reject if payload includes locked flags
-    if (isManagedCloud) {
-      const lockedInPayload = Object.keys(parsed.data.flags).filter((key) =>
-        GLOBAL_LOCKED_FLAGS.includes(key)
+    const lockedInPayload = lockedFlagsInPayload(Object.keys(parsed.data.flags), isManagedCloud);
+    if (lockedInPayload.length > 0) {
+      return json(
+        { error: `Cannot modify locked flags: ${lockedInPayload.join(", ")}` },
+        { status: 400 }
       );
-      if (lockedInPayload.length > 0) {
-        return json(
-          { error: `Cannot modify locked flags: ${lockedInPayload.join(", ")}` },
-          { status: 400 }
-        );
-      }
     }
 
     const validationResult = validatePartialFeatureFlags(parsed.data.flags);
@@ -137,6 +134,7 @@ export const action = dashboardAction(
       catalogKeys: Object.keys(getAllFlagControlTypes()) as FeatureFlagKey[],
       isManagedCloud,
       unlockLockedFlags: parsed.data.unlockLockedFlags ?? false,
+      graceMs: env.RUN_OPS_MINT_FLIP_GRACE_MS,
     });
 
     return json({ success: true });
@@ -401,6 +399,7 @@ export default function AdminFeatureFlagsRoute() {
         open={confirmOpen}
         onOpenChange={setConfirmOpen}
         initialValues={initialValues}
+        storedValues={allFlags}
         newValues={values}
         controlTypes={typedControlTypes}
         lockedKeys={unlocked ? [] : GLOBAL_LOCKED_FLAGS}
@@ -467,6 +466,7 @@ function ConfirmDialog({
   open,
   onOpenChange,
   initialValues,
+  storedValues,
   newValues,
   controlTypes,
   lockedKeys,
@@ -477,6 +477,7 @@ function ConfirmDialog({
   open: boolean;
   onOpenChange: (open: boolean) => void;
   initialValues: Record<string, unknown>;
+  storedValues: Record<string, unknown>;
   newValues: Record<string, unknown>;
   controlTypes: Record<string, FlagControlType>;
   lockedKeys: readonly string[];
@@ -488,34 +489,12 @@ function ConfirmDialog({
     .filter((key) => !lockedKeys.includes(key))
     .sort();
 
-  type Change =
-    | { key: string; type: "added"; newVal: string }
-    | { key: string; type: "removed"; oldVal: string }
-    | { key: string; type: "changed"; oldVal: string; newVal: string };
-
-  const changes = editableKeys.flatMap<Change>((key) => {
-    const wasSet = key in initialValues;
-    const isSet = key in newValues;
-    const oldVal = initialValues[key];
-    const newVal = newValues[key];
-
-    if (!wasSet && !isSet) return [];
-    if (wasSet && isSet && stableStringify(oldVal) === stableStringify(newVal)) return [];
-
-    if (!wasSet && isSet) {
-      return [{ key, type: "added" as const, newVal: String(newVal) }];
-    }
-    if (wasSet && !isSet) {
-      return [{ key, type: "removed" as const, oldVal: String(oldVal) }];
-    }
-    return [
-      {
-        key,
-        type: "changed" as const,
-        oldVal: String(oldVal),
-        newVal: String(newVal),
-      },
-    ];
+  const changes = buildFlagChangeList({
+    editableKeys,
+    lockedKeys,
+    initialValues,
+    storedValues,
+    newValues,
   });
 
   return (