Skip to content

Commit 2aa6420

Browse files
authored
fix(webapp): survive asset hash rotation across rolling deploys (#4260)
### Problem Webapp HTML references content-hashed /build assets, and each Docker image contains exactly one build with a hard 404 for unknown hashes. During a rolling deploy, a client holding HTML from the old build may request old asset hashes from a replica running the new image, causing missing styles or failed chunk loads. The page should recover automatically once a compatible build becomes available, without reload loops or unnecessary interruptions during normal deployments. ### What changed - Build changes alone do nothing — no polling, no automatic reloads. - If a CSS or JavaScript asset fails to load, a recovery overlay is shown immediately. - While the server still reports the same build, the client polls for a newer build using exponential backoff (up to ~60s). As soon as a newer build is detected, the page reloads automatically. - If no newer build appears within the timeout, recovery falls back to a manual Reload action. - If recovery still fails after the automatic reload, the client stops retrying and displays a final recovery screen instead of entering a reload loop. - Recovery preserves form values and scroll position across the automatic reload.
1 parent c936c79 commit 2aa6420

6 files changed

Lines changed: 312 additions & 1 deletion

File tree

.claude/rules/server-apps.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,12 @@ area: webapp
1414
type: fix
1515
---
1616
17-
Brief description of what changed and why.
17+
Fix pages occasionally loading unstyled during deploys. The dashboard now recovers automatically.
1818
EOF
1919
```
2020

2121
- **area**: `webapp` | `supervisor`
2222
- **type**: `feature` | `fix` | `improvement` | `breaking`
2323
- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).
24+
25+
The body ships **verbatim in user-facing release notes**. Keep it to 1–2 short sentences, non-technical, written for a dashboard user: describe what changed for them, never the implementation (no header names, endpoints, middleware, storage mechanisms, internal tools). See `.server-changes/README.md` for full guidance.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: fix
4+
---
5+
6+
Fix pages occasionally loading unstyled or failing to load during deploys. The dashboard now detects this and reloads to recover automatically, or prompts you to reload if it can't.
Lines changed: 272 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,272 @@
1+
// Recovers from deploys rotating the content-hashed /build assets out from
2+
// under a page. Policy:
3+
//
4+
// 1. A working page is never touched just because a new build exists. Remix
5+
// loader fetches (?_data=) are stamped with X-Build-Id by the server;
6+
// when a navigation reveals a different build, the navigation is turned
7+
// into a full document load — the user lands on the new build without
8+
// ever seeing an incompatible state.
9+
// 2. Only real incompatibility (a /build stylesheet/script 404 or a failed
10+
// chunk import) triggers recovery: an overlay goes up immediately, the
11+
// script polls /build-version with backoff, and reloads once the server
12+
// reports a different build than the page was rendered with
13+
// (window.__remixManifest.version). If versions never diverge or the
14+
// reload budget (one per observed build, 2 total) is spent, the overlay
15+
// offers a manual reload instead of leaving a dead page.
16+
// 3. Before a recovery reload, form fields and scroll position are
17+
// snapshotted to sessionStorage and restored (best-effort) after the
18+
// reload. history.state is deliberately not restored — the Remix router
19+
// owns it, and reviving a stale one can desync the router.
20+
//
21+
// Must render before <Links /> so the listener precedes the stylesheet.
22+
const script = `(function () {
23+
var VKEY = "trigger:assetRecovery";
24+
var SKEY = "trigger:recoverySnapshot";
25+
var MAX_RELOADS = 2;
26+
var RESET_AFTER = 300000;
27+
var SNAPSHOT_TTL = 30000;
28+
var CHECK_DELAYS = [0, 2000, 4000, 8000, 15000, 30000];
29+
var recovering = false;
30+
var navigated = false;
31+
32+
function ownVersion() {
33+
return window.__remixManifest && window.__remixManifest.version;
34+
}
35+
36+
function readJson(key) {
37+
try {
38+
return JSON.parse(sessionStorage.getItem(key) || "null");
39+
} catch (e) {
40+
return undefined;
41+
}
42+
}
43+
44+
function writeJson(key, value) {
45+
try {
46+
sessionStorage.setItem(key, JSON.stringify(value));
47+
return true;
48+
} catch (e) {
49+
return false;
50+
}
51+
}
52+
53+
// ---- form + scroll snapshot ------------------------------------------
54+
55+
function takeSnapshot() {
56+
var fields = [];
57+
var els = document.querySelectorAll("input, textarea, select");
58+
for (var i = 0; i < els.length; i++) {
59+
var el = els[i];
60+
var type = (el.getAttribute("type") || "").toLowerCase();
61+
if (type === "password" || type === "file" || type === "hidden") continue;
62+
fields.push({
63+
i: i,
64+
id: el.id || null,
65+
name: el.name || null,
66+
tag: el.tagName,
67+
value: el.value,
68+
checked: el.checked === true,
69+
});
70+
}
71+
writeJson(SKEY, {
72+
t: Date.now(),
73+
path: location.pathname,
74+
scrollY: window.scrollY,
75+
fields: fields,
76+
});
77+
}
78+
79+
function setNativeValue(el, value) {
80+
// Go through the prototype setter so React's value tracking notices the
81+
// change when the input event fires.
82+
var proto =
83+
el.tagName === "TEXTAREA"
84+
? window.HTMLTextAreaElement
85+
: el.tagName === "SELECT"
86+
? window.HTMLSelectElement
87+
: window.HTMLInputElement;
88+
var descriptor = Object.getOwnPropertyDescriptor(proto.prototype, "value");
89+
if (descriptor && descriptor.set) {
90+
descriptor.set.call(el, value);
91+
} else {
92+
el.value = value;
93+
}
94+
}
95+
96+
function restoreSnapshot() {
97+
var snapshot = readJson(SKEY);
98+
try {
99+
sessionStorage.removeItem(SKEY);
100+
} catch (e) {}
101+
if (!snapshot || snapshot.path !== location.pathname) return;
102+
if (Date.now() - (snapshot.t || 0) > SNAPSHOT_TTL) return;
103+
var els = document.querySelectorAll("input, textarea, select");
104+
for (var i = 0; i < snapshot.fields.length; i++) {
105+
var field = snapshot.fields[i];
106+
var el = (field.id && document.getElementById(field.id)) || els[field.i];
107+
if (!el || el.tagName !== field.tag || (el.name || null) !== field.name) continue;
108+
if (el.type === "checkbox" || el.type === "radio") {
109+
// click() keeps React state in sync with the DOM
110+
if (el.checked !== field.checked) el.click();
111+
} else if (field.value != null && el.value !== field.value) {
112+
setNativeValue(el, field.value);
113+
el.dispatchEvent(new Event("input", { bubbles: true }));
114+
el.dispatchEvent(new Event("change", { bubbles: true }));
115+
}
116+
}
117+
if (snapshot.scrollY) window.scrollTo(0, snapshot.scrollY);
118+
}
119+
120+
window.addEventListener("load", function () {
121+
setTimeout(restoreSnapshot, 100);
122+
});
123+
124+
function doReload() {
125+
takeSnapshot();
126+
location.reload();
127+
}
128+
129+
// ---- scenario 1: working page, navigation as the update point --------
130+
131+
var origFetch = window.fetch;
132+
window.fetch = function (input) {
133+
var result = origFetch.apply(this, arguments);
134+
try {
135+
var url = typeof input === "string" ? input : input && input.url;
136+
if (url && url.indexOf("_data=") !== -1) {
137+
result.then(function (response) {
138+
var server = response.headers.get("X-Build-Id");
139+
var mine = ownVersion();
140+
if (server && mine && server !== mine && !navigated && !recovering) {
141+
navigated = true;
142+
var target = new URL(url, location.origin);
143+
target.searchParams.delete("_data");
144+
location.assign(target.toString());
145+
}
146+
}, function () {});
147+
}
148+
} catch (e) {}
149+
return result;
150+
};
151+
152+
// ---- scenario 2: page is actually broken ------------------------------
153+
154+
function showOverlay(final) {
155+
if (!document.body) {
156+
document.addEventListener("DOMContentLoaded", function () {
157+
showOverlay(final);
158+
});
159+
return;
160+
}
161+
// ASCII only: documents are served without an explicit charset, so
162+
// non-ASCII here can render as mojibake on a broken page.
163+
var text = final ? "This page failed to load properly. Please reload." : "Loading...";
164+
var existing = document.getElementById("stale-asset-overlay-text");
165+
if (existing) {
166+
existing.textContent = text;
167+
if (final) document.getElementById("stale-asset-overlay-button").style.display = "";
168+
return;
169+
}
170+
var overlay = document.createElement("div");
171+
overlay.id = "stale-asset-overlay";
172+
overlay.style.cssText =
173+
"position:fixed;inset:0;z-index:2147483647;display:flex;flex-direction:column;gap:16px;align-items:center;justify-content:center;background:#121317;color:#d7d9dd;font:15px/1.5 system-ui,sans-serif;text-align:center;padding:24px";
174+
var message = document.createElement("p");
175+
message.id = "stale-asset-overlay-text";
176+
message.textContent = text;
177+
var button = document.createElement("button");
178+
button.id = "stale-asset-overlay-button";
179+
button.textContent = "Reload";
180+
button.style.cssText =
181+
"border:0;border-radius:4px;padding:7px 18px;background:#6366f1;color:#fff;font:inherit;cursor:pointer" +
182+
(final ? "" : ";display:none");
183+
button.onclick = doReload;
184+
overlay.appendChild(message);
185+
overlay.appendChild(button);
186+
document.body.appendChild(overlay);
187+
}
188+
189+
function reloadFor(serverVersion) {
190+
var state = readJson(VKEY);
191+
if (state === undefined) return showOverlay(true);
192+
state = state || {};
193+
if (Date.now() - (state.t || 0) > RESET_AFTER) state = {};
194+
// One reload per observed server version, MAX_RELOADS total: a page that
195+
// is still broken after reloading for this build gets the manual overlay
196+
// instead of reloading again.
197+
if (state.v === serverVersion || (state.reloads || 0) >= MAX_RELOADS) return showOverlay(true);
198+
if (!writeJson(VKEY, { v: serverVersion, reloads: (state.reloads || 0) + 1, t: Date.now() })) {
199+
return showOverlay(true);
200+
}
201+
doReload();
202+
}
203+
204+
function check(attempt) {
205+
origFetch("/build-version", { cache: "no-store" })
206+
.then(function (response) {
207+
return response.json();
208+
})
209+
.then(function (data) {
210+
var mine = ownVersion();
211+
if (data && data.version && mine && data.version !== mine) {
212+
reloadFor(data.version);
213+
} else {
214+
scheduleNext(attempt);
215+
}
216+
})
217+
.catch(function () {
218+
scheduleNext(attempt);
219+
});
220+
}
221+
222+
function scheduleNext(attempt) {
223+
var next = attempt + 1;
224+
if (next >= CHECK_DELAYS.length) return showOverlay(true);
225+
setTimeout(function () {
226+
check(next);
227+
}, CHECK_DELAYS[next]);
228+
}
229+
230+
function recover() {
231+
if (recovering) return;
232+
recovering = true;
233+
showOverlay(false);
234+
// __remixManifest is set by an inline script near the end of body; wait
235+
// for the document to finish parsing before comparing versions.
236+
if (document.readyState === "loading") {
237+
document.addEventListener("DOMContentLoaded", function () {
238+
check(0);
239+
});
240+
} else {
241+
check(0);
242+
}
243+
}
244+
245+
window.addEventListener(
246+
"error",
247+
function (event) {
248+
var el = event.target;
249+
if (!el || el === window) return;
250+
var url = el.tagName === "LINK" ? el.href : el.tagName === "SCRIPT" ? el.src : null;
251+
if (url && url.indexOf("/build/") !== -1) recover();
252+
},
253+
true
254+
);
255+
window.addEventListener("unhandledrejection", function (event) {
256+
var message = event.reason && event.reason.message;
257+
if (
258+
typeof message === "string" &&
259+
/dynamically imported module|Importing a module script failed|ChunkLoadError/i.test(message)
260+
) {
261+
recover();
262+
}
263+
});
264+
})();`;
265+
266+
export function StaleAssetRecovery({ isProduction }: { isProduction: boolean }) {
267+
if (!isProduction) {
268+
return null;
269+
}
270+
271+
return <script dangerouslySetInnerHTML={{ __html: script }} />;
272+
}

apps/webapp/app/entry.server.tsx

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,12 @@ export default function handleRequest(
5454
) {
5555
const url = new URL(request.url);
5656

57+
// Stale documents reference /build asset hashes that 404 after a deploy —
58+
// always revalidate HTML. Route-set headers win.
59+
if (!responseHeaders.has("Cache-Control")) {
60+
responseHeaders.set("Cache-Control", "no-cache");
61+
}
62+
5763
if (url.pathname.startsWith("/login")) {
5864
responseHeaders.set("X-Frame-Options", "SAMEORIGIN");
5965
responseHeaders.set("Content-Security-Policy", "frame-ancestors 'self'");

apps/webapp/app/root.tsx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import type { ToastMessage } from "~/models/message.server";
77
import { commitSession, getSession } from "~/models/message.server";
88
import tailwindStylesheetUrl from "~/tailwind.css";
99
import { RouteErrorDisplay } from "./components/ErrorDisplay";
10+
import { StaleAssetRecovery } from "./components/StaleAssetRecovery";
1011
import { AppContainer, MainCenteredContainer } from "./components/layout/AppLayout";
1112
import { ShortcutsProvider } from "./components/primitives/ShortcutsProvider";
1213
import { Toast } from "./components/primitives/Toast";
@@ -18,6 +19,11 @@ import { getUser } from "./services/session.server";
1819
import { getTimezonePreference } from "./services/preferences/uiPreferences.server";
1920
import { appEnvTitleTag } from "./utils";
2021

22+
// Derived here (not inside StaleAssetRecovery) so the shared component takes
23+
// the flag as a prop. NODE_ENV is statically replaced in browser bundles, and
24+
// the ErrorBoundary can't rely on loader data.
25+
const isProduction = process.env.NODE_ENV === "production";
26+
2127
export const links: LinksFunction = () => {
2228
return [{ rel: "stylesheet", href: tailwindStylesheetUrl }];
2329
};
@@ -99,6 +105,7 @@ export function ErrorBoundary() {
99105
<head>
100106
<meta charSet="utf-8" />
101107

108+
<StaleAssetRecovery isProduction={isProduction} />
102109
<Meta />
103110
<Links />
104111
</head>
@@ -125,6 +132,7 @@ export default function App() {
125132
<>
126133
<html lang="en" className="h-full" data-theme="dark">
127134
<head>
135+
<StaleAssetRecovery isProduction={isProduction} />
128136
<Meta />
129137
<Links />
130138
</head>

apps/webapp/server.ts

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,23 @@ if (ENABLE_CLUSTER && cluster.isPrimary) {
134134
const port = process.env.REMIX_APP_PORT || process.env.PORT || 3000;
135135

136136
if (process.env.HTTP_SERVER_DISABLED !== "true") {
137+
// Stamp every response with the build this replica is running. The
138+
// client-side stale-asset recovery reads it from Remix loader fetches to
139+
// turn an ordinary navigation into a full document load when the server
140+
// has moved to a new build.
141+
app.use((_req, res, next) => {
142+
res.set("X-Build-Id", build.assets.version);
143+
next();
144+
});
145+
146+
// Reports the build this replica is running. The client-side stale-asset
147+
// recovery polls it after a /build 404 and reloads only once the server
148+
// reports a different build than the one the page was rendered with.
149+
app.get("/build-version", (_req, res) => {
150+
res.set("Cache-Control", "no-store");
151+
res.json({ version: build.assets.version });
152+
});
153+
137154
const socketIo: { io: IoServer } | undefined = build.entry.module.socketIo;
138155
const wss: WebSocketServer | undefined = build.entry.module.wss;
139156
const apiRateLimiter: RateLimitMiddleware = build.entry.module.apiRateLimiter;

0 commit comments

Comments
 (0)