(metro-core): add manifest SHA-256 hashes and optional bundle cache layer integration#4576
(metro-core): add manifest SHA-256 hashes and optional bundle cache layer integration#4576zhongwuzw wants to merge 25 commits intomodule-federation:mainfrom
Conversation
✅ Deploy Preview for module-federation-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
…d add register function
…undle URL resolution
jbroma
left a comment
There was a problem hiding this comment.
hey @zhongwuzw , nice job with the implementation so far, I have few questions & one ideas about how to make this even better, let me know what you think!
There was a problem hiding this comment.
since there are no package.json changes in this PR, this is probably a leftover from previous changes, could you please verify this?
| const bundleContent = await fs.readFile( | ||
| saveBundleOpts.bundleOutput, | ||
| 'utf-8', | ||
| ); |
There was a problem hiding this comment.
perhaps we can use bundle directly here? I don't see any benefit of reading it again from the filesystem
There was a problem hiding this comment.
@jbroma Hi, I see we have --bundle-encoding in bundle-mf-remote — do we actually support non-UTF-8? https://github.com/zhongwuzw/core/blob/b2472772250dad91947eefa31b724582441f95e9/packages/metro-core/src/commands/bundle-remote/index.ts#L335
The option accepts utf8 | utf16le | ascii, but encoding info is never passed to the upload layer or stored in the manifest. The server/CDN has no idea what encoding the bundle was written in, and the client always decodes as UTF-8? If a user passes --bundle-encoding utf16le, the bundle will silently break at runtime I think. Did I miss anything?
There was a problem hiding this comment.
Probably not, it's fine to assume utf-8 for now - ideally we would store this information somewhere so we dont have to guess
| // Inject container bundle hash into metaData.buildInfo.hash | ||
| const containerFilename = federationConfig.filename; | ||
| if (bundleHashMap.has(containerFilename)) { | ||
| rawManifest.metaData.buildInfo.hash = |
There was a problem hiding this comment.
is this typed in the MF core or is this a non-standard field?
There was a problem hiding this comment.
shared[].hash is already typed in MF core (StatsShared.hash: string) — Metro generates it as an empty string and I populate it post-build. metaData.buildInfo.hash and expose[].hash are non-standard — StatsBuildInfo and StatsExpose don't have a hash field. Happy to add hash?: string to the core types if you'd like to formalize it.
There was a problem hiding this comment.
yes, let's standardize those then 👍
| afterResolve: (args) => { | ||
| // Register bundle hashes with cache layer for integrity verification | ||
| try { | ||
| const cacheLayer = (globalThis as any).__MFE_CACHE_LAYER__ as | ||
| | ICacheLayer | ||
| | undefined; | ||
| if (!cacheLayer) return args; | ||
|
|
||
| const __loadBundleAsync = | ||
| globalThis[`${__METRO_GLOBAL_PREFIX__ ?? ''}__loadBundleAsync`]; | ||
| const { origin, remoteInfo, remote } = args; | ||
| const manifestUrl = | ||
| 'entry' in remote ? (remote as any).entry : undefined; | ||
| if (manifestUrl && origin.snapshotHandler?.manifestCache) { | ||
| const manifest = | ||
| origin.snapshotHandler.manifestCache.get(manifestUrl); | ||
| if (manifest) { | ||
| // Container bundle hash | ||
| const containerHash = (manifest.metaData?.buildInfo as any)?.hash; | ||
| if (containerHash && remoteInfo.entry) { | ||
| cacheLayer.registerBundleHash(remoteInfo.entry, containerHash); | ||
| } | ||
|
|
||
| const loadBundleAsync = | ||
| __loadBundleAsync as typeof globalThis.__loadBundleAsync; | ||
| // Exposed + shared bundle hashes | ||
| const hashes = extractBundleHashes(manifest, manifestUrl); | ||
| for (const [url, hash] of hashes) { | ||
| // Strip query params — loadBundle looks up hashes by bare URL | ||
| cacheLayer.registerBundleHash(url.split('?')[0], hash); | ||
| } | ||
|
|
||
| if (!loadBundleAsync) { | ||
| throw new Error('loadBundleAsync is not defined'); | ||
| } | ||
| // Register manifest source for polling | ||
| cacheLayer.registerManifestSource(manifestUrl, extractBundleHashes); | ||
| } | ||
| } | ||
| } catch { | ||
| // non-critical — hash validation is best-effort | ||
| } | ||
| return args; | ||
| }, |
There was a problem hiding this comment.
I think we could extract this to the cache package and make it a separate runtime plugin instead - do you think it's possible?
| // For remote split bundles with cache enabled, convert relative paths to | ||
| // full URLs so they enter the same cache path as container bundles. | ||
| // In dev mode, getBundlePath returns relative paths unchanged, but we need | ||
| // full URLs for the cache layer (download + eval). | ||
| if (isSplitBundle && cacheLayer && publicPath && !isUrl(bundlePath)) { | ||
| bundlePath = joinComponents(publicPath, bundlePath); | ||
| } | ||
|
|
||
| // --- Cache layer: intercept bundles with full URLs (containers + remote split bundles) --- | ||
| if (cacheLayer && isUrl(bundlePath)) { | ||
| const { status } = await cacheLayer.loadBundle(bundlePath); | ||
| if (status === 'skipped') { | ||
| // Cache layer skipped — fall back to network load | ||
| const encodedBundlePath = bundlePath.replaceAll('../', '..%2F'); | ||
| await loadBundleAsync(encodedBundlePath); | ||
| } | ||
| // else: 'cache-hit' or 'downloaded' — bundle already eval'd by cache layer | ||
| } else { | ||
| // No cache: host split bundles (no publicPath), cache disabled, or native-cache not installed | ||
| const encodedBundlePath = bundlePath.replaceAll('../', '..%2F'); | ||
| await loadBundleAsync(encodedBundlePath); | ||
| } |
There was a problem hiding this comment.
This looks good but to me it feels out of place - with an optional layer we are introducing a lot of complexity into this module - do you think it would be possible to create wrapper around loadBundleAsync that introduces the cache layer and then load the MF wrapper?
I think we could rework the asyncRequire implementation, so that it would be possible to modify the actual loadBundleAsync and then add MF wrapper on top of it:
- InitializeCore runs ->
__loadBundleAsyncgets initialized with default implementation - Cache wrapper runs -> enhances
__loadBundleAsyncwith caching capabilities - MF metro-core wrapper runs -> adapts
__loadBundleAsyncto work with MF
This approach would most likely require splitting asyncRequire into two modules, something like this:
mf:init-async-require-> injects expo impl of async require if missingmf:adapt-async-require-> adapts existing impl of async require to work with federation
in the cache plugin you could then modify the resolver for either of those to ensure module with cache wrapper runs after init and before adapt
mf:async-require, left over for compatibility could be then just:
import `mf:init-async-require`;
import `mf:adapt-async-require`;
let me know what you think & if that makes sense to you
Description
Related Issue
Types of changes
Checklist