feat(components): stream package_component instead of base64 JSON

[dawsontoth]

Closes #2150.

package_component returned the whole component tarball as base64 inside the JSON envelope. This adds a streamed shape, which is what the operation should have been doing all along — the machinery was already in the repo, the handler just wasn't using it.

The problem

components/operations.js:354:

const payload = (await packageDirectory(pathToProject, req)).toString('base64');
return { project, payload };

For an archive of N bytes that peaks at roughly 4.7N resident in the shared (on Fabric, multi-tenant) Harper process: 2N to accumulate and Buffer.concat the gzip chunks, +1.33N for the base64 string, +1.33N again when Fastify serializes the envelope.

And it has a hard ceiling, not just pressure — .toString('base64') throws ERR_STRING_TOO_LONG once the result exceeds V8's string cap:

$ node -e "const b=require('buffer'); console.log(b.constants.MAX_STRING_LENGTH)"
536870888        # 512 MiB → ~384 MiB of gzipped tar is the ceiling

The browser pays the same tax again, which is the downstream symptom in HarperFast/studio#1591: bytes transfer fine, then Chrome kills the renderer while decoding.

The change

stream: true returns the packer's Readable with download headers attached. Nothing new had to be built:

• streamPackagedDirectory() (components/packageComponent.ts:115) already returns a tar-fs → gzip Readable, and its docstring says it exists to avoid "the Node.js 2GB Buffer cap that the buffered variant runs into for large components". The handler simply called the buffered variant instead.
• serverHandlers.js:172 already pipes any returned Readable carrying a headers Map, and honours preCompressed so an already-gzipped body isn't compressed twice.
• handleGetDeploymentPayload() and get_backup are the existing precedents on that path.

Memory is constant on both ends, nothing is staged to disk, and there is no size ceiling.

estimate: true returns { project, total_size, dangling_symlinks } from a single scanPackageDirectory() walk without packaging anything, so a caller can decide whether the download is worth starting. Studio wants this for the include-node_modules case, where its own get_components tree gives it no size to work from.

The base64 shape stays the default, so every existing caller is untouched. The remote operation's only real consumer is Studio — the CLI's package alias packages locally for deploy (bin/cliOperations.ts:14) and never reads a remote payload.

Notes for review

• estimate is checked before stream and wins if both are set. They are deliberately not declared as Joi exclusive peers: oxor conflicts on presence, so it would reject a caller that always sends both fields with one set to false.
• content-disposition interpolates project, which is already narrowed by path.parse().name and validated against PROJECT_FILE_NAME_REGEX before it gets there, so it can't inject header content.
• The project-resolution guard is a drive-by fix with a real reason to be in this PR: the node_modules fallback's catch only rethrows ENOENT, so any other realpath failure left pathToProject undefined and walked it anyway. That was survivable when the response was buffered; on the stream path headers are committed before the walk runs, so it would surface as a 200 with a truncated body. Its test drives a real ENOTDIR (a rootPath whose node_modules is a file) and fails without the guard.
• A mid-walk failure after headers are committed still truncates the response — inherent to streaming, and the same behaviour get_backup/get_deployment_payload already have.
• Fabric Connect can't carry this. Central Manager's proxy is message-passing, not an HTTP reverse proxy, and caps bodies at 2 MB ("Fabric Connect body size must be less than 2MB. Utilize Direct Connect for larger payloads."). A streamed download needs a direct Bearer connection; that's a Studio-side concern, noted on studio#1591.

Testing

New unitTests/components/packageComponentOperation.test.js — 11 tests against a real temporary components root, covering all three response shapes, skip_node_modules on the stream path, a full stream → gunzip → tar-extract round trip back to the source tree, the estimate totals with and without node_modules, and both resolution failures. The transport half is already covered by unitTests/server/serverHelpers/serverHandlers.test.js, so this stops at the handler's return value.

Ran green locally alongside every other suite that imports the changed modules — packageComponent, envOperations, deploymentOperations, serverHandlers, fastifyRoutes/operations — 94 passing.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces streaming and estimation capabilities to the packageComponent operation, allowing clients to stream packaged component directories as raw gzip bytes or request a size and symlink estimate instead of receiving a base64-encoded payload. It also adds corresponding Joi validation schema updates and comprehensive unit tests covering these new response shapes and error handling behaviors. The review comment was removed because it incorrectly suggests asserting a specific system error code (ENOTDIR) that is swallowed by the implementation, which instead throws a generic 'Unable to locate project' error; therefore, we have no additional feedback to provide.

[claude]

Reviewed; no blockers found.

[kriszyp]

Nice, good call.

_{🤖 Reviewed with Codex}

[kriszyp]

great
_{🤖 Reviewed with Codex}