feat(e2b): use Filesystem REST API for native binary I/O

[chcodex]

Summary

Fix E2B envd process.Process/Start server-streaming response parsing and replace shell+base64 file transfers with native binary HTTP via the E2B Filesystem REST API.

Root Cause

sint32 exit_code = 1 is a regular proto3 scalar field (not proto3_optional, not oneof). When exit code is 0, the field is omitted from both protobuf binary wire format and JSON output (proto3 default-value omission). The original code used DynamicMessage.hasField() to check presence — returns false for default values — causing exit code 0 to be treated as "no exit code received", resulting in exitCode = -2147483648 (Integer.MIN_VALUE).

Changes — Streaming Response Parsing

#	Change	Evidence
1	EndStreamResponse (flags & 0x02) — parse as JSON EndStreamMessage, throw on error, break cleanly on success	Connect Protocol Spec §Streaming-Response – bit 1 / Python SDK ServerStreamParser
2	Compressed frames (flags & 0x01) — throw explicit IOException	Connect spec – bit 0
3	Reserved bits (flags & 0xFC) — skip	Connect spec – 6 MSBs reserved
4	Proto3 scalar getField — replace hasField()+getField() with getField() for exit_code	Proto3 defaults / Python SDK event.event.end.exit_code
5	EndEvent.error to stderr — write when present (signal-killed processes)	Go handler / Python SDK command_handle.py#L115
6	JSON end event fix — always set end when JSON node exists; also parse error field	Python SDK uses JSON codec and reads event.event.end.error directly
7	Length mask removed — & 0x7FFFFFFF stripped	Connect spec — Message-Length is 4-byte unsigned int
8	Evidence comments — references added inline to Connect spec, Python SDK, Go handler, Proto3 docs	Each code location links to its authoritative source
9	buildEnvdRequest() extracted — deduplicate envd request header construction	Used by both runShell and drainStartStream

Changes — Filesystem REST API

Replace shell+base64 file transfers (33% overhead, 512KB truncation) with native binary HTTP.

#	File	Change
1	E2bEnvdProcessClient.java	Added uploadFile() (POST multipart) + downloadFile() (GET). Refactored buildFilesystemRequest() to share headers without hardcoding HTTP method.
2	E2bSandbox.java	doHydrateWorkspace(): chunked Python base64 → single uploadFile() + tar xf. doPersistWorkspace(): protobuf streaming → tar file + downloadFile() GET. Added public uploadFile()/downloadFile() delegates.
3	E2bSandboxFilesystem.java	New. Overrides uploadFiles()/downloadFiles() to use REST API. Falls back to super for non-E2B sandboxes (test mocks).
4	E2bFilesystemSpec.java	Overrides createFilesystem() → new E2bSandboxFilesystem().
5	SandboxFilesystemSpec.java	Added createFilesystem() default method returning SandboxBackedFilesystem().
6	HarnessAgent.java	new SandboxBackedFilesystem() → sandboxFilesystemSpec.createFilesystem().

Performance (before → after)

Operation	Before	After
Workspace hydrate (10KB)	~20 Python subprocess calls + base64	1 HTTP POST + tar xf
Workspace persist	protobuf RPC streaming (memory-heavy)	1 shell tar + 1 HTTP GET (raw binary)
Single file upload	33% base64 shell overhead	raw binary multipart
Single file download	33% base64 + 512KB truncation bug	raw binary GET, no truncation

Tests

Unit Tests — Streaming Parsing (E2bEnvdProcessClientTest) — 14 tests

Test	Verifies
Existing (7)	codec selection, frame parsing, malformed base64, missing-end sentinel, single end frame
endStreamResponseWithErrorThrows	flags=0x02 with error JSON → SandboxRuntimeException
endStreamResponseWithoutErrorBreaksCleanly	data frame + EndStreamResponse {} → clean break
exitCodeZeroDefaultWhenOmitted	end:{} (exitCode omitted) → exit=0 (proto3 default via getField)
exitCodeNonZero	end:{exitCode:42} → exit=42
endEventErrorWrittenToStderr	end:{error:"signal: 9"} → stderr contains error
compressedFrameThrowsIOException	flags=0x01 → IOException
reservedFlagsFrameIsSkipped	flags=0x04 → skipped, normal events processed

Unit Tests — Filesystem (E2bSandboxFilesystemTest) — 6 tests

Test	Verifies
uploadFiles_delegatesToE2bSandbox	Delegation to E2bSandbox.uploadFile()
uploadFiles_returnsFailureOnException	Exception → FileUploadResponse.error
downloadFiles_delegatesToE2bSandbox	Delegation to E2bSandbox.downloadFile()
downloadFiles_returnsFailureOnException	Exception → FileDownloadResponse.error
uploadFiles_fallsBackToSuperWhenNotE2bSandbox	Non-E2B sandbox → SandboxBackedFilesystem fallback
downloadFiles_fallsBackToSuperWhenNotE2bSandbox	Same for download

Integration Tests (E2bEnvdProcessClientIntegrationTest) — 2 tests (require -DE2B_API_KEY)

Test	Verifies
captureRawConnectFrames	Raw Connect protocol frame flags from real server
signalKillViaRunShell	Thread A runs sleep 120; Thread B kills via killall; verifies EndEvent.error captured to stderr

Test Plan

# All tests
mvn test -pl agentscope-extensions/agentscope-extensions-sandbox/agentscope-extensions-sandbox-e2b -am

# Single test class
mvn test -pl agentscope-extensions/agentscope-extensions-sandbox/agentscope-extensions-sandbox-e2b \
  -Dtest="E2bSandboxFilesystemTest"

# Integration tests (requires E2B API key)
mvn test -pl agentscope-extensions/agentscope-extensions-sandbox/agentscope-extensions-sandbox-e2b \
  -DE2B_API_KEY=<key> -DE2B_TEMPLATE_ID=code-interpreter-v1

[codecov]

Codecov Report

❌ Patch coverage is 48.95105% with 73 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...e/extensions/sandbox/e2b/E2bEnvdProcessClient.java	41.93%	51 Missing and 3 partials ⚠️
.../agentscope/extensions/sandbox/e2b/E2bSandbox.java	0.00%	18 Missing ⚠️
...cope/extensions/sandbox/e2b/E2bFilesystemSpec.java	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[jujn]

Suggest supplementary test

[AgentScopeJavaBot]

🤖 AI Review

This PR improves the Connect protocol frame parsing in E2bEnvdProcessClient by properly handling end-stream frames (flag 0x02), compressed frames (flag 0x01), and unknown flags — replacing the previous naive flags != 0x00 check. It also adds SLF4J debug logging for better diagnosability and a clean parseEndStreamResponse() / EndStreamMessage record for end-stream error parsing. The changes are well-scoped and align with the Connect protocol specification. A few minor robustness gaps were identified around defensive null handling and a silently-swallowed exception that should be addressed for consistency with the new logging additions.

(inline comments could not be attached — line numbers fell outside PR hunks. See archived report.)

[AgentScopeJavaBot]

🤖 AI Review

This PR improves the Connect protocol frame parsing in E2bEnvdProcessClient by properly handling end-stream frames (flag 0x02), compressed frames (flag 0x01), and unknown flags — replacing the previous naive flags != 0x00 check. It also adds SLF4J debug logging for better diagnosability and a clean parseEndStreamResponse() / EndStreamMessage record for end-stream error parsing. The changes are well-scoped and align with the Connect protocol specification. A few minor robustness gaps were identified around defensive null handling and a silently-swallowed exception that should be addressed for consistency with the new logging additions.

(inline comments could not be attached — line numbers fell outside PR hunks. See archived report.)