Skip to content

[Fix] Self-hosted tasks stuck at Booting environment with fetch failed#563

Merged
daniel-lxs merged 3 commits into
developfrom
fix/docker-worker-trpc-env-boot-02o0x98rgoe7u
Jul 20, 2026
Merged

[Fix] Self-hosted tasks stuck at Booting environment with fetch failed#563
daniel-lxs merged 3 commits into
developfrom
fix/docker-worker-trpc-env-boot-02o0x98rgoe7u

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Opened on behalf of Daniel Riccio. Follow up by mentioning @roomote-roomote, in the web UI, or in Discord.

Related issue

Fixes #555

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

What changed

Self-hosted Docker tasks no longer die immediately at boot with fetch failed when the deployment’s public TRPC_URL cannot be reached from the task sandbox.

When DOCKER_WORKER_NETWORK is set, the controller injects the in-network API alias (http://api:3001) as the worker TRPC_URL instead of forwarding a public reverse-proxy URL (including cases that already point at the api host but still carry a /_roomote-api path). Local Docker without a control network still rewrites localhost to host.docker.internal.

Spawn fails fast if required launch env (AUTH_TOKEN, TRPC_URL, R_APP_URL) is empty, logs the resolved worker TRPC_URL without userinfo credentials or query/hash, and logs injected env key names (not secret values).

Operator note: TRPC_URL / AUTH_TOKEN are still passed via docker exec -e, so they intentionally do not appear in docker inspect Config.Env or a later plain docker exec shell. Deploy the controller image/service that contains this change (worker image rebuild alone is not enough).

Supersedes the approach in #556. Merged onto current develop (preflight / abort-aware spawn). Includes a patch changeset for the product release notes.

How it was tested

  • @roomote/controller Vitest: spawn-docker-worker.test.ts (39 passed), including control-plane TRPC URL cases and log sanitization
  • Pre-push suite (oxlint, residual eslint, check-types:fast, knip)
  • Reporter verification on self-hosted Docker: boot no longer fetch failed (shutdown persistence error reported separately)

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

Linked work items

Closes #555

Self-hosted sandboxes with DOCKER_WORKER_NETWORK blackhole the docker
gateway, so public TRPC_URL values fail with fetch failed at boot.
Also fail fast when required launch env is missing and log injected keys.
@roomote-roomote

roomote-roomote Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

No new code issues found. See task

Reviewed f45d9d6

@easonLiangWorldedtech

Copy link
Copy Markdown

✅ Test Results — Self-hosted Docker Worker Setup

Environment: Self-hosted Roomote on Windows via Docker Compose
Branch tested: PR #563 (commit 4f5f399f)
Controller image: roomote-app:local-self-host (PR--563)
Worker image: roomote-worker:local-self-host


✅ What Works

  1. TRPC_URL correctly set to in-network address

    trpcUrl:"http://api:3001"
    ✅ Worker spawned successfully for task run #173 in 5493ms
    

    PR fix confirmed working — worker receives http://api:3001 instead of public URL.

  2. Worker bootstrap succeeds (task run [Improve] Mark Daytona recommended and clarify API key scopes #172)

    Setup completed in 80.63s
    [SandboxServer] Listening on http://localhost:4200
    

    Full setup flow completes successfully — no more "fetch failed" at boot!


⚠️ Issue Found — Worker Shutdown TRPC Error

After worker completes setup and shuts down, it throws a TRPCClientError:

[SandboxServer] Shutting down...
[debug] [workerRuntimeEvents] Failed to persist event for task run 172: 
  Cannot access resources from a different run
Caught error: Cannot access resources from a different run: TRPCClientError: 
  Cannot access resources from a different run
    at TRPCClientError.from (file:///sandbox/worker/dist/worker.js:425:3077)

Details:

  • Worker [Improve] Mark Daytona recommended and clarify API key scopes #172 completed setup successfully (80.63s), then shutdown
  • During shutdown, worker attempts to persist runtime events and update compute usage via TRPC
  • All 4 persistence calls fail with Cannot access resources from a different run
  • This appears to be a TRPC client error where the worker is using an incorrect/different task run ID when calling the API

Controller logs confirm: Worker spawned successfully, but SleepCheck reports failed=1 because worker container exits after shutdown.


📊 Summary

Item Status
TRPC_URL in-network routing (#563 fix) ✅ Working
Worker bootstrap (no fetch failed) ✅ Working
Full setup flow completes ✅ Working
Worker shutdown DB persistence ❌ Error: "Cannot access resources from a different run"

The core issue this PR targets (fetch failed at boot) is resolved. The shutdown error appears to be a separate concern — possibly related to how the worker TRPC client resolves task run IDs during cleanup.

…trpc-env-boot-02o0x98rgoe7u

# Conflicts:
#	apps/controller/src/compute-providers/__tests__/spawn-docker-worker.test.ts
#	apps/controller/src/compute-providers/spawn-docker-worker.ts
@daniel-lxs
daniel-lxs marked this pull request as ready for review July 20, 2026 21:27
@daniel-lxs
daniel-lxs merged commit 97bf6dd into develop Jul 20, 2026
17 checks passed
@daniel-lxs
daniel-lxs deleted the fix/docker-worker-trpc-env-boot-02o0x98rgoe7u branch July 20, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Worker containers missing TRPC_URL/AUTH_TOKEN env vars — task stuck at 'Booting environment'

3 participants