brev create returns 'unexpected EOF' rc=1 even after the workspace was actually created

[robobryce]

What

brev create <name> --type <T> sometimes exits with non-zero return code and a stderr body like:

WARN  RESTY Post "https://brevapi.us-west-2-prod.control-plane.brev.dev/api/organizations/<org-id>/workspaces?cli_version=v0.6.323&local=true&os=linux&utm_source=cli": unexpected EOF, Attempt 1
ERROR RESTY Post "https://brevapi.us-west-2-prod.control-plane.brev.dev/api/organizations/<org-id>/workspaces?cli_version=v0.6.323&local=true&os=linux&utm_source=cli": unexpected EOF
[Worker 1] m8i-flex.2xlarge Failed: ... Post "...": unexpected EOF
Warning: Only created 0/1 instances
could only create 0/1 instances

…even though the workspace was actually created — it shows up in brev ls --json immediately after, with status=DEPLOYING then status=RUNNING after a few minutes, and is fully usable. The non-zero exit is therefore a false negative: the API call appears to have completed server-side but the client got an EOF before reading the success response.

When it happens

Observed multiple times across our automation today, on m8i-flex.2xlarge (AWS) and previously on n2d-standard-8 (GCP). Intermittent — probably 1-in-5 or so during peak hours. Repro is unfortunately just "run brev create enough times".

Why it matters for automation

Downstream tooling that treats brev create's exit code as truth concludes "create failed" and either:

• Tries to clean up by calling brev delete <name> (which then succeeds, terminating the perfectly-good workspace).
• Marks an internal record (DB row, etc.) as destroyed/failed, leaving the orphan VM in Brev that no automation will reconcile.
• Bails the whole pipeline before whatever was supposed to run on the new workspace.

Our workaround (in our deploy script and in the manager's BrevDriver):

async def _workspace_was_created(self, instance_name: str) -> bool:
    try:
        return (await self.lookup_env_id(instance_name)) is not None
    except BrevError:
        return False

# After any non-zero `brev create` exit, call _workspace_was_created.
# If the workspace is in `brev ls --json`, treat the create as a success
# (log a warning) and proceed. If it's not, surface the original error.

What would fix this on the CLI side

Pick whichever fits the architecture best:

1. Server-side write idempotency + client retry. If POST /workspaces is keyed by a client-supplied request ID, the CLI can retry on EOF without risking a duplicate workspace. Today's behaviour suggests the write is succeeding before the response is fully returned, so a retry would either re-fetch the result (if the API is idempotent) or surface the actual created object.
2. CLI fallback on EOF: when the response is truncated mid-payload, the CLI internally calls GET /workspaces?name=<name> to check whether the workspace was actually created, and treats that as the source of truth before failing.
3. At minimum, a clearer signal in the error message that the EOF might mean "successfully created but response truncated" — so callers know to verify via brev ls rather than assume failure and clean up.

Workaround for anyone hitting this today

Verify via brev ls --json after every non-zero brev create exit before treating it as a hard failure.

[robobryce]

Update — unexpected EOF doesn't always mean "successfully created but response truncated". Hit this again today on m8i-flex.2xlarge. Same stderr pattern (RESTY Post .../workspaces: unexpected EOF → Warning: Only created 0/1 instances → rc=1), and the workspace did show up in brev ls --json afterwards — but the VM was in the broken state I described in #378, not the clean-false-negative state this issue assumed.

After ssh'ing to the orphan:

HOSTNAME=ip-172-31-33-15            # default AWS hostname; should be brev-<env-id>
BREV_ENV_ID=unset                   # never exported system-wide
ls /etc/brev/metadata.json          # No such file or directory
ls /opt/setup.sh                    # No such file or directory
systemctl is-active instance-oneshot.service
inactive
systemctl status instance-oneshot.service
Unit instance-oneshot.service could not be found.
systemctl --failed
apply_instance_tag_service.service  loaded failed failed apply_instance_tag_service AWS Helper Service
cloud-final.service                 loaded failed failed Cloud-init: Final Stage
nvidia-cdi-refresh.service          loaded failed failed Refresh NVIDIA CDI specification file

So the EOF on the create POST really does sometimes leave the workspace half-provisioned: the VM gets allocated and brev ls shows RUNNING, but Brev's first-boot script never runs. build_status stays empty (matches #378), the hostname is never rewritten, BREV_ENV_ID is never set, and cloud-final.service is in failed state — meaning the build-job that would have written /opt/setup.sh and the instance-oneshot.service unit was never enqueued.

Implication for client-side workarounds: the brev ls --json-after-rc!=0 check that this issue's "Workaround for anyone hitting this today" suggests is not sufficient on its own. Workspace-present can be either the recoverable case (this issue) or the stuck case (#378), and the ls row alone doesn't disambiguate them. Two options for callers:

1. Verify by build_status. Reject build_status == "" (or non-COMPLETED) as still-stuck, treat the EOF as a hard failure, and let the user retry. Brittle if build_status takes a few seconds to settle on the happy path.
2. Delete + retry. After rc!=0 + unexpected EOF, brev delete the orphan unconditionally and re-issue brev create. Wastes a few minutes when the workspace was actually fine (the "true" brev create returns 'unexpected EOF' rc=1 even after the workspace was actually created #382 case) but never lands on a stuck VM. Our manager just switched to this approach (https://github.com/brycelelbach-private/gateroom/pull/232).

Either way, the upstream fix would be cleaner — server-side write idempotency + client retry on EOF, or a CLI-internal GET /workspaces?name=… after EOF that returns the actual build_status. Cross-linked from #378 since the two issues have the same root cause from the client's vantage point.