Skip to content

driver/docker-container: verify buildkitd readiness before returning client - #4029

Closed
lyao-77 wants to merge 1 commit into
docker:masterfrom
lyao-77:fix/docker-container-first-build-readiness
Closed

driver/docker-container: verify buildkitd readiness before returning client#4029
lyao-77 wants to merge 1 commit into
docker:masterfrom
lyao-77:fix/docker-container-first-build-readiness

Conversation

@lyao-77

@lyao-77 lyao-77 commented Aug 20, 2026

Copy link
Copy Markdown

Problem

The docker-container driver has a first-build bootstrap race. When a build connects to a freshly created (never-bootstrapped) builder after the buildkitd container reports Running but before buildkitd has bound its socket, the build fails immediately with no retry:

error: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory
ERROR: failed to build: listing workers for Build: failed to list workers: Unavailable: connection error: desc = "error reading server preface: EOF"

It shows up most when several builds target the same new builder concurrently (CI that creates a builder and fans out builds): the first build triggers bootstrap, and siblings that connect during the socket-bind window fail hard. Same symptom as #1570 (reported without a root cause).

Root cause (against current master)

The bootstrap path is race-tolerant, but the "already Running" fast path is not:

  • driver/driver.goBoot() skips Bootstrap() (and the readiness wait() it performs) whenever Info() reports Running, then calls Client().
  • driver/docker-container/driver.goClient() -> Dial() execs buildctl dial-stdio with no readiness check. The exec succeeds even while buildkitd is still binding its socket; the failure only surfaces on the first RPC as error reading server preface: EOF.
  • driver/driver.goBoot() only retries when the error matches ErrNotRunning, which the docker-container driver never returns, so the connection error is fatal.

By contrast the bootstrap path tolerates the race: create() ignores the container name-conflict, and wait() polls buildctl debug workers ~15x with backoff until buildkitd answers. Only the Running fast path lacks that readiness wait.

Fix

Run the existing wait() readiness loop in Client() before dialing, so every returned client is backed by a responsive buildkitd. Once buildkitd answers this is a single cheap buildctl debug workers probe on the warm path.

I considered the alternative of classifying dial-stdio failures as ErrNotRunning so Boot()'s retry loop covers the window, but that alone isn't viable: the gRPC dial is lazy, so the exec succeeds and the error only appears on the first RPC after Boot() has already returned the client. An explicit readiness probe (this change) is needed either way.

Reproducer

cpu-quota throttles the container, widening the buildkitd startup window so the race fires reliably.

#!/usr/bin/env bash
set -u
mkdir -p /tmp/bxrace && cd /tmp/bxrace
printf 'FROM scratch\nCOPY marker /marker\n' > Dockerfile
echo hi > marker

docker buildx rm -f racer 2>/dev/null
docker buildx create --name racer --driver docker-container --driver-opt cpu-quota=3000

docker buildx build --builder racer --quiet . >out.boot 2>&1 &
ctr=buildx_buildkit_racer0
for _ in $(seq 1 6000); do [ "$(docker inspect -f '{{.State.Running}}' $ctr 2>/dev/null)" = true ] && break; done
for i in 1 2 3 4 5; do docker buildx build --builder racer --quiet . >out.$i 2>&1 & sleep 0.15; done
wait

grep -l 'server preface' out.* && echo RACE REPRODUCED

Related: #1570

…client

Boot() skips Bootstrap() (and the readiness wait() it performs) whenever
Info() reports the container as Running. On a freshly created builder the
container can report Running before buildkitd has bound its socket, so a
build that connects in that window has dial-stdio succeed but the first RPC
fail with "error reading server preface: EOF" (surfacing as
"dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory"
from buildctl). Boot() only retries errors matching ErrNotRunning, which the
docker-container driver never returns, so the failure is fatal.

Reuse the existing wait() loop in Client() so every returned client is backed
by a responsive buildkitd. Once buildkitd answers this is a single cheap
"buildctl debug workers" probe.

Signed-off-by: lyao-77 <lyao@confluent.io>
@crazy-max

Copy link
Copy Markdown
Member

@lyao-77 Should work with #4027 if you can test

@crazy-max crazy-max closed this Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants