driver/docker-container: verify buildkitd readiness before returning client - #4029
Closed
lyao-77 wants to merge 1 commit into
Closed
driver/docker-container: verify buildkitd readiness before returning client#4029lyao-77 wants to merge 1 commit into
lyao-77 wants to merge 1 commit into
Conversation
…client Boot() skips Bootstrap() (and the readiness wait() it performs) whenever Info() reports the container as Running. On a freshly created builder the container can report Running before buildkitd has bound its socket, so a build that connects in that window has dial-stdio succeed but the first RPC fail with "error reading server preface: EOF" (surfacing as "dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory" from buildctl). Boot() only retries errors matching ErrNotRunning, which the docker-container driver never returns, so the failure is fatal. Reuse the existing wait() loop in Client() so every returned client is backed by a responsive buildkitd. Once buildkitd answers this is a single cheap "buildctl debug workers" probe. Signed-off-by: lyao-77 <lyao@confluent.io>
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
docker-containerdriver has a first-build bootstrap race. When a build connects to a freshly created (never-bootstrapped) builder after the buildkitd container reportsRunningbut before buildkitd has bound its socket, the build fails immediately with no retry:It shows up most when several builds target the same new builder concurrently (CI that creates a builder and fans out builds): the first build triggers bootstrap, and siblings that connect during the socket-bind window fail hard. Same symptom as #1570 (reported without a root cause).
Root cause (against current
master)The bootstrap path is race-tolerant, but the "already
Running" fast path is not:driver/driver.go—Boot()skipsBootstrap()(and the readinesswait()it performs) wheneverInfo()reportsRunning, then callsClient().driver/docker-container/driver.go—Client()->Dial()execsbuildctl dial-stdiowith no readiness check. The exec succeeds even while buildkitd is still binding its socket; the failure only surfaces on the first RPC aserror reading server preface: EOF.driver/driver.go—Boot()only retries when the error matchesErrNotRunning, which the docker-container driver never returns, so the connection error is fatal.By contrast the bootstrap path tolerates the race:
create()ignores the container name-conflict, andwait()pollsbuildctl debug workers~15x with backoff until buildkitd answers. Only theRunningfast path lacks that readiness wait.Fix
Run the existing
wait()readiness loop inClient()before dialing, so every returned client is backed by a responsive buildkitd. Once buildkitd answers this is a single cheapbuildctl debug workersprobe on the warm path.I considered the alternative of classifying
dial-stdiofailures asErrNotRunningsoBoot()'s retry loop covers the window, but that alone isn't viable: the gRPC dial is lazy, so the exec succeeds and the error only appears on the first RPC afterBoot()has already returned the client. An explicit readiness probe (this change) is needed either way.Reproducer
cpu-quotathrottles the container, widening the buildkitd startup window so the race fires reliably.Related: #1570