route bake targets across pods with random loadbalance

[areebahmeddd]

loadbalance=random on the kubernetes driver was sending all bake targets to the same pod.
the client connection was cached after the first dial, so pod selection only happened once.

fix this by creating a new connection per target when loadbalance=random is enabled.
also disable the shared local mount session optimization in this mode.

Closes #3382

[AkihiroSuda]

loadbalance=random on the kubernetes driver was sending all bake targets to the same pod. the client connection was cached after the first dial, so pod selection only happened once.

fix this by creating a new connection per target when loadbalance=random is enabled.

What about loadbalance=sticky ?

[AkihiroSuda]

Also, please squash the commits

[areebahmeddd]

What about loadbalance=sticky ?

no change. it reuses the same shared connection path

Also, please squash the commits

sure 🙂

second review

[crazy-max]

I was looking at

buildx/driver/kubernetes/podchooser/podchooser.go

Lines 29 to 45 in 0b358c7

	func (pc *RandomPodChooser) ChoosePod(ctx context.Context) (*corev1.Pod, error) {
	pods, err := ListRunningPods(ctx, pc.PodClient, pc.Deployment, pc.StatefulSet)
	if err != nil {
	return nil, err
	}
	if len(pods) == 0 {
	return nil, errors.New("no running buildkit pods found")
	}
	randSource := pc.RandSource
	if randSource == nil {
	randSource = rand.NewSource(time.Now().Unix())
	}
	rnd := rand.New(randSource) // #nosec G404 -- no strong seeding required
	n := rnd.Int() % len(pods)
	logrus.Debugf("RandomPodChooser.ChoosePod(): len(pods)=%d, n=%d", len(pods), n)
	return pods[n], nil
	}

Seems to still looks capable of picking the same pod for all targets in one bake no? ChoosePod() creates a new RNG on every call and seeds it with time.Now().Unix(). With this PR, multiple targets can create fresh clients and dial within the same second, so each call can get the same seed and produce the same first random value iiuc.

Maybe we should seed the RNG once at the package level, with concurrency handled, or use another safe random selection path? Otherwise this PR may still reproduce the original issue.

[areebahmeddd]

Hm, let me take a look

[areebahmeddd]

gentle ping! @crazy-max @AkihiroSuda 🙂

[AkihiroSuda]

gentle ping! @crazy-max @AkihiroSuda 🙂

#3861 (comment)

[areebahmeddd]

ah yes, done : )
cc @AkihiroSuda

[crazy-max]

LGTM thanks