Add option to require ready admission checks when selecting worker cluster in MultiKueue

[fg91]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Multi-kueue currently only considers quota reservations but not admission checks when selecting a worker cluster.

When using Kueue with the kueue.x-k8s.io/provisioning-request admission check, one typically sets very high quotas as only the provisioning request determines whether a workload can be scheduled, see e.g. here:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: "dws-cluster-queue"
spec:
    ...
          resources:
            ...
            - name: "nvidia.com/gpu"
              nominalQuota: 1000000000 # "Infinite" quota 
  admissionChecks:
    - dws-prov

When using such a ClusterQueue with "infinite" resource quota but with an admission check in the worker clusters of MultiKueue, the current behaviour is that all worker clusters immediately grant a quota reservation and the first one to do so is picked as the one running the workload. The provisioning request is immediately deleted in all other worker clusters.

This behaviour can be undesirable because the picked cluster might not be the one that would have granted the provisioning request first.

In this PR I'm introducing a new MultiKueue config flag that enables exactly this behaviour:

In a mu
multiKueue:
  requireAllAdmissionChecksReady: true

By default, the new feature is disabled, hence fully backwards compatible. If enabled, not only the quota reservation but also the admission checks are evaluated when picking a worker cluster. This way, with the AllAtOnce strategy, the provisioning request is kept in all worker clusters until one of them provisions the provisioning request.

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Yes, there is a new feature flag multiKueue.requireAllAdmissionChecksReady users can enable. By default, there is no change in behaviour.

-->

MultiKueue can now consider admission checks (e.g. for the provisioning request api integration) when determining which worker cluster to run the workload on. Set `multiKueue.requireAllAdmissionChecksReady: true` to activate this behaviour.

[linux-foundation-easycla]

• ❌ - login: @fg91 / name: Fabio M. Graetz, Ph.D. . The commit (4220846, 94280b5, a5178ad) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	4220846
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/695f8551d2110a000880c353
😎 Deploy Preview	https://deploy-preview-8465--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

Welcome @fg91!

It looks like this is your first PR to kubernetes-sigs/kueue 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kueue has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fg91
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @fg91. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mimowo]

/ok-to-test
Could you please open a PR to update the MultiKueue KEP so that we can learn better about the use-case and discuss the API.

About the API I have two hesitations:

• could be a knob on the MultiKueueConfig which would provide a better granularity (per CQ) rather than global
• I prefer to use enum fields rather than bool, because maybe in the future we will have more strategies like "RequireProvisioningRequestReady". So maybe "WorkerSelectionStrategy: FistQuotaReserved", default, and "WorkerSelectionStrategy: FirstWithAllAdmissionChecksReady", or "WorkerSelectionStrategy: FirstWithAdmissionsChecksReady" (and there would be another field to specify the set of awaited admission checks.

wdyt?

[fg91]

Thank you for the quick reply!

Could you please open a PR to update the MultiKueue KEP so that we can learn better about the use-case and discuss the API.
About the API I have two hesitations:

Will open the KEP PR and I'm happy to change the api after we discussed how best to expose this.

[mimowo]

sgtm, it is perfectly reasonable (and encouraged) to start prototyping / implementing along with the design. My comment was just to inform that the KEP update will be required to get merged before merging the actual implementation.

[k8s-ci-robot]

@fg91: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-test-integration-extended-main	4220846	link	true	/test pull-kueue-test-integration-extended-main
pull-kueue-verify-main	4220846	link	true	/test pull-kueue-verify-main
pull-kueue-test-e2e-main-1-35	4220846	link	true	/test pull-kueue-test-e2e-main-1-35

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.