[Swarming] Push preprocess tasks to swarming queue

[IvanBM18]

Overview

This change adds support for scheduling tasks to the new Swarming backend. Because Swarming uses a different execution model and to be able to later account for backpressure it requires its own separate preprocess queue and a much lower default target size (5) to prevent unbounded task queuing.

By refactoring the cron scheduling logic, we can now simultaneously feed both the Swarming and Batch environments at their respective ideal rates.

Changes

• Add a distinct SWARMING_PREPROCESS_TARGET_SIZE_DEFAULT set to 5 to support the Swarming backend's task capacity needs.
• Define Swarming-specific queue mappings (SWARMING_QUEUES).
• Refactored the scheduler_fuzz so that the schedulers are composed of multiple helper functions to avoid complex inheritance:
  • Renamed them to Providers, as they no longer schedule tasks, they only look for them
• Update ChromeFuzzTaskScheduler to independently schedule Swarming tasks alongside standard Batch tasks.
  • Now it also looks for Android jobs as well
• Update schedule_fuzz_test.py to match the updated scheduler class signatures.

TODO

• src/clusterfuzz/_internal/base/feature_flags.py: Update this value based off dev & stage metrics and tests.

[IvanBM18]

The Swarming pool has a hard limit of 25 (LINUX) bots running 1 task each.

• At average 1 hour per fuzzing/swarming task, those 25 bots can finish 4 to 5 tasks every 10 minutes (the interval the cron job runs)
• Because the 2,000(in prod) preprocess tworkers almost instantly process the preprocess queue, the target size acts as an injection rate more than a buffer.

So, Injecting 5 tasks every 10 minutes matches the expected Swarming rate, preventing an infinitely growing backlog of stale tasks. This is still the default value, the real value is managed trough a feature flag, we will later tweak this feature flag based on metrics & how swarming handled this workload, so that we have a more acqurate value

[letitz]

I think we should prevent the infinitely-growing queue of tasks using CountTasks and a pretty low limit on the utask_main queue size, such that we can aim for a bit more saturation here (e.g. 10 tasks, such that we never expect the queue to be empty). This works for now though.

[fernandofloresg]

lgtm just had one question

[letitz]

One big comment about reworking the main logic here.

[letitz]

TODO should reference a bug link instead.

[IvanBM18]

I also agree its best to link to a bug, i previously asked gemini if we can add any go link or bug references in open source code/reviews in github, basically it said:

Throughout the repos you maintain (READMEs, code comments, etc.): Don't put any Google internal info, including go/ and b/ links. If you're sure the referenced content can be public, move it to GitHub issues or README or Wiki

Although im aware ai can allucinate and im not a open source expert, but as far as i know, thats the reason for this repo be full of #TODO(metzman): comments

So your call, can we include bug links in here?

[letitz]

@javanlacerda or @ViniciustCosta would know I'm sure!

[letitz]

I think we should prevent the infinitely-growing queue of tasks using CountTasks and a pretty low limit on the utask_main queue size, such that we can aim for a bit more saturation here (e.g. 10 tasks, such that we never expect the queue to be empty). This works for now though.

[letitz]

Looking good, a bunch of small comments left to address.

[letitz]

Do we use IS_SWARMING_JOB anywhere? Can we remove it for simplicity? (this can be in a followup)

[IvanBM18]

Yes we do, each time we define a new job we don't necessarily need specific swarming dimensions for it, so for this cases we have been using IS_SWARMING_JOB instead.