Task priority being ignored

[kyleabbott]

I am seeing an issue with task priority in the following setup:

1. hq v0.25.0 (built from source)
2. multiple running hq workers
3. multiple jobs with tasks that have no explicit priority set (default to zero - validated via hq task info to have priority==0)
4. multiple jobs where all tasks have priority -5
5. a new job is submitted where all tasks have priority -1 (validated via hq task info)

(all jobs submitted via JDF)

The issue is that the job in (5) ends up being allocated almost all of the worker resources while thousands of runnable tasks in (3), that hq sees as having higher priority, are in the WAITING state. Somehow the job being submitted last with all of its tasks having priority -1 has jumped the queue of priority 0 tasks.

I have not used the priority feature much, but I have used it occasionally prior to upgrading to v0.25.0 and did not notice this issue before.

[spirali]

This seems like a bug. Can you share with us the resource requests of these tasks? Or ideally the part of job definition file?

[kyleabbott]

Here is hq task info for a couple of randomly selected tasks:

From new job:
+--------------+----------------------------------------------------------------------------------+
| Task ID      | 24039                                                                            |
| State        | RUNNING                                                                          |
| Resources    | cpus: 1 compact                                                                  |
| Priority     | -1                                                                               |

From earlier submitted job:
+--------------+-------------------------------------------------------------------------------+
| Task ID      | 42                                                                            |
| State        | WAITING                                                                       |
| Resources    | cpus: 1 compact                                                               |
| Priority     | 0                                                                             |

the JDFs are very simple:

[[task]]
id = 24039
command = ["bin/foo", "--bar", ...]
cwd = "/path/to/stuff"
priority = -1

[task.env]
ENV_VAR_A = "1"
ENV_VAR_B = "1"

And the priority == 0 job above looks the same, except the "priority" line is omitted from the JDF.

[spirali]

I was able replicate some problems with priorities in v0.25. I will look on it if it make sense to fix it or wait for new scheduler that complete redo scheduler (it should be merged in few days).

[kyleabbott]

I believe at least one cause of lower priority tasks being ran while there are higher priority tasks waiting is in the case that there are multiple workers that all have more tasks assigned to them than they have CPUs. If my understanding is correct, the existing scheduler does not invoke the rebalance code path if there is not at least a single underloaded worker. In this case it's possible for the higher priority tasks to be assigned to a subset of workers, while other workers only have lower priority tasks assigned. I am not sure if this 100% explains the issue I have been seeing, but I believe it at least contributing... In one case I noticed only a small constant number of higher priority tasks running, which happened to exactly match the number of CPUs of a single worker. Upon inspection indeed all of that job's tasks were assigned to that single worker for some reason, while the other workers were packed with lower priority tasks. No rebalancing was taking place because all workers were overloaded.

Not sure how the new scheduling algorithm would deal with this scenario?

[spirali]

You are right that something like this is probably happening and this was the source of main problems with priorities and all changes was mostly about patching specific situations.

Nevertheless, I have merged new scheduler in the main (#1080), so in tomorrows nightly it should be there. Can you please check if your problem persists?

The new scheduler completely removes the concept of underloaded/overloaded workers and basically do targeted scheduling to workers (it still supports prefilling tasks to workers, but only when it is safe wrt priorities and proactively retracts tasks when a high priority task is submitted later). I am now writing a paper about the new scheduler, so it will be hopefully documented one day:)

[kyleabbott]

You are right that something like this is probably happening and this was the source of main problems with priorities and all changes was mostly about patching specific situations.

Nevertheless, I have merged new scheduler in the main (#1080), so in tomorrows nightly it should be there. Can you please check if your problem persists?

The new scheduler completely removes the concept of underloaded/overloaded workers and basically do targeted scheduling to workers (it still supports prefilling tasks to workers, but only when it is safe wrt priorities and proactively retracts tasks when a high priority task is submitted later). I am now writing a paper about the new scheduler, so it will be hopefully documented one day:)

I will test the priorities with the new scheduler, and I look forward to reading the paper :)