Job scheduling efficiency improvements

[machshev]

The nightly and weekly regressions for OpenTitan take significant resources to run. It is helpful to have more flexibility in the scheduler so that we can make better use of those resources.

Some ideas:

• Test ordering - for the nightly regression runs most of the tests are complete in the first hour or so. By starting the longest running tests first and infilling with shorter running tests we can make the overall regression run time shorter, but keeping our scheduling slots busy.
• Split run schedule
  • Generate a schedule and save to disk
  • Restore the schedule and run it
  • Restore the schedule and run a subset - potentially on a different machine
  • Restore a schedule and based on the working files resume a WIP run
• More refined max parallel job control
  • Tool specific max parallel - different tools have different resource requirements so it's appropriate to have different limits proportional to the resource usage
  • Time based job control - it's common to run tests overnight/weekends when there may less contention of resources (compute or licenses). However jobs may have to be stopped if they don't finish within the expected timeslot if they are starving resources from other users. It would be useful to have the ability to set time dependent resource usage limits, so big jobs can be started any time and make use of the available resources at the time.
  • License aware scheduling - periodically query the license server so that the number of parallel jobs can be dynamically changed to reflect the currently available licenses. Avoiding starving other users, while making more efficient use of licenses.
  • General resource aware scheduling - periodically poll an external service to know how much resource is required for each job. This mechanism could be used for license server query, but also potentially other things like the machines memory and CPU usage.