Avoid abuse of priority parameter

[mattias-p]

Today the job_create method has a priority parameter. Any user can set any priority they like and we trust users not to abuse this. There is no penalty if they do.

The priority feature is needed on single-machine Backend installations with both a GUI and a batch client. A job from the GUI ought to be processed as soon as possible instead of waiting in line for the entire batch to finish.

My current understanding is that singular jobs (from job_create) need to run at a higher priority than batch jobs (from batch_create), but that finer granularity than this is not required.

I could think of the these options to improve the situation:

1. Remove the priority parameter from the job_create and batch_create methods. Let these use methods always create jobs with their respective "default" priorities.
2. Configure priorities for "singular" and "batch" jobs per RPCAPI instance.
3. Add an API key parameter to job_create and set priority per API key. Perhaps there could be a default priority for the API key is omitted.

Edit: At first I didn't realize the importance of the priority feature. I have since updated the description with my new understanding.

Stabilization

Tracking issue

Affected interfaces

Incompatibly changed:

• job_create
• batch_create

Dependencies

None

Overlapping proposals

None

[mattias-p]

Talking to @matsstralbergiis we came up with another option:

4. Configure priority per queue.

   This options is superior ot both options 1 and 2 above.

   1. Instead of having job_create and batch_create default to creating jobs at different priorities in the same queue, they could default to creating jobs in different queues. If clients need to be able to create jobs at different priorities, different queues could be used for different priority levels instead.
   2. Instead of having different default priorities for job_create and batch_create on different zm-rpcapi instances, they could have different default queues on different zm-rpcapi instances.

   If we do the prioritization based on queues we should remove the test_results.priority field from the database. The behavior of the system would be easier to reason about this way.