Improved control over queue usage

[mattias-p]

Backend has a feature to put jobs into different queues. Each zm-testagent instance is configured to process jobs in some queues and ignore others.

Today job_create and batch_create each has a queue parameter. We do not validate queue arguments actually matches a queue the administrator intends to be used. In practice we just expect everyone to use the default queue unless told otherwise. There's no way to make sure users don't put jobs in queues with special purposes or where they otherwise don't belong.

I propose we simplify the queue selection in the following way.

• Add a configuration property RPCAPI.default_job_queue accepting an integer, defaulting to 0.
• Add a configuration property RPCAPI.default_batch_queue accepting an integer, defaulting to 0.
• Add a configuration property RPCAPI.additional_job_queues accepting a set of integers, defaulting to the empty set.
• Add a configuration property RPCAPI.additional_batch_queues accepting a set of integers, defaulting to the empty set.
• Make job_create validate its queue arguments against RPCAPI.default_job_queue and RPCAPI.additional_job_queues.
• Make batch_create validate its queue arguments against RPCAPI.default_batch_queue and RPCAPI.additional_batch_queues.

Edit history

• 2023-07-23: Issue created.
• 2023-11-17: Minor clarification.
• 2024-02-12: Update problem statement and design proposal taking into account what I've learned from discussing with @matsstralbergiis and @matsduf.

Stabilization

Tracking issue

Affected interfaces

Additions:

• RPCAPI.default_job_queue
• RPCAPI.default_batch_queue
• RPCAPI.additional_job_queues
• RPCAPI.additional_batch_queues

Compatibly changed:

• job_create
• batch_create

Dependencies

None

Overlapping proposals

None

[mattias-p]

After #1121 and maybe #1120, do we even need multiple queues for anything? Since several testagents now get along on the same queue, can't everybody just always put everything in the same queue and give high priority to singular jobs (from job_create) and low priority to batch jobs (from batch_create)? Is there another use case for the queues?

I'm nominating an additional option:

3. Remove the queue parameter from job_create and batch_create, and remove the ZONEMASTER.lock_on_queue configuration parameter.

Edit: I've come to realize that there are indeed other user cases for queues.

[matsstralbergiis]

At the moment I am using the queue to differ between different kinds of jobs. Our mass testing of the se-zone will probably run on another server (very expensive AWS instance) than the primary one (cheap instance in AWS) we are using for the interactive testing from the webserver.
If the primary server is allowed to take jobs from the same queue we will probably need to choose another instance type to prevent the server running out of resources.
At the moment I would prefer that we keep the queue handling.

[matsduf]

If you have one database for both GUI tests and batch tests and both are added to the same queue, then I think that the tests added to the batch will be processed before any tests from GUI that were added after the batch. Yes, we still want queues.

[mattias-p]

If you have one database for both GUI tests and batch tests and both are added to the same queue, then I think that the tests added to the batch will be processed before any tests from GUI that were added after the batch.

Not if the singular jobs are prioritized over batch jobs. This is the default case since default job_create sets priority to 10 and batch_create sets it to 5.

There is an argument to be made around latencies though. Let's say there are 20 workers are processing a single queue prioritizing GUI jobs, that a job takes on average 100 seconds. When a GUI job is added while all the workers are in the middle of processing a large batch, I believe the expected waiting time for the GUI job to get started is 100/20=5 seconds. The worst case is of course when a GUI job is added to the queue right after all workers get busy at the same time with a large batch (>=20 jobs). In this case the expected waiting time for the GUI job to start is close to 100 seconds (a little less depending on the distribution of how long it takes to process a job).

If such latencies are unacceptable then I agree that we need multiple queues just to be able to reduce latencies.

[mattias-p]

At the moment I am using the queue to differ between different kinds of jobs.

If we want to make reports on a certain kinds of jobs, then we need some way to tag jobs. A queue could be such a tag.

Our mass testing of the se-zone will probably run on another server (very expensive AWS instance) than the primary one (cheap instance in AWS) we are using for the interactive testing from the webserver. If the primary server is allowed to take jobs from the same queue we will probably need to choose another instance type to prevent the server running out of resources.

One possibility would be to have a cheap AWS instance for the base system with a single queue prioritizing the interactive tests and a testagent with only a few workers. When it's time for mass testing we power up the expensive AWS instance and start adding batch jobs to the single queue. At this point the cheap testagent would start helping out with the batch as best it can, but the expensive test agent would help out and even prioritize the interactive tests.

I don't think that would cause any resource exhaustion in the cheap AWS instance, other than the latency problem I described in my comment just above this one.

Another negative consequence is that the mass testing could take slightly longer if there is lots traffic from the GUI during the mass testing. I don't know if this is significant or negligible.

[mattias-p]

I just realized yet another reason to use multiple queues. If you want to make one test from each of a set of geographical locations. Then you must be able to control which testagent takes on which job.

[mattias-p]

@matsstralbergiis wrote:

Our mass testing of the se-zone will probably run on another server (very expensive AWS instance) than the primary one (cheap instance in AWS) we are using for the interactive testing from the webserver. If the primary server is allowed to take jobs from the same queue we will probably need to choose another instance type to prevent the server running out of resources.

I replied:

One possibility would be to have a cheap AWS instance for the base system with a single queue prioritizing the interactive tests and a testagent with only a few workers. When it's time for mass testing we power up the expensive AWS instance and start adding batch jobs to the single queue. At this point the cheap testagent would start helping out with the batch as best it can, but the expensive test agent would help out and even prioritize the interactive tests.

I don't think that would cause any resource exhaustion in the cheap AWS instance, other than the latency problem I described in my comment just above this one.

Based on the above I had an offline discussion with @matsstralbergiis. He explained to me that the cheap AWS instance will exhaust its execution budget if it helps out with the mass testing, and of course we can't have that. Multiple queues would fulfill this requirement.