Add support for Work Queues

[ameyapathare]

It would be very helpful to be able to setup/configure our work queues via terraform. Right now, we create an 'on-demand', 'priority', and 'default' work queue for each work pool. We are also contemplating creating a separate work queue per data source, which would involve setting up hundreds of work queues.

Work queues are the last remaining resource before we switch over to using this terraform provider to provision our prefect infrastructure!

[parkedwards]

@jamiezieziula i've assigned this to you to take a look

at a high level, here are some pointers that should you get started:

• Contributing guide has environment setup steps

• Think of this provider as a jazzed-up API client, so there will be code to add in the /internal/api section of this repo to set up a new section that interacts with the /work_pools/{pool_name}/queues API. This is the best starting point - we're setting up the interface between the provider and the Prefect Cloud API

  • In /internal/api, we'll set new struct data models for the new client (eg. WorkQueuesClient), the representation of the resource (eg. WorkQueue), and optionally the create/update payloads (eg. WorkQueueCreate or WorkQueueUpdate). See this existing example for WorkPool
  • In /internal/client, we'll set the actual API methods that correspond with GET POST DELETE etc. See this existing example for WorkPool - customarily, we'll create the List, Get, Create, Update, and Delete methods -- essentially, CRUD + a List method (which hits POST /filter)
  • Be sure to add your new client interface into the root level PrefectClient interface

• when looking at our API schema, you might see 2 different "groups" of work-queue API endpoints. for this provider, we'll want to use the one that is a part of the /work_pools/{pool_name}/queues API (so the one always associated with a particular work pool) and NOT the one inside of the /work_queue API (this is for the old agent workloads, which is on its way out)

• As far as the actual Terraform resources, we'll want to support datasource, resource, and import functionality for this new Work Queue object type. See the matrix here.

  • datasource allows querying existing resources, eg. data "some_object" "my_object_name" {}, so you can read down attributes (like the id or others)
  • resource allows fully managing the resource in Terraform
  • import allows linking existing resources into the TF lifecycle

• The way we structured this repo, you'll have to set up a separate:

  • datasource in the /internal/provider/datasources directory, in a work_queue.go file
  • resource in the /internal/provider/resources directory, in a work_queue.go file
  • import support goes into the associated resources file, inside of a reserved ImportState method on the resource class/struct. See this example for a service_account import, via a commonly used pattern where we allow passing a resource ID or a resource name via the prefix name/<your resource name here>

• For datasource and resource configurations in the code - the top level function methods are special + reserved, so they represent the logic that the Terraform provider framework will expect + call. You can pull those over from an existing, corresponding datasources/<name>.go / resources/<name>.go. The logic to update will predominantly be:

  • The Schema methods + the *Model structs
  • The Read method, as we'll be calling the client methods you defined first + have the flexibility to perform certain kinds of checks

• Finally, there are root-level configurations where you'll need to add your newly created datasources.NewWorkQueueDataSource and resources.NewWorkQueueResource

[ondramie]

hi, what's up with this? seems kinda closed, kinda open?

[mitchnielsen]

Thanks for reaching out @ondramie, it's definitely open and part of our milestone for parity with the Prefect API. We're working on prioritizing that work and getting more of our team up to speed on the provider to speed up development.

[ondramie]

hi, @mitchnielsen, so its it open for anybody to tackle?

[mitchnielsen]

@ondramie definitely, we haven't started on this one yet so if you're interested that'd be great!

[nick-amplify]

Hey @mitchnielsen, thinking about tackling this. Has anyone else already started?

[mitchnielsen]

That sounds great @nick-amplify, appreciate the contribution 🤝

[nick-amplify]

@mitchnielsen I've got everything working except for the tests!

For tests like internal/provider/datasources/work_pool_test.go, it looks like you create a new workspace with workspace := testutils.NewEphemeralWorkspace() which is then passed as the workspace for test objects.

You can't do this for prefect local server, my personal accounts already have the 1 allotted workspace, and our paid accounts only allow for the dev/prod workspaces we are paying for.

Is there a way that I am missing that I could create new workspaces in order to run the tests for this repo?

[mitchnielsen]

Great question, we should probably put some more thought into this and update our contributing guide.

For now, you can write the tests (with the ephemeral workspace logic) and push them to your branch, and then someone from the Prefect team can 'approve' the CI job that runs the tests in our own Prefect Cloud infrastructure. It's a slightly longer feedback loop compared to being able to trigger them on your own before pushing code, but it should do the trick for now.

EDIT: created #354 to clarify some of this.