Expose scheduler/worker creation from Job

[victoronascimento]

Hi @maxcountryman !

I am trying to separate job enqueuing/scheduling from processing according to different environments. It would be really helpful to have some utility functions here to ease this process. I can work on them if the design seems ok to you!

Since Job has all its attributes private to the module, the only way to create a worker or scheduler is to either keep a reference to the queue used in the job creation (since I want DLQ configured) OR expose some kind of new_scheduler and new_worker that does not "run" it.

Currently, I need to keep a job reference in a global state (like Axum state) so that I can enqueue/schedule jobs from other parts of the code. But the place where initialization happens and where I have a reference for the Queue are kind of distant as to say. And the place where I start running workers/schedulers is where I have the same CancellationToken and shutdown signal handlers. So, it would facilitate a lot to have these new_worker and new_scheduler there without having to keep queues some in the global state.

IMHO, this would also make other parts of the code cleaner (like in JobHandle where you cannot reuse this creation).

If that is okay I can open a PR.

Thanks for this incredible work!

[maxcountryman]

Could you mock up what you have in mind in a little more detail? Maybe some code snippets of the proposed API would help. 🙂

[victoronascimento]

Yeah of course! I think this would be as easy as having:

    pub fn new_scheduler(self) -> Scheduler {
        let queue = self.queue.clone();
        let job = self.clone();
        Scheduler::new(queue, job)
    }

    pub fn new_worker(self) -> Worker {
        let queue = self.queue.clone();
        let job = self.clone();
        Worker::new(queue, job)
    }

Just to make this clearer...

I have a crate named core where my domain logic is coded. There, there is no "server" or "web". I consider jobs here as part of the core logic of a bounded domain, say, users. Jobs are defined and tested in this user "namespace" but should be started and ran together with a server (that only provides administrative tasks such as readiness checks, monitoring and so on).

So, I have something like: users::jobs::init(&pool); that builds up jobs. In the "web" crate I'd like to start workers/schedulers so that they share the same cancellation token to be called in the server shutdown signal (not sure if this is helping or making it harder to understand...).

Without being able to build workers and schedulers only from a job instance, I have to make users::jobs::init(&pool) return something like a tuple with the Queue and the Job instance so that I can call Worker::new(queue.clone(), job.clone).

With these new functions I could simply:

    // Only the job is needed at this point
    let mut task_worker = job.new_worker();
    tokio::task::spawn(async move { task_worker.run().await });

Maybe I missed something here? Does it make sense?

[maxcountryman]

I feel like at one point there were From impls for Worker and Scheduler, but I might be misremembering.

I'm not sure the extra methods would add much beyond From unless I'm missing something.

So then you could just write:

let mut task_worker: Worker = job.into();
tokio::task::spawn(async move { task_worker.run().await });

[victoronascimento]

Hmmm you are right (I am still not used to these common traits and keeps forgeting them...).

I will try this approach in a PR!

[maxcountryman]

Closed via #100.