Skip to content

feat(taskprocessing): add worker command for synchronous task processing#59015

Open
Copilot wants to merge 8 commits intomasterfrom
copilot/add-taskprocessing-worker-command
Open

feat(taskprocessing): add worker command for synchronous task processing#59015
Copilot wants to merge 8 commits intomasterfrom
copilot/add-taskprocessing-worker-command

Conversation

Copy link

Copilot AI commented Mar 17, 2026

Adds a taskprocessing:worker CLI command that runs a dedicated loop processing tasks via synchronous (ISynchronousProvider) providers, with starvation prevention via a single cross-type getNextScheduledTask query.

Summary

New command: occ taskprocessing:worker

  • Polls for and processes tasks from all preferred synchronous providers
  • --timeout <s>: exit after N seconds (default: run forever)
  • --interval <s>: sleep between idle polling cycles (default: 1s)
  • --once: process at most one task then exit
  • --taskTypes <id>: restrict to specific task type IDs (repeatable)

Starvation prevention

Instead of shuffling providers and issuing one getNextScheduledTask call per provider, all eligible task type IDs are collected first and passed in a single query. This guarantees the globally oldest scheduled task is always picked next, regardless of per-provider queue depth:

// Build eligible taskTypeId → provider map for all preferred sync providers
$eligibleProviders = [];
foreach ($providers as $provider) { ... }

// One query across all types — picks oldest task, preventing starvation
$task = $this->taskProcessingManager->getNextScheduledTask(array_keys($eligibleProviders));
$provider = $eligibleProviders[$task->getTaskTypeId()];
$this->taskProcessingManager->processTask($task, $provider);

Test fixes

  • createTask() helper now accepts an optional task type so task type IDs match provider expectations
  • testProcessesFirstMatchingProvidertestProcessesCorrectProviderForReturnedTaskType: updated mock expectations to reflect that getPreferredProvider is called for every eligible provider and getNextScheduledTask receives all eligible type IDs
  • Added testPicksOldestTaskAcrossMultipleEligibleProviders: verifies that with two eligible providers the task returned by getNextScheduledTask determines which provider runs

TODO

  • ...

Checklist

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI changed the title [WIP] Add OCC command for task processing worker Add occ taskprocessing:worker dedicated synchronous task processing command Mar 17, 2026
Copilot AI requested a review from marcelklehr March 17, 2026 18:37
Copilot AI changed the title Add occ taskprocessing:worker dedicated synchronous task processing command Add --taskTypes whitelist option to taskprocessing:worker Mar 17, 2026
Copilot AI changed the title Add --taskTypes whitelist option to taskprocessing:worker taskprocessing:worker: add --taskTypes whitelist option Mar 17, 2026
@marcelklehr marcelklehr changed the title Fix provider starvation in WorkerCommand::processNextTask Implement occ taskprocessing:worker command Mar 18, 2026
Copilot AI changed the title Implement occ taskprocessing:worker command Fix task type starvation in WorkerCommand::processNextTask Mar 18, 2026
@marcelklehr marcelklehr changed the title Fix task type starvation in WorkerCommand::processNextTask Implement occ taskprocessing:worker command to run tasks Mar 18, 2026
@marcelklehr marcelklehr requested a review from julien-nc March 18, 2026 13:52
Copy link
Member

@julien-nc julien-nc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay 👍 ❤️
Lgtm (didn't try it yet though). Will approve after trying it.

@julien-nc julien-nc self-requested a review March 18, 2026 14:17
Copy link
Member

@julien-nc julien-nc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great

@szaimen
Copy link
Contributor

szaimen commented Mar 18, 2026

Hey, is this the implementation for #52629 or is this something else?

Copilot stopped work on behalf of marcelklehr due to an error March 18, 2026 17:23
@marcelklehr
Copy link
Member

@szaimen This is something else. We discovered that the way TaskProcessing\SynchronousBackgroundJob works has some flaws that prevent seamless parallelism. So, for taskprocessing we've opted to leave the background job realm entirely and create a new command that runs the tasks in the background. (The alternative, in our view, would be to have one job per task which seems excessive.)

#52629 is separate from this effort and I realize that this PR distances us further from a unified solution. I'll pick up #52629 in our team discussion next week.

Copilot AI changed the title Implement occ taskprocessing:worker command to run tasks feat(taskprocessing): add worker command for synchronous task processing Mar 18, 2026
@marcelklehr marcelklehr force-pushed the copilot/add-taskprocessing-worker-command branch 2 times, most recently from cdd1d26 to b6812e1 Compare March 19, 2026 08:40
Copy link
Contributor

@kyteinsky kyteinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Copilot AI and others added 7 commits March 19, 2026 11:08
Co-authored-by: marcelklehr <986878+marcelklehr@users.noreply.github.com>
…ing:worker command

Co-authored-by: marcelklehr <986878+marcelklehr@users.noreply.github.com>
Co-authored-by: marcelklehr <986878+marcelklehr@users.noreply.github.com>
…uffling providers

Co-authored-by: marcelklehr <986878+marcelklehr@users.noreply.github.com>
…ick the oldest task

Co-authored-by: marcelklehr <986878+marcelklehr@users.noreply.github.com>
…tion-prevention test

Co-authored-by: marcelklehr <986878+marcelklehr@users.noreply.github.com>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
@marcelklehr marcelklehr force-pushed the copilot/add-taskprocessing-worker-command branch from b6812e1 to ad5e709 Compare March 19, 2026 10:08
Co-authored-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants