Setup Distributed Background Worker Infrastructure

[Priyanshu1-62]

Overview

Right now, all processes are handled by single main server. As a result, either end-user has to wait for task completion which depends on external service providers, or that task will be executed asynchronously within the same process. This makes it difficult to handle retries, scaling, distributed execution, process crashes.

Description

Setup a worker service to handle asynchronous workflow and distributed execution.

Tasks

• Core infra: Setup standalone worker service with TS config, graceful shutdown handling, shared config management, docker config setup.
• Queue infra: Setup Redis connection, implement BullMQ, queue abstraction layer, queue health monitoring.