Implement Job Manager to orchestrate background jobs #28
Description
We need to implement a robust Job Manager to orchestrate background processing tasks within the Icegate system.
While the core library structures exist, we need a cohesive Manager component capable of handling different types of jobs. The primary driver for this requirement is the need to run the Shifter Job (a separate task which will handle moving data from WAL to Iceberg), but the Manager must be designed to be generic and extensible.
Key Requirements:
- Generic Orchestration: The Job Manager must be able to register, schedule, and execute various job types without being hardcoded to a specific logic.
- Extensibility: It should serve as a platform for multiple job kinds (e.g., Shifter, Compaction, Retention) using a unified
JobDefinitionandJobRegistryapproach. - Lifecycle Management: Handle the full lifecycle of jobs: creation, state persistence (Started, Running, Completed, Failed), and failure recovery.
- Integration: Provide clear interfaces for defining new job types (like the upcoming Shifter job) so they can be easily plugged into the system.
- State: The Job Manager must be stateless, and the entire state must be stored in S3.
This task focuses on the infrastructure (the Manager) required to run the Shifter job, ensuring the system is ready to support this and future data processing workloads.
Metadata
Metadata
Assignees
Labels
No labels