Implement Shifter Job for WAL to Iceberg Data Movement #29
Description
We need to implement a specific job type called Shifter, which is responsible for moving data from the Write-Ahead Log (WAL) to Iceberg tables.
This job will run within the JobManager infrastructure (defined in issue #28) and is critical for the system's data ingestion pipeline.
Key Objectives:
-
Job Implementation:
- Create a new job definition (e.g., with code
SHIFTER) that utilizes the existingJobDefinitionstructure. - Implement the core logic to read batched data from the WAL (S3/storage).
- Transform and write this data into the target Iceberg tables.
- Create a new job definition (e.g., with code
-
Task Executors:
- Define specific
TaskExecutorFns required for the Shifter workflow (e.g.,read_wal_segment,write_to_iceberg,commit_transaction). - Ensure tasks are idempotent where possible to handle retries gracefully.
- Define specific
-
Integration:
- Register the Shifter job with the
JobRegistryso it can be scheduled by theJobsManager. - Ensure configuration options are available (e.g., batch sizes, target table configurations).
- Register the Shifter job with the
Technical Context:
- This implementation will likely reside in a new module or crate (e.g.,
icegate-shifteror part oficegate-ingest) but must implement the traits defined inicegate-jobmanager. - It should leverage the
icegate-commonschemas andicegate-storageabstractions.
Metadata
Metadata
Assignees
Labels
No labels