[EPIC] TMS: Persistence, scheduler, and in-job Spark path #13760

Describe the proposal

Implement the persistence and in-job execution foundation for the in-process Table Maintenance Service (TMS) as a plugins/tms module (same SPI / mapper packaging pattern as idp-basic, not in core), including the policy-content schema previously tracked under #13764.

This epic delivers:

• Typed TMS execution peers on Iceberg maintenance policy content (timeTrigger, commitTrigger, minIntervalMs, jobOptions, updaterOptions) with validation, SerDe / REST round-trip, docs, and nearest-policy / minIntervalMs resolution
• Relational DDL and DAOs for scheduled_tasks (db-scheduler), table_maintenance_job, and table_snapshot_metrics
• Three db-scheduler pools (expand / cron / commit), soft-slot occupancy claim storage, REST Feature lifecycle, and OccupancyGate
• In-Spark Job gates: sample metrics at snapshot S0, snapshot-drift early exit, and recommender maintain/skip
• Cluster-static gravitinoAuth / ircAuth bags from gravitino.conf, overlaid at runJob — not stored in policy content
• Execution of four maintenance types (compaction, manifest-rewrite, snapshot-expiry, orphan-cleanup) inside Spark Jobs
• Commit-chain runs with Iceberg OCC (validateFromSnapshot) and validation metrics

Design: apache/gravitino#13386. Related: apache/gravitino#13488. Policy-content epic #13764 is merged into this list and closed.

Task list

Ordered by dependency: policy content/schema first, then persistence/scheduler, then in-job path.

Policy content schema (from #13764)

• [Subtask] TMS: Policy content parse, validation, and REST round-trip #13765 — Built-in maintenance policy type placeholders (narrowed in [#13765] feat(policy): Add TMS policy content fields and REST round-trip #13766)
• [Subtask] TMS: Domain PolicyExecutionSpec and executable peers #13800 — Domain: PolicyExecutionSpec + executable peers (PR #13885)
• [Subtask] TMS: Flat peer SerDe, DTO round-trips, and minIntervalMs defaults #13888 — Flat peer SerDe, DTO round-trips, minIntervalMs defaults (PR #13887)
• [Subtask] TMS: OpenAPI / user docs for content peers #13815 — OpenAPI / user docs for TMS content peers (PR #13814)
• [Subtask] TMS: Nearest-policy selection and minIntervalMs resolution #13779 — Nearest-policy selection and minIntervalMs resolution (PR #13817)

Persistence, scheduler, and in-job Spark path

• [Subtask] TMS: plugins/tms skeleton, DDL, and DAOs #13761 — plugins/tms skeleton, DDL, and DAOs
• [Subtask] TMS: Scheduler pools and soft-slot occupancy claim #13773 — Scheduler pools and soft-slot occupancy claim (PR #13774)
• [Subtask] TMS: Bind policy timeTrigger to the expand scheduler #13983 — Bind policy timeTrigger to expand scheduler (PR #13897)
• [Subtask] TMS: Nearest-policy resolver and expand table enumerator #13984 — Nearest-policy resolver and expand table enumerator (PR #14055)
• [Subtask] TMS: Cron-pool Spark submit (control plane) #14020 — Cron-pool Spark submit (control plane) (PR #14019)
• [Subtask] TMS: In-job S0/drift/recommend gates and OccupancyGate REST Feature #13777 — In-job S0/drift/recommend gates + OccupancyGate / REST Feature lifecycle (PR #13841)
• [Subtask] TMS: In-job recommenders and terminal occupancy reclaim #14065 — In-job recommenders and terminal occupancy reclaim (PR #14054)
• [Subtask] TMS: Cluster-static gravitinoAuth / ircAuth overlay bags #13797 — Cluster-static gravitinoAuth / ircAuth overlay bags (#13865)
• [Subtask] TMS: In-job MaintenanceTypeHandler pipeline skeleton #13824 — In-job MaintenanceTypeHandler pipeline skeleton (PR #13825)
• Four maintenance types in Spark Job execution (fill handler execute hooks)
• Commit-chain, Iceberg OCC, and validation metrics