Skip to content

[#13760] feat(tms): Wire in-job REST, expand-cron-commit path, and Iceberg Spark job - #14223

Merged
roryqi merged 10 commits into
apache:branch-tms-schedulerfrom
lasdf1234:feature/tms-ab-stack-mega
Oct 26, 2026
Merged

roryqi merged 10 commits into
apache:branch-tms-schedulerfrom
lasdf1234:feature/tms-ab-stack-mega

Conversation

@lasdf1234

@lasdf1234 lasdf1234 commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Follow-up on the TMS scheduler pools from Epic #13760. Targets branch-tms-scheduler.

This PR stacks the remaining in-job and expand/cron/commit slices (one commit per subtask):

Why are the changes needed?

In-job Spark needs HTTP hooks for metrics and soft-slot release. Expand must choose a nearest maintenance policy per table and turn timeTrigger into cron work. Commit-chain maintenance must enqueue from IRC table updates. Operators need a built-in Iceberg maintenance job that reports metrics and releases occupancy.

Fix: #14205
Fix: #13984
Fix: #14215
Fix: #14216
Fix: #14217
Fix: #14020
Fix: #14218
Fix: #14219
Fix: #14220
Fix: #14221
Fix: #14222

Parent epic: #13760

Does this PR introduce any user-facing change?

Yes:

  • Maintenance action REST: metrics/before, metrics/after, and finish
  • Expand / cron / commit scheduling behavior under plugins/maintenance
  • Built-in Iceberg maintenance Spark job template and related jobConf shape
  • Module path rename plugins/tms → plugins/maintenance

How was this patch tested?

  • Unit tests: ./gradlew :plugins:maintenance:test -PskipITs -PskipDockerTests=true
  • Plugin ITs under org.apache.gravitino.maintenance.integration.test (REST, expand→cron, IRC→commit) via path-filtered TMS Test / cron suite tms

@lasdf1234 lasdf1234 changed the title [#13760] feat(tms): In-job metrics/finish, expand-cron-commit stack, and Spark job [#13760] feat(tms): Wire in-job REST, expand-cron-commit path, and Iceberg Spark job Oct 26, 2026
@lasdf1234
lasdf1234 force-pushed the feature/tms-ab-stack-mega branch from b00587f to aa2b325 Compare October 26, 2026 02:28
@lasdf1234
lasdf1234 force-pushed the feature/tms-ab-stack-mega branch from aa2b325 to 4f4ecec Compare October 26, 2026 02:40
@lasdf1234 lasdf1234 self-assigned this Oct 26, 2026
@lasdf1234
lasdf1234 requested a review from roryqi October 26, 2026 02:41
@lasdf1234

Copy link
Copy Markdown
Collaborator Author

lasdf1234#73
lasdf1234#75
lasdf1234#76
lasdf1234#77
lasdf1234#78
lasdf1234#79
lasdf1234#80
lasdf1234#81

The review has been conducted on the PR in the private repository.

lasdf1234 and others added 3 commits October 26, 2026 12:01
Fixes apache#14221.

Squashed from fork #77 (B10).

Co-authored-by: Cursor <cursoragent@cursor.com>
…ce and add plugin ITs

Add plugins/maintenance CI (TMS Test workflow + cron suite) modeled on
idp-basic, and deepen the action-callback REST IT with a metrics/before →
after?finish happy path (embedded).

Fixes apache#14222.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ue coverage

Exercise live scheduler pools / EventBus with an embedded Iceberg catalog.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234
lasdf1234 force-pushed the feature/tms-ab-stack-mega branch from ebda5e2 to 58f7455 Compare October 26, 2026 04:05
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Code Coverage Report

Overall Project 71.66% -0.02% 🟢
Files changed 70.94% 🟢

Module Coverage
aliyun 19.74% 🔴
api 54.37% 🟢
authorization-common 85.96% 🟢
authorization-ranger 14.44% 🔴
aws 53.54% 🟢
aws-bundle 0.0% 🔴
azure 32.1% 🔴
azure-bundle 0.0% 🔴
catalog-common 29.49% 🔴
catalog-fileset 82.37% 🟢
catalog-glue 73.24% 🟢
catalog-hive 83.63% 🟢
catalog-jdbc-common 57.98% 🟢
catalog-jdbc-doris 87.56% 🟢
catalog-jdbc-mysql 81.91% 🟢
catalog-jdbc-postgresql 84.02% 🟢
catalog-jdbc-starrocks 79.16% 🟢
catalog-kafka 75.21% 🟢
catalog-lakehouse-generic 84.58% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 85.93% 🟢
catalog-lakehouse-paimon 84.36% 🟢
catalog-model 77.99% 🟢
cli 44.62% 🟢
client-java 79.34% 🟢
client-java-runtime 0.0% 🔴
common 60.24% 🟢
core 87.46% +0.02% 🟢
filesystem-hadoop3 76.89% 🟢
flink 55.44% 🟢
flink-common 32.2% 🔴
flink-runtime 0.0% 🔴
gcp 32.2% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 17.84% 🔴
hive-metastore-common 62.3% 🟢
hive-metastore3-libs 58.8% 🟢
iceberg-aliyun-bundle 0.0% 🔴
iceberg-common 66.53% 🟢
iceberg-rest-server 77.08% 🟢
idp-basic 86.77% 🟢
integration-test-common 0.0% 🔴
jobs 62.01% -9.09% 🟢
k8s-job-executor 99.81% 🟢
lance-common 39.9% 🔴
lance-rest-server 69.51% 🟢
lineage 59.39% 🟢
maintenance 74.86% 🟢
optimizer 83.95% 🟢
optimizer-api 40.24% +4.12% 🟢
server 91.43% 🟢
server-common 83.85% 🟢
spark 61.79% 🟢
tencent 81.78% 🟢
trino-connector 63.31% 🟢
Files
Module File Coverage
core MaintenanceAuthConfigs.java 91.24% 🟢
jobs MaintenanceActionHttpClient.java 83.87% 🟢
BuiltInJobTemplateProvider.java 81.82% 🟢
MaintenanceTriggerGate.java 75.71% 🟢
IcebergMaintenanceJob.java 22.36% 🔴
maintenance AlwaysMaintainInJobRecommender.java 100.0% 🟢
InJobGateOutcome.java 100.0% 🟢
InJobRecommender.java 100.0% 🟢
MetricsJson.java 100.0% 🟢
RecommendDecision.java 100.0% 🟢
CollectedCommitChain.java 100.0% 🟢
CommitChainMember.java 100.0% 🟢
CommitJobUpsertResult.java 100.0% 🟢
TableSnapshotMetricsSQLProviderFactory.java 100.0% 🟢
MaintenanceMapperPackageProvider.java 100.0% 🟢
TableSnapshotMetricsBaseSQLProvider.java 100.0% 🟢
TableMaintenanceActionH2Provider.java 100.0% 🟢
TableSnapshotMetricsH2Provider.java 100.0% 🟢
TableMaintenanceActionPostgreSQLProvider.java 100.0% 🟢
TableSnapshotMetricsPostgreSQLProvider.java 100.0% 🟢
TableMaintenanceActionOperations.java 100.0% 🟢
TableMaintenanceInJobPipeline.java 95.65% 🟢
NearestMaintenancePolicyResolver.java 95.12% 🟢
PolicyExpandPipeline.java 93.44% 🟢
PolicyExpandReconciler.java 93.33% 🟢
TableMaintenanceActionSQLProviderFactory.java 93.1% 🟢
TableMaintenanceActionBaseSQLProvider.java 92.31% 🟢
CommitJobSpecFactory.java 90.36% 🟢
MaintenanceMetricsService.java 89.74% 🟢
MaintenanceJobConfBuilder.java 88.41% 🟢
TableMaintenanceConfig.java 88.31% 🟢
TableMaintenanceActionMetaService.java 87.91% 🟢
PendingCronExpandOccupancyGate.java 87.5% 🟢
OccupancyGate.java 86.96% 🟢
CommitAuthConsistency.java 86.49% 🟢
MaintenanceJobSpecFactory.java 84.21% 🟢
TableMaintenanceActionPO.java 83.33% 🟢
MaintenanceJobSpec.java 82.29% 🟢
TableMaintenanceSchedulerPools.java 82.18% 🟢
MaintenanceSparkSubmitPipeline.java 81.32% 🟢
TableMaintenanceInJobSampler.java 80.95% 🟢
PolicyExpandCrons.java 80.0% 🟢
NearestPolicyTableEnumerator.java 78.57% 🟢
RelationalCommitJobScheduler.java 77.33% 🟢
TypeDispatchingInJobRecommender.java 76.67% 🟢
PolicyExpandLifecycleHook.java 75.64% 🟢
CompactionInJobRecommender.java 75.0% 🟢
RelationalCronJobScheduler.java 74.38% 🟢
MaintenanceJobTemplates.java 72.73% 🟢
TriggerExprInJobRecommender.java 71.43% 🟢
OccupancyClaim.java 71.43% 🟢
ExpandTableCandidate.java 70.59% 🟢
PolicyExpandRegistrar.java 70.0% 🟢
CronJobIds.java 67.31% 🟢
MaintenanceCommitSubmitPipeline.java 65.85% 🟢
TableMaintenanceRESTUtils.java 60.87% 🟢
TableSnapshotMetricsMetaService.java 56.67% 🔴
AutocommitDataSources.java 51.61% 🔴
PostCommitMaintenanceListener.java 40.68% 🔴
TableSnapshotMetricsPO.java 40.0% 🔴
CommitOccupancyGate.java 33.33% 🔴
TableMaintenanceRESTFeature.java 23.81% 🔴
MaintenanceTableRefs.java 12.5% 🔴
MaintenanceJobSubmitter.java 11.11% 🔴
SnapshotHead.java 0.0% 🔴
CommitChainCollector.java 0.0% 🔴
CommitJobScheduler.java 0.0% 🔴
CronJobScheduler.java 0.0% 🔴
CronSubmitGate.java 0.0% 🔴
ExpandOccupancyGate.java 0.0% 🔴
PolicyTableEnumerator.java 0.0% 🔴
MetalakeNames.java 0.0% 🔴
TableMaintenanceActionMapper.java 0.0% 🔴
TableSnapshotMetricsMapper.java 0.0% 🔴
TableMaintenanceBinder.java 0.0% 🔴
optimizer-api GravitinoAuthSettings.java 71.79% 🟢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants