Skip to content

[tiering] Expose the tiering reader waiting ratio - #4786

Open
beryllw wants to merge 1 commit into
apache:mainfrom
beryllw:tiering/readers-awaiting-split-ratio
Open

beryllw wants to merge 1 commit into
apache:mainfrom
beryllw:tiering/readers-awaiting-split-ratio

Conversation

@beryllw

@beryllw beryllw commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Purpose

Closes #4782.

Expose the proportion of tiering readers waiting for splits to help identify spare reader capacity.

Brief change log

Add waitingReadersRatio under fluss.tieringService, calculated from the existing waiting-reader set and source parallelism.

Tests

Extend existing scheduling tests to cover assignment, waiting, generation failure, and reader re-registration. All 18 targeted enumerator and metrics tests passed.

API and Format

No Java API or storage format changes.

Documentation

Document the metric and its startup, failover, and rescaling semantics.

Generated-by: OpenAI Codex

@beryllw beryllw changed the title [flink] Expose the tiering reader waiting ratio [tiering] Expose the tiering reader waiting ratio Oct 21, 2026
@beryllw
beryllw force-pushed the tiering/readers-awaiting-split-ratio branch from 680f041 to ca80e55 Compare October 21, 2026 14:04
@beryllw
beryllw requested a review from luoyuxia October 21, 2026 14:05

@zuston zuston left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, @beryllw

Quick question: are we planning to use this metric to drive autoscaling for the Flink tiering job?

@beryllw

beryllw commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Quick question: are we planning to use this metric to drive autoscaling for the Flink tiering job?

Yes, plan to use it alongside tiering latency as an additional autoscaling signal. When latency is high but readers are consistently waiting for splits, adding parallelism may not help. This metric would help the autoscaler avoid unnecessary scale-ups in those cases.

@beryllw
beryllw requested a review from zuston October 21, 2026 15:54
@beryllw
beryllw force-pushed the tiering/readers-awaiting-split-ratio branch from ca80e55 to a066fec Compare October 22, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[flink] Expose the tiering reader waiting ratio

2 participants