[server] Support DescribeTabletServers API for safe scaling and upgrades by morazow · Pull Request #3743 · apache/fluss

morazow · 2026-07-22T13:20:28Z

Summary

Adds a read-only DescribeTabletServers RPC to the Coordinator, following the pattern of GetClusterHealth ([server] Support Cluster Health API for safe rolling upgrades #3399 / [server] Add Cluster Health API implementation #3400).
New Admin.describeTabletServers() (@PublicEvolving) returns, per tablet server, the same four counters getClusterHealth() reports cluster-wide, scoped to that server: numReplicas, inSyncReplicas, numLeaderReplicas, activeLeaderReplicas.
Counters are computed from in-memory CoordinatorContext state on the coordinator event thread (via AccessContextEvent); tablet servers forward the request to the coordinator over the internal listener.
Live and shutting-down servers are always reported (even with zero replicas), and dead servers still holding assigned replicas remain visible, so a scale-in gate cannot treat them as drained.
This backs the Kubernetes Operator FIP's serverGreen(serverId) rolling-upgrade gate, the scale-in safety gate (numReplicas == 0), and per-server tablet load reporting.

Test Plan

New unit test DescribeTabletServersTest (fluss-server) covering: empty cluster, full ISR, out-of-ISR replicas, inactive leaders, leaderless buckets, evacuated servers, dead-but-assigned servers, partitioned tables, and reconciliation of per-server sums with computeClusterHealth.
New ITCase FlussAdminITCase#testDescribeTabletServersDuringRollingUpgrade: healthy cluster is green and sums match cluster health; a stopped server stays reported, non-empty and non-green; after restart all servers become green again.
mvn spotless:check clean on fluss-rpc, fluss-server, fluss-client, fluss-flink-common.

🤖 AI-assisted changes - reviewed by human developer

affo

Matches the expectations for unlocking a detailed description of the tablet server and assessing a server health on a per-server basis.

Thanks for this contribution!

affo

@morazow and the community.
I also noticed that there is no way as of now to list the tags of servers, I do wonder if would be the right place to do that in your opinion :)

morazow · 2026-07-31T08:46:50Z

@affo good idea! So you think it makes sense as follow up PR or fine to add here?

affo · 2026-07-31T12:46:33Z

@morazow I would directly include that in this PR, as semantically it would make sense as we "describe" tablet servers. But I would also like to have an additional feedback on this one as well 🤝

fresh-borzoni

@morazow Thank you for the PR!
Reading this alongside #1410 and #4061, the split looks like:
#1410 DescribeCluster for topology + ServerTag, getClusterHealth for aggregates, this one for per-server counters
#4061 the cache both health APIs read from.

If it holds, two things follow here: the tag commit is redundant with #1410, and both PRs currently claim ApiKey 1065 so one needs renumbering.

I'd hold the API-shape questions until that settles rather than have you redo work.

let me know when it settles, so I can help with API-shape

fresh-borzoni · 2026-08-21T10:38:13Z

+            assertThat(isServerGreen(server)).isTrue();
+        }
+
+        // The per-server counters must sum up to the cluster-wide health counters.


A leaderless bucket is counted by cluster health but by no server here, so these sums can differ while every server still looks green. The waitUntil above doesn't rule that out. testPerServerSumsReconcileWithClusterHealth already covers this.
Do we need it?

affo · 2026-08-21T13:39:13Z

@fresh-borzoni at issue #1389 we conducted quite a big discussion also on the performance of this call given that it runs in the coordinator loop itself.

This led me to shape in draft a PR for an health cache that can be used by both cluster health and per-tablet health: #4061.

The cache uses a new mechanism thought for the performance updates, a new CoalescingRefreshCache class that embeds the logic for no-read-lock, but write lock, and enhances the logic to allow for different dirty states to optimize heavy updates on the coordinator side.

The same mechanism can be re-used across caches as well.

What would you guys think (@fresh-borzoni , @morazow )?

In case that one seems like a good approach we can:

rebase this one and wire the cache mechanism
on the second thing for server tags, yeah we already agreed on that issue that that should stay in getServerNodes, my only point was not to perform a direct ZK blocking call but to re-use the cache.

I re-read your comment above and realized you are aware of #4061 🤝 😅 , I misread at the beginning.
If you may also give feedback to that one now that you have the whole context it would be amazing 🚀

morazow · 2026-08-24T11:19:03Z

Thanks @affo , @fresh-borzoni for review!

It would make sense to wait for #4061 and rebase to use the new cache mechanism.

I'll also drop the serverTags commit from this PR 🤝

affo · 2026-10-21T12:30:29Z

@morazow as per #4061 (comment) this one is unblocked and does not need to wait 🤝

morazow · 2026-10-21T12:57:19Z

yeap, going to rebase and fix conflicts 🤝

polyzos · 2026-10-23T05:47:21Z

+        }
+
+        // tagged servers are being drained, so report them even when dead and empty
+        ctx.getServerTags()


As agreed in #1389 and earlier in this thread, server tags are moving to the server nodes API in #1410. Dropping the tag commit here keeps the two APIs from overlapping. That would also remove the tag field from the response and the client class, along with the two tag tests.

polyzos · 2026-10-23T05:50:56Z

+            assertThat(isServerGreen(server)).isTrue();
+        }
+
+        // The per-server counters must sum up to the cluster-wide health counters.


Following up on @fresh-borzoni's earlier comment, which is now outdated after the test moved: a bucket without a leader counts in cluster health but on no server, so these totals can differ even while every server looks healthy. The wait above doesn't rule that out, so this check could be flaky. The reconciliation unit test already covers this reliably, so dropping this block seems safest.

polyzos · 2026-10-23T05:52:06Z

+ *       numReplicas && activeLeaderReplicas == numLeaderReplicas}.
+ * </ul>
+ *
+ * <p>Note: a bucket without an elected leader is attributed to no server's {@code


This note explains why the per-server check isn't enough on its own: a bucket without a leader can leave every server that hosts it looking healthy. So the operator also has to ask for cluster health separately, and the two answers can come from different moments. As @fresh-borzoni suggested, computing both in one pass, or adding a per-server count of buckets without a leader, would let the operator decide from a single response. Since this is new public API, it's easiest to settle these things first.

polyzos · 2026-10-23T05:55:42Z

Thanks for the rebase @morazow, I left a few inline notes on the points from the earlier discussion that are still open: the server tags, a single consistent answer for the upgrade check, and the totals check in the rolling-upgrade test.

The CI failures are unrelated here, so we can ignore them.

…upgrades Add a read-only DescribeTabletServers RPC to the Coordinator, following the pattern of GetClusterHealth (apache#3399). It returns, per tablet server, the same four counters getClusterHealth() reports cluster-wide, scoped to that server: numReplicas, inSyncReplicas, numLeaderReplicas and activeLeaderReplicas. It also returns leaderlessReplicas: the number of replicas on that server whose bucket has no elected leader. This lets operational tooling (e.g. the Fluss Kubernetes Operator): - gate scale-in on a server being empty (numReplicas == 0), - gate rolling upgrades on a per-server green predicate (inSyncReplicas == numReplicas && activeLeaderReplicas == numLeaderReplicas && leaderlessReplicas == 0), - report per-server tablet load in cluster status. Every reported server is green exactly when getClusterHealth() reports GREEN, so a single response is enough for a rolling-upgrade gate. The counters are computed from in-memory CoordinatorContext state on the coordinator event thread via AccessContextEvent. Tablet servers forward the request to the coordinator. Live and shutting-down servers are always reported, even with zero replicas, and dead servers that still hold assigned replicas remain visible so a scale-in gate cannot treat them as drained. Closes apache#3570

morazow · 2026-10-23T08:38:32Z

Hello @polyzos, @fresh-borzoni thanks for the detailed review!

I have addressed your findings, please have another look 🤝

morazow force-pushed the describe-tablet-servers-3570 branch 3 times, most recently from b3d64f5 to 2aad05f Compare July 23, 2026 12:20

morazow marked this pull request as ready for review July 23, 2026 14:12

affo approved these changes Jul 24, 2026

View reviewed changes

affo reviewed Jul 31, 2026

View reviewed changes

affo mentioned this pull request Aug 12, 2026

[rebalance] Change the implementation of Admin#getServerNodes() as introducing a new API to get cluster information #1389

Open

1 of 2 tasks

affo mentioned this pull request Aug 21, 2026

[server] Add off-loop coordinator health cache (mechanism only) #4061

Draft

fresh-borzoni reviewed Aug 21, 2026

View reviewed changes

morazow force-pushed the describe-tablet-servers-3570 branch from 482d2ba to 128cc24 Compare September 14, 2026 11:15

morazow force-pushed the describe-tablet-servers-3570 branch from 4ff2e5f to c167c2c Compare October 22, 2026 12:28

polyzos reviewed Oct 23, 2026

View reviewed changes

morazow force-pushed the describe-tablet-servers-3570 branch from c167c2c to ed8eb42 Compare October 23, 2026 08:35

morazow requested review from fresh-borzoni and polyzos October 23, 2026 08:37

Conversation

morazow commented Jul 22, 2026

Summary

Test Plan

Uh oh!

affo left a comment

Choose a reason for hiding this comment

Uh oh!

affo left a comment

Choose a reason for hiding this comment

Uh oh!

morazow commented Jul 31, 2026

Uh oh!

affo commented Jul 31, 2026

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

affo commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

morazow commented Aug 24, 2026

Uh oh!

affo commented Oct 21, 2026

Uh oh!

morazow commented Oct 21, 2026

Uh oh!

polyzos Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

polyzos Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

polyzos Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

polyzos commented Oct 23, 2026

Uh oh!

morazow commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

affo commented Aug 21, 2026 •

edited

Loading