Skip to content

[test] Stabilize cluster health test setup - #4734

Merged
swuferhong merged 1 commit into
apache:mainfrom
gyang94:codex/fix-cluster-health-4722
Oct 20, 2026
Merged

swuferhong merged 1 commit into
apache:mainfrom
gyang94:codex/fix-cluster-health-4722

Conversation

@gyang94

@gyang94 gyang94 commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fixes #4722.

Stabilize FlussAdminITCase.testClusterHealthDuringRollingUpgrade when the shared cluster is still processing asynchronous table creation/deletion and leader activation from test setup and the preceding test.

Brief change log

  • In testCreateTableWithAggregateEngineAndWalChangelog, wait for every bucket leader of the default table and the five successfully created tables before teardown. Becoming leader completes KV initialization, so teardown cannot delete schemas while those leaders are still using them for KV recovery.
  • In testClusterHealthDuringRollingUpgrade, retry the initial GREEN status and replica-count assertions against the same health response before stopping a tablet server. This allows previous table deletions and coordinator-side leader activation to settle.
  • Keep the existing checks for degraded health after stopping a server and restored health after restarting it.

Tests

Run locally with Java 11:

mvn -B -o -pl fluss-client -am \
  -Dtest=FlussAdminITCase \
  -Dsurefire.failIfNoSpecifiedTests=false verify

Passed: all 47 tests in both Surefire executions of the verify lifecycle (94 test executions, no failures/errors/skips). The lifecycle also passed Checkstyle, Spotless and the license check.

Also ran the two affected tests together five consecutive times, each with a fresh test cluster:

mvn -B -o -pl fluss-client -am \
  '-Dtest=FlussAdminITCase#testCreateTableWithAggregateEngineAndWalChangelog+testClusterHealthDuringRollingUpgrade' \
  -Dsurefire.failIfNoSpecifiedTests=false test

All five runs passed (10 test executions). Confirmed that the table-creation test ran immediately before the cluster-health test, with no SchemaNotExistException during the test methods. An external INFO-level Log4j configuration was used for log inspection.

Additional log inspection found schema-recovery exceptions during FlussClusterExtension.afterAll, after both test methods had completed successfully, when closing servers triggered further leader changes. These shutdown diagnostics are outside the initial-health assertion fixed here.

API and Format

No changes. This PR only changes tests.

Documentation

No documentation changes needed.

Generative AI disclosure

AI-assisted implementation and local validation using OpenAI Codex (GPT-6).

Wait for leader initialization of all tables created by the merge-engine
test before teardown deletes the schemas needed by KV recovery.

Retry the initial cluster health assertions against one response until
the cluster is GREEN before stopping a tablet server. This allows prior
table deletion and leader activation to settle without racing a second
health query.

Fixes apache#4722

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 31/31
@gyang94
gyang94 force-pushed the codex/fix-cluster-health-4722 branch from 9e4e5c0 to b2c8d42 Compare October 20, 2026 06:29
@swuferhong
swuferhong merged commit d8a3d5b into apache:main Oct 20, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[test] Unstable test FlussAdminITCase.testClusterHealthDuringRollingUpgrade

2 participants