Skip to content

Backport #8935: Parallelize cluster node health checks (release-13.2) - #8938

Merged
Volkan Cetin (vlkncetin) merged 3 commits into
release-13.2from
backport-8935-release-13.2
Oct 20, 2026
Merged

Volkan Cetin (vlkncetin) merged 3 commits into
release-13.2from
backport-8935-release-13.2

Conversation

@vlkncetin

Copy link
Copy Markdown
Collaborator

Backports #8935 (fadced9).

Applies the merged implementation and regression tests unchanged.

(cherry picked from commit fadced9)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cancellation test has a synchronization race that can cause nondeterministic failures.

1 open issue
What changed in this PR

Backports parallel cluster health checks to release-13.2, reducing probe latency while preserving result ordering.

Changes:

  • Establish source connections and run checks concurrently.
  • Preserve source-major output using a result matrix.
  • Add concurrency and cancellation regression tests.
File Description
src/​test/​regress/​citus_tests/​test/​test_cluster_node_health.py Tests concurrent probes, results, and cancellation cleanup.
src/​backend/​distributed/​operations/​health_check.c Parallelizes connectivity checks while preserving output order.

Review effort: Balanced

Comment thread src/test/regress/citus_tests/test/test_cluster_node_health.py Outdated
Refactor checks_are_in_flight to include connection count check.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.23810% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.15%. Comparing base (6d8ea44) to head (943fd75).

❌ Your project check has failed because the head coverage (29.15%) is below the target coverage (87.50%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (6d8ea44) and HEAD (943fd75). Click for more details.

HEAD has 122 uploads less than BASE
Flag BASE (6d8ea44) HEAD (943fd75)
17_regress_check-follower-cluster_v13.4.0_all 1 0
16_17_upgrade 1 0
17_regress_check-follower-cluster_13.4-1_all 1 0
17_regress_check-follower-cluster_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-columnar-isolation_v13.4.0_all 1 0
17_regress_check-columnar-isolation_13.4-1_v13.4.0_coordinatoronly 1 0
17_regress_check-enterprise-isolation-logicalrep-2_13.4-1_v13.4.0_coordinatoronly 1 0
15_regress_check-follower-cluster 1 0
17_regress_check-columnar-isolation 1 0
17_regress_check-columnar_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-columnar-isolation_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-add-backup-node_v13.4.0_all 1 0
16_regress_check-follower-cluster 1 0
17_regress_check-add-backup-node_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-split_13.4-1_v13.4.0_coordinatoronly 1 0
15_regress_check-columnar-isolation 1 0
15_arbitrary_configs_3 1 0
17_regress_check-enterprise-isolation-logicalrep-2 1 0
16_regress_check-enterprise-failure 1 0
16_regress_check-enterprise-isolation-logicalrep-3 1 0
15_regress_check-add-backup-node 1 0
17_regress_check-enterprise-isolation_13.4-1_v13.4.0_coordinatoronly 1 0
15_16_upgrade 1 0
17_regress_check-add-backup-node_13.4-1_all 1 0
15_regress_check-split-output-plugin-denied 1 0
15_regress_check-enterprise-isolation-logicalrep-3 1 0
16_regress_check-enterprise-isolation 1 0
17_regress_check-enterprise_13.4-1_v13.4.0_coordinatoronly 1 0
15_regress_check-vanilla 1 0
17_regress_check-enterprise-isolation-logicalrep-1 1 0
17_regress_check-enterprise-isolation-logicalrep-3_13.4-1_all 1 0
17_regress_check-columnar_v13.4.0_all 1 0
17_arbitrary_configs_4 1 0
15_arbitrary_configs_4 1 0
16_arbitrary_configs_4 1 0
16_regress_check-columnar 1 0
17_regress_check-enterprise_v13.4.0_all 1 0
17_regress_check-split_13.4-1_all 1 0
17_regress_check-enterprise-isolation_v13.4.0_all 1 0
16_regress_check-multi-mx 1 0
15_regress_check-operations 1 0
17_regress_check-operations 1 0
17_regress_check-operations_13.4-1_all 1 0
17_regress_check-operations_v13.4.0_all 1 0
17_regress_check-enterprise-isolation-logicalrep-1_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-split_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-columnar_13.4-1_v13.4.0_coordinatoronly 1 0
15_regress_check-enterprise 1 0
15_regress_check-split 1 0
17_cdc_installcheck 1 0
17_regress_check-split 1 0
17_regress_check-enterprise-isolation_13.4-1_all 1 0
17_regress_check-failure 1 0
16_regress_check-failure 1 0
15_regress_check-failure 1 0
16_regress_check-vanilla 1 0
17_regress_check-vanilla 1 0
16_regress_check-split 1 0
17_regress_check-enterprise-isolation-logicalrep-1_13.4-1_v13.4.0_coordinatoronly 1 0
16_regress_check-enterprise-isolation-logicalrep-1 1 0
16_regress_check-multi-1-create-citus 1 0
15_regress_check-multi-1 1 0
16_regress_check-multi-1 1 0
17_regress_check-multi-1_13.4-1_v13.4.0_coordinatoronly 1 0
17_regress_check-operations_13.4-1_v13.4.0_coordinatoronly 1 0
17_regress_check-isolation_13.4-1_v13.4.0_coordinatoronly 1 0
17_regress_check-isolation_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-isolation_v13.4.0_all 1 0
15_arbitrary_configs_2 1 0
17_regress_check-multi-1_v13.4.0_all 1 0
16_regress_check-multi 1 0
17_regress_check-enterprise-isolation-logicalrep-2_13.4-1_all 1 0
17_regress_check-multi_13.4-1_all 1 0
17_regress_check-enterprise_13.4-1_v13.4.0_workeronly 1 0
15_regress_check-multi 1 0
15_regress_check-enterprise-isolation-logicalrep-1 1 0
16_arbitrary_configs_5 1 0
17_regress_check-multi-1-create-citus 1 0
17_regress_check-follower-cluster_13.4-1_v13.4.0_coordinatoronly 1 0
17_regress_check-enterprise-isolation-logicalrep-3 1 0
15_regress_check-enterprise-isolation-logicalrep-2 1 0
17_arbitrary_configs_1 1 0
15_arbitrary_configs_1 1 0
15_regress_check-isolation 1 0
17_regress_check-operations_13.4-1_v13.4.0_workeronly 1 0
15_regress_check-multi-1-create-citus 1 0
17_regress_check-multi_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-multi_v13.4.0_all 1 0
17_arbitrary_configs_2 1 0
17_regress_check-enterprise 1 0
17_regress_check-multi_13.4-1_v13.4.0_coordinatoronly 1 0
16_arbitrary_configs_1 1 0
17_regress_check-multi-1_13.4-1_all 1 0
17_regress_check-multi-1_13.4-1_v13.4.0_workeronly 1 0
17_regress_check-columnar-isolation_13.4-1_all 1 0
17_arbitrary_configs_0 1 0
16_arbitrary_configs_2 1 0
17_regress_check-isolation 1 0
17_regress_check-follower-cluster 1 0
15_regress_check-multi-mx 1 0
17_regress_check-enterprise-isolation-logicalrep-2_v13.4.0_all 1 0
17_regress_check-columnar 1 0
15_regress_check-columnar 1 0
15_regress_check-enterprise-failure 1 0
17_regress_check-multi 1 0
16_arbitrary_configs_0 1 0
17_regress_check-query-generator 1 0
17_regress_check-enterprise-isolation-logicalrep-3_13.4-1_v13.4.0_workeronly 1 0
15_17_upgrade 1 0
16_regress_check-split-output-plugin-denied 1 0
17_regress_check-split-output-plugin-denied 1 0
17_regress_check-enterprise-isolation_13.4-1_v13.4.0_workeronly 1 0
16_regress_check-enterprise-isolation-logicalrep-2 1 0
17_arbitrary_configs_3 1 0
16_regress_check-columnar-isolation 1 0
17_regress_check-add-backup-node 1 0
17_regress_check-enterprise-isolation-logicalrep-1_13.4-1_all 1 0
16_regress_check-enterprise 1 0
17_regress_check-enterprise-failure 1 0
15_arbitrary_configs_0 1 0
16_regress_check-isolation 1 0
17_regress_check-isolation_13.4-1_all 1 0
Additional details and impacted files
@@                Coverage Diff                @@
##           release-13.2    #8938       +/-   ##
=================================================
- Coverage         88.72%   29.15%   -59.57%     
=================================================
  Files               289      289               
  Lines             64641    64335      -306     
  Branches           8153     8063       -90     
=================================================
- Hits              57351    18758    -38593     
- Misses             4874    43383    +38509     
+ Partials           2416     2194      -222     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlkncetin
Volkan Cetin (vlkncetin) merged commit 27446f8 into release-13.2 Oct 20, 2026
498 of 599 checks passed
@vlkncetin
Volkan Cetin (vlkncetin) deleted the backport-8935-release-13.2 branch October 20, 2026 14:59
ibrahim halatci (ihalatci) pushed a commit that referenced this pull request Oct 21, 2026
Backport the health-check parallelization from fadced9, including the cancellation-test synchronization correction from release-13.2 (#8938).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ibrahim halatci (ihalatci) pushed a commit that referenced this pull request Oct 21, 2026
…#8939)

Backports #8935 to release-14.0.

- Parallelizes source connections and connectivity probes across source
nodes.
- Preserves result ordering and true/false/NULL semantics.
- Includes the cancellation-test synchronization correction from the
release-13.2 backport (#8938).

Production code matches #8935; both changed files match #8938.

Co-authored-by: Volkan Cetin <volkanetin@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants