[Blockstore] Add GetClusterCapacity handler #3720

Rattysed · 2025-06-19T16:31:54Z

Add GetClusterCapacity handler that collects cluster utilization information from YDB and DiskRegistry and returns it grouped by disk types.
This handler will be used by Disk Manager to intelligently distribute newly created disks across shards.

For collecting data from Disk Registry, an analogous handler has been added. For collecting from YDB, GetStorageStatsRequest is used (https://github.com/ydb-platform/ydb/blob/911050b753e7fb532f13324f65d2a167e8045e37/ydb/core/protos/sys_view.proto#L353-L358)

github-actions · 2025-06-20T16:45:09Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit cc7d416.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8666	8663	0	1	1	1

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit cc7d416.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
45	44	0	0	0	1

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit cc7d416.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
43	42	0	0	0	1

github-actions · 2025-06-30T17:08:42Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 666caee.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8723	8707	0	0	8	8

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 666caee.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
112	112	0	0	0	0

cloud/blockstore/libs/storage/protos/disk.proto

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp

cloud/blockstore/private/api/protos/disk.proto

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp

github-actions · 2025-06-30T22:57:54Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 851ed53.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8690	8677	0	3	10	0

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 851ed53.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
54	54	0	0	0	0

github-actions · 2025-07-03T12:26:26Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 6e8457a.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8691	8664	0	5	22	0

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 6e8457a.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
342	328	0	2	12	0

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 6e8457a.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
329	315	0	2	12	0

github-actions · 2025-07-04T01:33:16Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit adffe69.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8691	8675	0	3	13	0

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit adffe69.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
329	315	0	2	12	0

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit adffe69.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
329	315	0	2	12	0

github-actions · 2025-07-22T15:14:44Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit c0cd413.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
5526	5525	0	0	0	1	0

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp

github-actions · 2025-07-23T17:05:07Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit dd6a9a9.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
5526	5525	0	0	0	1	0

Rattysed · 2025-07-24T22:51:27Z

ручка GetStorageStats не даёт информации о том, какой конкретно кластер YDB насколько занят. Внутри могут быть любые облачные сервисы, например NFS

github-actions · 2025-08-04T23:00:32Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 82dfa58.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
5588	5587	0	0	0	1	0

github-actions · 2025-08-06T09:58:26Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-release-msan: all tests PASSED for commit 82dfa58.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
4936	4936	0	0	0	0	0

github-actions · 2025-08-06T10:01:02Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-release-asan: all tests PASSED for commit 82dfa58.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
4954	4954	0	0	0	0	0

github-actions · 2025-08-06T10:07:37Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-release-ubsan: all tests PASSED for commit 82dfa58.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
4967	4967	0	0	0	0	0

github-actions · 2025-08-06T11:34:04Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 82dfa58.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
5588	5587	0	0	0	1	0

SvartMetal self-requested a review June 19, 2025 16:56

drbasic previously approved these changes Jun 23, 2025

View reviewed changes

Rattysed mentioned this pull request Jun 30, 2025

[Disk Manager] Introduce internal/pkg/cells.storageYDB to be used for Cells metadata storing #3752

Merged

Rattysed force-pushed the users/rattysed/issue-3614-getclustercapacity-handler branch from cc7d416 to 666caee Compare June 30, 2025 15:16

SvartMetal requested changes Jun 30, 2025

View reviewed changes

Rattysed dismissed drbasic’s stale review via cad1c79 June 30, 2025 20:32

Rattysed marked this pull request as draft July 1, 2025 11:50

Rattysed marked this pull request as ready for review July 21, 2025 08:39

Rattysed force-pushed the users/rattysed/issue-3614-getclustercapacity-handler branch from adffe69 to 198d1b2 Compare July 21, 2025 10:06

Rattysed added blockstore Add this label to run only cloud/blockstore build and tests on PR large-tests Launch large tests for PR labels Jul 21, 2025

SvartMetal reviewed Jul 23, 2025

View reviewed changes

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Jul 23, 2025

View reviewed changes

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Jul 23, 2025

View reviewed changes

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Jul 23, 2025

View reviewed changes

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Jul 23, 2025

View reviewed changes

cloud/blockstore/libs/storage/service/service_actor_actions_get_cluster_capacity.cpp Outdated Show resolved Hide resolved

Rattysed requested review from SvartMetal and drbasic July 23, 2025 15:12

SvartMetal previously approved these changes Jul 23, 2025

View reviewed changes

drbasic previously approved these changes Jul 23, 2025

View reviewed changes

Rattysed marked this pull request as draft July 24, 2025 22:49

Rattysed dismissed drbasic’s stale review via 2dc236a July 31, 2025 13:21

Rattysed added 17 commits August 4, 2025 20:30

add handlers in dr and actions

4e4fbdb

rename method

d182f26

fix broken build

b4e6d8a

rename fields and methods

3142070

fix broken build

f9a9fc6

edit log

1c177d7

add ut, it shows, that code is bad :(

09f1e30

change sending mechanism to pipe send into bsc

4aa2d99

add debug log

bc7b2ce

change PDiskFilter, remove debug log. Waiting for ydb sync :(

ee097d5

remove ut

f3a61fd

fix test, change ydb gathering logic

5f9b8a7

style fixes

ac6c902

tiny style fix

4e2bd1b

change the way of how we gather data from ydb

d96e210

add some safety checks

53ea539

fix codestyle

82dfa58

Rattysed force-pushed the users/rattysed/issue-3614-getclustercapacity-handler branch from ed15a7b to 82dfa58 Compare August 4, 2025 20:30

SvartMetal approved these changes Aug 5, 2025

View reviewed changes

drbasic approved these changes Aug 6, 2025

View reviewed changes

Rattysed added asan Launch builds with address sanitizer along with regular build msan Launch builds with memory sanitizer along with regular build ubsan Launch builds with undefined behaviour sanitizer along with regular build labels Aug 6, 2025

Rattysed merged commit dc263bc into main Aug 6, 2025
34 of 36 checks passed

Rattysed deleted the users/rattysed/issue-3614-getclustercapacity-handler branch August 6, 2025 10:36

[Blockstore] Add GetClusterCapacity handler #3720

[Blockstore] Add GetClusterCapacity handler #3720

Uh oh!

Conversation

Rattysed commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 23, 2025

Uh oh!

Rattysed commented Jul 24, 2025

Uh oh!

github-actions bot commented Aug 4, 2025

Uh oh!

github-actions bot commented Aug 6, 2025

Uh oh!

github-actions bot commented Aug 6, 2025

Uh oh!

github-actions bot commented Aug 6, 2025

Uh oh!

Uh oh!

github-actions bot commented Aug 6, 2025

Uh oh!

Uh oh!

Rattysed commented Jun 19, 2025 •

edited

Loading

github-actions bot commented Jun 20, 2025 •

edited

Loading

github-actions bot commented Jun 30, 2025 •

edited

Loading

github-actions bot commented Jun 30, 2025 •

edited

Loading

github-actions bot commented Jul 3, 2025 •

edited

Loading

github-actions bot commented Jul 4, 2025 •

edited

Loading