DAOS-18834 cart: Add bulk, rpc and corpc metrics by frostedcmos · Pull Request #18037 · daos-stack/daos

frostedcmos · 2026-04-17T17:08:29Z

Refactored how per-context metrics are stored: crt_internal_types.h now has a list of metrics and provides helper macros for manipulating them
Added metrics for: rpc send,receive,completed,reply,failures corpc initiate,complete,failures, bulk create,destroy,bind,failures
Old metrics converted to new format

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

- Refactored how per-context metrics are stored: crt_internal_types.h now has a list of metrics and provides helper macros for manipulating them - Added metrics for: rpc send,receive,completed,reply,failures corpc initiate,complete,failures, bulk create,destroy,bind,failures - Old metrics converted to new format Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

github-actions · 2026-04-17T17:08:45Z

Ticket title is 'cart: add rpc, corpc and bulk counters'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-18834

daosbuild3 · 2026-04-18T05:53:00Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18037/2/execution/node/1032/log

- Update telemetry_utils.py with new metrics Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

daosbuild3 · 2026-04-21T23:02:19Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18037/5/execution/node/965/log

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

soumagne · 2026-04-23T16:28:54Z

+	X(CM_RPCS_REPLY_FAILED, D_TM_COUNTER, "Total number of failed replies", "rpcs")            \
+	X(CM_RPCS_SENT, D_TM_COUNTER, "Total number of RPCs sent", "rpcs")                         \
+	X(CM_RPCS_COMPLETED, D_TM_COUNTER, "Total number of RPCs completed successfully", "rpcs")  \
+	X(CM_RPCS_COMPLETED_ERR, D_TM_COUNTER, "Total number of sent RPCs completed with error",   \


if I follow the previous naming logic, it would be RPC_FWD_FAILED ?

'FWD' in the naming here (line 473) is for cases when we forward rpc to another target (with bulk binding etc), not for when we call HG_Forward() :) This particular metric is for any rpcs that completed with an error.

ok well that's definitely confusing then, well also because you had that in the list before the reply, maybe clarify what is forwarded then.

Rearranged metircs and added wording to FWD metric

soumagne · 2026-04-23T16:29:53Z

+	X(CM_RPC_WAITQ_DEPTH, D_TM_GAUGE, "Current count of enqueued RPCs", "rpcs")                \
+	X(CM_RPC_QUOTA_EXCEEDED, D_TM_COUNTER, "Total number of exceeded RPC quota events",        \
+	  "events")                                                                                \
+	X(CM_RPCS_RECV, D_TM_COUNTER, "Total number of RPCs received", "rpcs")                     \


would be better to keep CM_RPC prefix without the s to avoid having CM_RPC with and without the plural

agreed, will change to CM_RPC

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

daosbuild3 · 2026-04-23T17:29:59Z

Test stage Build on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18037/8/execution/node/250/log

daosbuild3 · 2026-04-23T17:32:49Z

Test stage Build on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18037/8/execution/node/298/log

daosbuild3 · 2026-04-23T17:37:00Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18037/8/execution/node/344/log

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

daosbuild3 · 2026-04-24T02:44:08Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18037/10/testReport/

tanabarr

Go changes LGTM

kjacque · 2026-04-24T23:48:44Z

Looked at both C and Go changes. LGTM.

frostedcmos · 2026-05-11T18:03:24Z

Verified some of new counters locally by running self_test and monitoring metrics:
`14475 - Metric Set: engine_net_cm_bulk_create (Type: Counter)
14476 Total number of bulks created
14477 Metric Labels Value
14478 ------ ------ -----
..
14483 Counter (context=2, provider=ofi+tcp, rank=0) 2

14509 - Metric Set: engine_net_cm_bulk_free (Type: Counter)
14510 Total number of bulks that were freed
14511 Metric Labels Value
14512 ------ ------ -----
...
14517 Counter (context=2, provider=ofi+tcp, rank=0) 2

`

daltonbohning · 2026-05-12T16:00:55Z

This conflicts with #18161

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

kjacque

Go changes still LGTM. :)

daltonbohning · 2026-05-12T17:41:58Z

-        "engine_net_quota_exceeded",
-        "engine_net_glitch",
-        "engine_net_failed_addr",
-        "engine_net_req_timeout",


engine_net_req_timeout is still being used by this test so the test needs to be updated with the new metric name. This PR should probably run with Features: telemetry to catch any other issues like this

daos/src/tests/ftest/telemetry/telemetry_pool_metrics.py

Line 150 in 51205bb

"engine_net_req_timeout",

good catch, will do with feature

tanabarr

go changes LGTM

- Linting fixes Features: telemetry Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

Features: telemetry Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos requested review from a team as code owners April 17, 2026 17:08

- Change metrics path format to match daos

5c58221

- Update telemetry_utils.py with new metrics Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos requested review from a team as code owners April 21, 2026 15:45

frostedcmos added 2 commits April 21, 2026 16:04

Merge branch 'master' into frostedcmos/DAOS-18834

3943678

fix cart metric names in telemetry list

ca3d9bc

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos marked this pull request as draft April 21, 2026 17:36

- fix in bulk_free to ignore decoded bulks

70e28b3

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos requested a review from soumagne April 22, 2026 21:38

frostedcmos added 2 commits April 23, 2026 16:06

formatting

bb6049d

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

Merge branch 'master' into frostedcmos/DAOS-18834

bc0ebac

soumagne reviewed Apr 23, 2026

View reviewed changes

- address review comments

6605c1d

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos added 2 commits April 23, 2026 17:38

- update usage of renamed metrics

3bc384b

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

- Clarifying forward metric

71b1dc8

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

tanabarr previously approved these changes Apr 24, 2026

View reviewed changes

frostedcmos marked this pull request as ready for review April 24, 2026 16:34

frostedcmos requested review from liw and soumagne April 24, 2026 16:34

frostedcmos requested a review from jgmoore-or April 24, 2026 17:12

kjacque previously approved these changes Apr 24, 2026

View reviewed changes

soumagne reviewed May 1, 2026

View reviewed changes

Comment thread src/cart/crt_internal_types.h

soumagne previously approved these changes May 1, 2026

View reviewed changes

Merge branch 'master' into frostedcmos/DAOS-18834

38e9cfa

frostedcmos requested a review from a team May 12, 2026 14:50

daltonbohning removed the request for review from a team May 12, 2026 16:01

Merge branch 'master' into frostedcmos/DAOS-18834

98ab262

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos dismissed stale reviews from soumagne, kjacque, and tanabarr via 98ab262 May 12, 2026 16:36

frostedcmos requested review from kjacque, soumagne and tanabarr May 12, 2026 16:41

kjacque previously approved these changes May 12, 2026

View reviewed changes

soumagne previously approved these changes May 12, 2026

View reviewed changes

daltonbohning requested changes May 12, 2026

View reviewed changes

tanabarr previously approved these changes May 12, 2026

View reviewed changes

- Fixed wrong metric name in the test

eda228a

- Linting fixes Features: telemetry Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos dismissed stale reviews from tanabarr, soumagne, and kjacque via eda228a May 12, 2026 20:25

- retrigger

2754173

Features: telemetry Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

Conversation

frostedcmos commented Apr 17, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Apr 18, 2026

Uh oh!

daosbuild3 commented Apr 21, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Apr 23, 2026

Uh oh!

daosbuild3 commented Apr 23, 2026

Uh oh!

daosbuild3 commented Apr 23, 2026

Uh oh!

daosbuild3 commented Apr 24, 2026

Uh oh!

tanabarr left a comment

Choose a reason for hiding this comment

Uh oh!

kjacque commented Apr 24, 2026

Uh oh!

Uh oh!

frostedcmos commented May 11, 2026

Uh oh!

daltonbohning commented May 12, 2026

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tanabarr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

github-actions Bot commented Apr 17, 2026 •

edited

Loading