Skip to content

Aggregate LNet counters across multiple EFA interfaces in mount_fsx.sh - #1203

Open
davidk-mir wants to merge 1 commit into
awslabs:mainfrom
davidk-mir:fix/efa-counter-aggregation
Open

Aggregate LNet counters across multiple EFA interfaces in mount_fsx.sh#1203
davidk-mir wants to merge 1 commit into
awslabs:mainfrom
davidk-mir:fix/efa-counter-aggregation

Conversation

@davidk-mir

Copy link
Copy Markdown
Contributor

Purpose

Relates to #989, which added the EFA transport verification to mount_fsx.sh.

verify_efa_transport() reads LNet counters with awk '/send_count:/ {print $2}'. lnetctl net show -v --net efa prints one send_count:/recv_count: line per EFA interface, so on instances with more than one EFA interface the pipeline emits a newline-separated list of numbers instead of a single integer. The subsequent [[ "$send_after" -le "$send_before" ]] comparison then fails with a bash "integer expression expected" error, aborting the verification (and, via the ERR trap, the lifecycle script) — on exactly the instance families most likely to use EFA.

Changes

  • Sum the per-interface counters inside awk ({sum += $2} END {print sum + 0}) for the before/after send_count and recv_count reads, so the comparison always receives a single integer regardless of how many EFA interfaces the instance exposes. sum + 0 prints 0 rather than an empty string when no lines match, preserving the existing comparison semantics.

Test Plan

Environment:

  • AWS Service: SageMaker HyperPod (Slurm), FSx for Lustre created with EFA enabled
  • Instance type: instances with multiple EFA interfaces (p5/p6 families), plus a single-EFA-interface control
  • Number of nodes: multi-node cluster

Test commands:

bash -n 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

# on an EFA-enabled node with the FSx mount:
lnetctl net show -v --net efa | awk '/send_count:/ {sum += $2} END {print sum + 0}'

Test Results

  • Before: on a multi-EFA-interface instance, verify_efa_transport() aborts with integer expression expected — the awk pipeline emits one value per interface as a newline-separated list.
  • After: counters aggregate to a single integer, the sum increases under generated I/O, and the verification passes/fails based on actual EFA traffic as intended. Behavior on single-interface instances is unchanged.

Checklist

  • I have read the contributing guidelines.
  • I am working against the latest main branch.
  • I have searched existing open and recently merged PRs to confirm this is not a duplicate.
  • The contribution is self-contained with documentation and scripts.
  • External dependencies are pinned to a specific version or tag (no latest). (N/A — no dependencies added)
  • A README is included or updated with prerequisites, instructions, and known issues. (N/A — focused bug fix)
  • New test cases follow the expected directory structure. (N/A — not a test case)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…transport

verify_efa_transport() reads send_count/recv_count from `lnetctl net show
-v --net efa` with `awk '/send_count:/ {print $2}'` and compares the
before/after values with `[[ "$send_after" -le "$send_before" ]]`.

On an instance with a single EFA interface this prints one line and the
comparison works. On instances with multiple EFA interfaces (e.g. p5/p6
families expose more than one EFA device), `lnetctl net show -v --net
efa` reports one send_count/recv_count line per interface, so the awk
pipeline emits a newline-separated list of numbers instead of a single
integer. The subsequent `[[ ... -le ... ]]` integer comparison then fails
with "integer expression expected", and verify_efa_transport aborts with
a bash error instead of actually verifying whether traffic is flowing
over EFA — on exactly the instance types most likely to use it.

Fix by summing the per-interface counters in awk (`{sum += $2} END
{print sum + 0}`) so the comparison always receives a single integer,
regardless of how many EFA interfaces the instance exposes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant