Order FSx automount after configure-efa-fsx-lustre-client.service when EFA is configured

[KeitaW]

Purpose

mount_fs() in mount_fsx.sh mounts FSx for Lustre with x-systemd.automount ordered only on network-online.target. When EFA is configured for the client (via configure_efa_lustre() → configure-efa-fsx-lustre-client.service), the automount can fire before that service finishes setting up the EFA LNet transport. The filesystem then comes up over TCP even on a node where EFA was meant to be active, with no visible error.

This gap was surfaced by the SageMaker support team while triaging a HyperPod + EFA-enabled FSx for Lustre incident: nodes did not always come back cleanly on reboot, and the automount had no ordering dependency on the EFA client configuration. The FSx documentation for EFA-enabled filesystems does recommend adding such a dependency (mount-fs-auto-mount-onreboot).

Changes

• In mount_fs(), build the mount options into a variable and, only when EFA_CONFIGURED=true, append x-systemd.after=configure-efa-fsx-lustre-client.service.
• Non-EFA mounts are unchanged (backward-compatible).

Why after= and not requires=

The FSx docs suggest x-systemd.requires=configure-efa-fsx-lustre-client.service. This PR deliberately uses x-systemd.after= (ordering only) instead, because requires=:

• fails the mount outright if the EFA config service fails — losing the safe TCP fallback the Lustre client otherwise keeps (a degraded-but-working node becomes a node with no filesystem); and
• pulls the service in even when an operator has deliberately disabled it — e.g. when running EFA-off as a mitigation for an EFA-side issue, a requires= dependency would try to start the disabled service back up, fighting that operational choice.

after= gives the correct ordering when EFA is on, is a no-op when the service is disabled/masked, and preserves TCP fallback if configuration fails. To turn EFA off operationally, disable (or mask) configure-efa-fsx-lustre-client.service; the mount still succeeds over TCP.

Relationship to #1204

Complementary, non-overlapping. #1204 hardens the service side (retries the IMDS query so configure-efa-fsx-lustre-client.service doesn't fail on a boot-time IMDS race). This PR fixes the mount side (orders the automount after that service so it doesn't race ahead of EFA setup). Together they make reboot recovery behave like a fresh lifecycle run for FSx+EFA nodes. Neither depends on the other; they touch different parts of mount_fsx.sh and can merge in either order.

Test Plan

Environment: SageMaker HyperPod (Slurm), FSx for Lustre created with EFA enabled; EFA-capable GPU instances.

Commands:

bash -n 1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

# on a node after boot, with EFA configured:
systemctl show -p After fsx.mount | tr ' ' '\n' | grep configure-efa   # ordering present
systemd-analyze verify fsx.mount 2>&1 || true
# reboot and confirm the mount comes up over EFA (LNet efa net present,
# send/recv counters move under I/O) rather than silently on TCP.

Results so far:

• bash -n passes.
• opts-building logic verified both ways: non-EFA produces the original option string unchanged; EFA_CONFIGURED=true appends x-systemd.after=configure-efa-fsx-lustre-client.service.
• Done — full on-node reboot verification on an EFA-enabled HyperPod Slurm cluster (ml.g6.16xlarge worker, EFA-enabled PERSISTENT_2 FSx, this branch’s lifecycle scripts unmodified): after reboot the automount fired during EFA setup and the x-systemd.after= ordering held the mount until the service finished; the mount came up with an @efa NID and EFA LNet counters moving under direct I/O. Backward compat also verified: with configure-efa-fsx-lustre-client.service masked, reboot → mount succeeds immediately over TCP (tcp-only NIDs). Full logs: verification comment.

Credit

Thanks to the SageMaker support engineer who flagged the missing automount → EFA-config ordering dependency during incident triage.

Checklist

• I have read the contributing guidelines.
• I am working against the latest main branch.
• I have searched existing open and recently merged PRs to confirm this is not a duplicate (nearest is Harden configure-efa-fsx-lustre-client startup against IMDS race at boot #1204, which is complementary and touches the service side, not the mount options).
• The contribution is self-contained.
• External dependencies are pinned. (N/A — no new dependencies)
• A README is included or updated. (N/A — focused ordering fix)
• New test cases follow the expected directory structure. (N/A — not a test case)

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[svnisar]

Full on-node reboot verification on an EFA-enabled HyperPod cluster is the remaining step before merge.

Can you confirm you have performed this verification?

Additionally, can you also test backward compat: mask configure-efa-fsx-lustre-client.service, reboot, and confirm the mount still succeeds over TCP.

[KeitaW]

@svnisar Both verifications are now done on a real HyperPod Slurm cluster. Summary below, full logs available.

Environment: HyperPod Slurm cluster (controller ml.c5.4xlarge + worker ml.g6.16xlarge), provisioned end-to-end with this PR branch's base-config lifecycle scripts unmodified; EFA-enabled FSx for Lustre PERSISTENT_2 (1000 MB/s/TiB), same AZ/subnet as the worker. Lustre client 2.15.6, EFA driver 3.0.0, kefalnd 1.2.2.

1. Full on-node reboot verification (EFA path)

After systemctl reboot, the journal captured exactly the race this PR closes — the automount fired while configure-efa-fsx-lustre-client.service was still configuring LNet, and the ordering held the mount back until the service finished:

08:59:00.877 systemd[1]: Set up automount fsx.automount.
08:59:15.746 systemd[1]: Starting Configure EFA FSx Lustre Client...
08:59:19.386 systemd[1]: fsx.automount: Got automount request for /fsx, triggered by 2370  ← during EFA setup
08:59:24.705 python3[1590]: Successfully added all EFA interfaces
08:59:25.102 systemd[1]: Finished Configure EFA FSx Lustre Client.
08:59:25.118 systemd[1]: Mounting /fsx...                                                  ← held until service finished
08:59:25.720 systemd[1]: Mounted /fsx.

Without the x-systemd.after= ordering, the 08:59:19 trigger would have mounted ~5s before the EFA LNet existed — i.e. the silent TCP fallback this PR prevents.

Post-reboot the mount runs over EFA/LNet, not TCP (mount success alone can't distinguish the two, so quoting NIDs and counters):

# lctl list_nids
10.1.160.205@tcp
160.205.70.0@efa
# systemctl show -p After fsx.mount
After=... configure-efa-fsx-lustre-client.service ...
# lnetctl net show -v --net efa   (64 MiB direct write+read in between)
send_count: 20 → 214, recv_count: 20 → 278     ← bulk data moving over the efa LNI

(osc.*.import current_connection shows @tcp — expected, MGS/MDS/OSC control connections always ride TCP; the data path is what the efa counters show. The service also sets EFA as the preferred network via udsp.)

2. Backward compat (service masked → TCP fallback)

Masked the service, rebooted. The unit never ran this boot (Loaded: masked, no journal lines), and the mount still succeeded — immediately, over TCP:

09:05:26.965 systemd[1]: Set up automount fsx.automount.
09:05:45.668 systemd[1]: fsx.automount: Got automount request for /fsx, triggered by 2187
09:05:45.670 systemd[1]: Mounting /fsx...        ← 2 ms after trigger: after= is a true no-op when masked
09:05:47.516 systemd[1]: Mounted /fsx.

# lctl list_nids
10.1.160.205@tcp          ← tcp only, no efa net
# 64 MiB direct write/read over TCP: 387/451 MB/s, tcp send/recv counters moved

So after= gives ordering when the service runs and costs nothing when it's disabled — with requires= this boot would have had no filesystem instead.

Two operational notes from the runs, for anyone reproducing:

• The official setup.sh installs the unit as a regular file under /etc/systemd/system/, so a bare systemctl mask fails with "File …service already exists" — systemctl disable works as-is (I moved the unit aside to test a true mask).
• First attempt used ml.g5.8xlarge, where the EFA hardware rejects kefalnd's memory registration (REG_MR … err -22 → kefalnd_dev_init: can't allocate PD → lnetctl net add --net efa fails EINVAL). FSx-EFA clients need Nitro v4+ per the FSx docs; on such nodes the service fails at boot but, with this PR's after= (not requires=), the mount itself still comes up over TCP.

Updating the PR description's test plan section with these results.