Harden configure-efa-fsx-lustre-client startup against IMDS race at boot - #1204
Conversation
… race at boot
configure_efa_lustre() downloads and runs the configure-efa-fsx-lustre-client
setup.sh sample, which installs a systemd service that queries EC2 instance
metadata (IMDS) to configure the EFA/Lustre client. That unit is Type=oneshot,
ordered only on network-online.target, with no Restart= directive.
network-online.target indicates the network interface is up, not that the
IMDS endpoint is actually answering requests. On a cold boot there is a
window where the unit can start, query IMDS, get nothing back, and exit
non-zero, and because there is no restart policy, the service is then
permanently in a failed state for the rest of the instance's life, with
nothing to prompt a retry once IMDS does come up moments later. Since this
happens before the EFA/Lustre client is actually configured, the failure
mode looks identical to EFA never being available on that instance, and is
easy to misdiagnose as a hardware/AMI issue instead of a boot-ordering race.
Add a systemd drop-in for configure-efa-fsx-lustre-client.service, installed
before setup.sh runs so it is already present when the unit is created and
started:
- ExecStartPre runs a small readiness script that polls the IMDSv2 token
endpoint and a metadata read (short curl timeouts) for up to ~2 minutes
before allowing the main service to start.
- Restart=on-failure / RestartSec=10s so a transient failure is retried
instead of leaving the unit permanently failed.
- StartLimitIntervalSec/StartLimitBurst sized generously so the retries
are not throttled by systemd's default start-limit before IMDS comes up.
This only changes how the vendored systemd unit is supervised; it does not
modify the unit or scripts shipped in the vendor zip itself.
|
Author mentioned |
|
@svnisar thanks for looking into this. I recovered sanitized production lifecycle evidence from our custom Ansible-based implementation of the equivalent fix. We do not execute the upstream mount_fsx.sh directly; our lifecycle automation ports the relevant behavior into custom Ansible tasks. It installs the same IMDS readiness helper and systemd override before executing AWS’s EFA/Lustre client setup, as done in this PR. After mounting FSx, it runs an equivalent end-to-end transport check, including the multi-interface counter aggregation proposed in #1203. A representative run recorded in our logs today looks like: means: The transport check reads the aggregate EFA LNet send and receive counters, performs a 64 MiB direct write/read through the Lustre mount, and reads the counters again. It exits non-zero unless both totals increase. The verification task completes (ansible playbook ended with failed=0, providing us the end-to-end evidence that Lustre traffic used EFA. Hence, our CloudWatch logs do not include the numeric counter output anymore, however, we did observe & verify non-zero counters manually when we first fixed this Let me know if you have more questions |
|
@svnisar asked for testing evidence, so here it is. I verified this on real hardware too against a real EFA-enabled FSx for Lustre file system, including the failure path. Test environment and limitations are stated at the end. Summary: the mechanism works. The drop-in does attach to the unit that Before: the unpatched unit dies permanentlyVendor unit, no drop-in, IMDS unavailable (injected, see limitations): I then restored IMDS and left the unit alone for 60 s: still The reason this matters is the silent fallback. With the unit failed and no EFA LNet config, the mount still succeeded and I/O still worked, with no error surfaced: Worth noting that After, item 1: the drop-in really does attachThis was the sequencing claim I most wanted to check, since the drop-in is written before the unit exists.
After, item 2: red path, retried and recoveredPR readiness script unmodified. IMDS blocked, then unblocked mid-loop:
After, item 4: EFA transport in useSame Thousands of packets on EFA against tens on TCP, so bulk data is on EFA and TCP carries only control traffic. Non-regression on the non-EFA path
On
|
| case | result | harden reached |
|---|---|---|
| EFA fs, IMDS up | FSx filesystem is EFA-compatible |
yes |
| non-EFA fs, IMDS up | different AZ ... not supported cross-AZ |
no |
| EFA fs, IMDS down | [WARN] Could not determine instance AZ |
no |
| EFA fs, IMDS up again | EFA-compatible |
yes |
In the IMDS-down case configure_efa_lustre still returns rc=0, so the script reports success while skipping EFA for the node's whole lifetime, and since the unit is never created there is nothing to retry on the next boot. That lands the node in exactly the silent-TCP-fallback state shown at the top. Relevant too: the AWS CLI resolves its region from IMDS on these nodes (aws configure list shows region ... imds), so the describe-file-systems call in the same function fails by a second independent path.
Since you already install /usr/local/sbin/wait-for-ec2-imds, calling it (or an equivalent short retry loop) before the first IMDS read in verify_fsx_efa_compatibility() would close this. Happy to leave it as a follow-up issue if you would rather keep this PR narrow, but as it stands the fix is partial: it protects the unit's IMDS calls and not the lifecycle script's.
I also swept the repo for the same pattern elsewhere: 9 files query IMDS and 1 creates a Type=oneshot unit, and the intersection is empty, so no sibling script needs the same treatment.
Test environment and limitations
g6.8xlarge, DLAMIami-0f29bf1fbe374da5f(Ubuntu 24.04.4, kernel 6.17.0-1019-aws), systemd 255, Lustre client 2.15.6, EFA driver 3.0.0g, kefalnd 1.2.2, 1 EFA interface.- FSx for Lustre
PERSISTENT_2,EfaEnabled: true, 4800 GiB, 1000 MB/s/TiB, metadata AUTOMATIC, same AZ (us-east-1c) and same /16 as the client, EFA-enabled security group. - Vendor sample
configure-efa-fsx-lustre-client.zipmd5ea03346719817100bcc2598d5b10761d. - Not tested on HyperPod. HyperPod was not available to me for this, so I ran the same lifecycle-script code paths on a standalone EC2 client against the same kind of file system. The systemd mechanism, the vendor
setup.shand the EFA transport are the same; HyperPod's own orchestration oflifecycle_script.pywas not exercised. Ansible was installed manually since the DLAMI does not ship it. - The cold-boot race was simulated, not reproduced. Every failure case used
iptables -I OUTPUT 1 -d 169.254.169.254 -j DROP, a total sustained IMDS outage, rather than the few seconds of early-boot unavailability a real race produces. That is harsher on the failure side, which suits testing a retry path, but it does not reproduce the natural timing. The one genuine boot I did test (instance reboot with the drop-in installed) came up clean on the first attempt, so it did not hit the race naturally.
3 participants
Purpose
configure_efa_lustre()inmount_fsx.sh(added in #849) downloads and runs theconfigure-efa-fsx-lustre-clientsample from the FSx documentation, whosesetup.shinstalls a systemd service that queries EC2 instance metadata (IMDS) to configure the EFA/Lustre client.That unit is
Type=oneshot, ordered only onnetwork-online.target, with noRestart=directive — and its IMDS queries have no retry.network-online.targetmeans a link has an IP; it does not guarantee the IMDS endpoint is answering yet. On a cold boot the unit can start, miss IMDS by a few seconds, exit non-zero, and stayfailedfor the rest of the instance's life with nothing to retry it. The node then runs without the EFA LNet configuration and Lustre silently falls back to TCP — a failure mode easy to misdiagnose as a hardware/AMI problem.Changes
harden_efa_client_service_startup(), invoked beforesetup.shruns so the override is already in place when the unit is created and started:/usr/local/sbin/wait-for-ec2-imds, a small readiness script that polls the IMDSv2 token endpoint plus a metadata read (short curl timeouts, 2-minute deadline);configure-efa-fsx-lustre-client.servicewithExecStartPre=pointing at that script,Restart=on-failure/RestartSec=10s, andStartLimitIntervalSec=600/StartLimitBurst=20so retries aren't cut off by the default start-limit.Test Plan
Environment:
Test commands:
Test Results
network-online.target), the service failed within seconds of boot and remained permanentlyfailed;kefalndnever loaded, noefaLNet net existed, and the Lustre mount ran over TCP with no visible error.ExecStartPrewaits for IMDS readiness before the vendor script runs; transient failures are retried perRestart=on-failure; the service reachesactiveand EFA transport verification (LNet send/recv counter deltas under generated I/O) passes.Checklist
mainbranch.latest). (N/A — no new dependencies; the vendor zip URL is the one already used by this script)By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.