[BE] [float8] Run test_everything.sh in float8 test CI using linux.aws.h100.4

[danielvegamyhre]

Fixes #2477

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2541

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit 2530c2d with merge base c57226b ():

CANCELLED JOB - The following job was cancelled. Please retry:

• Run Regression Tests / test-nightly (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://downloa... / linux-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[danielvegamyhre]

fyi @vkuzo

[drisspg]

this is the main way we are testing our integrations, add back

Also seems that there still exsits tests that can run non not 4 gpus that are worth doing

[danielvegamyhre]

Also seems that there still exsits tests that can run non not 4 gpus that are worth doing

Can you clarify what you mean here? We could use linux.aws.h100.8 for tests that require GPUs (e.g., 3d parallel). I have an in progress MoE test that requires 8 GPUs (#2481) but for the float8 distributed tests they require 4 max i believe

[drisspg]

My comment was just that we now are requesting 4 gpus for all of these tests where previously we only needed 1, I am not sure if this ends up being harder to schedule / add smore delays

[danielvegamyhre]

My comment was just that we now are requesting 4 gpus for all of these tests where previously we only needed 1, I am not sure if this ends up being harder to schedule / add smore delays

Hmm I see, perhaps we can see how it goes over the next week or so and if it's slowing us down then we can revisit

[vkuzo]

how about:

1. split test_everything.sh into one piece for single GPU and another one for multi GPU
2. keep the single GPU one in the current target
3. make a new target for multi GPU

[vkuzo]

@danielvegamyhre IMO worth reverting or forward fixing asap, because this PR seems to run the single GPU float8 tests twice, once from the original code and once again from test_everything.sh

[danielvegamyhre]

Fix forward: #2561