regression test #234
Open
regression test #234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: - change to pass multiple outer optimizers in the tests
This was referenced Jul 14, 2025
facebook-github-bot
added
the
CLA Signed
Summary: - move the training loop to a separate file - convert it into a class so that methods can be overridden without having to duplicate code
Summary: - since we made a simplifying assumption that we will only ever have 1 inflight fragment, we can simplify some of the logic particularly getting rid of the local step in manager state - we'll just use the manager's step to determine which fragment to sync - this also allows us to easily support heterogenous hardware by tuning the sync_every setting that will make slower/faster machines to perform less/more local steps before they sync - we can also perform quorum right before preparing a fragment sync - this easily ensures that all replicas will have the same max step and sync the same fragment - fix some numeric issues - the sign of the pseudogradient - inplace lerp when mixing local and global parameters
Summary: - add a test that uses fixtures to validate against previous runs of the test - the fixutres can be written using `WRITE_FIXTURE=true pytest -vs ./torchft/test_diloco_mocked_updates.py` - when writing fixtures, the test also numerically validates the implementation of streaming diloco
d4l3k
reviewed
Jul 15, 2025
There was a problem hiding this comment.
Should we move these to a /test/* dir? It might get real annoying to grep through torchft/ if there's a bunch of testing specific autogenned files
| "0": { | ||
| "layers.0.weight": [ | ||
| [ | ||
| 1.0 |
There was a problem hiding this comment.
is it expected that all of these are whole numbers? I would expect the weights to be a lot more varied?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
WRITE_FIXTURE=true pytest -vs ./torchft/test_diloco_mocked_updates.pyStack created with Sapling. Best reviewed with ReviewStack.