Skip to content

Add multigpu pt #288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Add multigpu pt #288

wants to merge 10 commits into from

Conversation

gkumbhat
Copy link
Collaborator

@gkumbhat gkumbhat commented Dec 1, 2023 •
edited
Loading

Changes

  • Add FSDP configuration for PT trainer
  • Add torch elastic launch

NOTE: This PR to be merged only after merge and rebase with #287

This currently gives following error:

ValueError: expected to be in states [<TrainingState.IDLE: 1>] but current state is TrainingState.FORWARD_BACKWARD

gkumbhat and others added 10 commits November 26, 2023 14:49
@gkumbhat @alex-jw-brooks
🚧 Initiate HF Trainer
f6ae261 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat @alex-jw-brooks
🚧 Replace custom training loop with HF Trainer
da9418d 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat
🔥 Remove old training execution functions
c442b06 
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat @alex-jw-brooks
🚧 Fix missing imports and arguments
565cb3c 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat @alex-jw-brooks
🐛 Remove explicit conversion of model to dtype
fc6e689 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat @alex-jw-brooks
🎨 Fix formatting
3498d3d 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat
🎨 Fix linting
0fbac9f 
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat
🎨 Fix formatting
f051d83 
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat @alex-jw-brooks
🎨 Fix linting
4561ae1 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
@gkumbhat @alex-jw-brooks
🚧 Add FSDP configuration for executing training across multi-gpu
b720824 
Co-authored-by: Alex-Brooks <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@evaline-ju evaline-ju Awaiting requested review from evaline-ju evaline-ju will be requested when the pull request is marked ready for review evaline-ju is a code owner

@gabe-l-hart gabe-l-hart Awaiting requested review from gabe-l-hart gabe-l-hart will be requested when the pull request is marked ready for review gabe-l-hart is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@gkumbhat