Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training verbose logs #46

Open
xenia-kra opened this issue Jun 27, 2023 · 3 comments
Open

Training verbose logs #46

xenia-kra opened this issue Jun 27, 2023 · 3 comments

Comments

@xenia-kra
Copy link

I'm trying to execute a training process with composer run.py --config-path yamls/hydra-yamls --config-name SD-2-base-256.yaml, after changing the configuration to use a custom data loader. Im getting some generic error
AttributeError("'IterableDatasetDict' object has no attribute '_distributed'") from unspecified source. How can I get more details?

@mvpatel2000
Copy link
Collaborator

Can you provide some more information on what your custom dataloader is? It looks like you are having some trouble running in a distributed setting with your dataloader. Torch dataloaders (and our streaming dataset) handle all the distributed magic for you -- I'm not super sure what you are running in this case

@xenia-kra
Copy link
Author

I was hoping, in general, to solve the problem when I have an ability to debug. Specifically, how do I know what file/line does this error come from?

@mvpatel2000
Copy link
Collaborator

Can you please provide a full trace / logs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants