Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: fix ddp #81

Merged
merged 2 commits into from
Nov 27, 2024
Merged

Fix: fix ddp #81

merged 2 commits into from
Nov 27, 2024

Conversation

talebolano
Copy link
Contributor

Fixes #39
修正了多进程训练时,个别进程可能因为gt为0,导致模型输出没有以dn_结尾的结果。进而导致损失中'loss_vfl_dn_{id}', 'loss_bbox_dn_{id}', 'loss_giou_dn_{id}', 'loss_fgl_dn_{id}', 'loss_ddf_dn_{id}', 'loss_vfl_dn_{id}'不存在,导致训练卡死的问题

修正了多进程时因为gt为0,模型没有dn结果以及loss_ddf_dn不存在,导致训练卡死的问题
修正dn_num_boxes为0导致的nan
@Peterande Peterande merged commit b9da032 into Peterande:master Nov 27, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

模型多卡训练卡死
2 participants