RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got

[zhanwenchen]

Likely an issue from float16 => float32

Traceback (most recent call last):
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 394, in <module>
    main()
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 390, in main
    train(cfg, local_rank, args.distributed, logger)
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 197, in train
    loss_dict = model(images, targets)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/apex-0.1-py3.8-linux-x86_64.egg/apex/amp/_initialize.py", line 198, in new_fwd
    output = old_fwd(*applier(args, input_caster),
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 52, in forward
    x, result, detector_losses = self.roi_heads(features, proposals, targets, logger)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 69, in forward
    x, detections, loss_relation = self.relation(features, detections, targets, logger)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 80, in forward
    refine_logits, relation_logits, add_losses = self.predictor(proposals, rel_pair_idxs, rel_labels, rel_binarys, roi_features, union_features, logger)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py", line 81, in forward
    obj_dists, obj_preds, edge_ctx = self.context_layer(roi_features, proposals, logger)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/model_transformer.py", line 250, in forward
    obj_embed = self.obj_embed1(obj_labels)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/localtmp/pct4et/conda_envs/relaug/lib/python3.8/site-packages/torch/nn/functional.py", line 2044, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

[zhanwenchen]

Only modifying visual_genome get_groundtruth did not solve the issue.

Also, this only affects predcls.