Why calculate loss using unaugmented gaze direction when augmentation is enabled?

[indexss]

In eve.py, start from line 114, if I use refine_net_do_offset_augmentation = True, According to the logic of your code, the gaze direction after offset augmentation will be stored as side_g_initial, while the gaze that has not been augmented will be stored as side_g_initial_unaugmented. The PoG after offset augmentation will be stored as side_PoG_cm_initial_augmented, and the unaugmented one will be stored as side_PoG_cm_initial_unaugmented.

But in function calculate_losses_and_metrics(), the code you use is:

interm_key = (side + '_g_initial_unaugmented'
                          if self.training and config.refine_net_do_offset_augmentation
                          else side + '_g_initial')

which means that if the config.refine_net_do_offset_augmentation = True, then use the unaugmented gaze direction, but if the config.refine_net_do_offset_augmentation = False, The code used to calculate augmentation above does not execute; the side + '_g_initial' you are using is still the gaze direction that has not been augmented.

Btw, if I change the code to:

interm_key = (side + '_g_initial'
                          if self.training and config.refine_net_do_offset_augmentation
                          else side + '_g_initial')

which means that if the config.refine_net_do_offset_augmentation = True, then use augmented data, the side_g_initial and side_g_initial_unaugmented will become Nan after 3 batches.