Non-equivalence between batch_size and accumulation_steps

[14thibea]

Hi everyone!

I tried to implement a reproducibility test to check that increasing the accumulation_steps to the value of the batch_size would give the same value... But actually it doesn't.
I don't know from which component this discrepancy comes from, maybe as the batch size is different the images are actually not loaded in the same order? This issue could be investigated.
But actually even if setting accumulation_steps to mimick the batch_size don't lead exactly to the same results, maybe at least it brings closer the performance: then it would really correspond to the same batch size, but with a different seed.