Purpose of flip_sin_to_cos in Sinusoidal Position Embeddings for Diffusion Models

[ahmedbelgacem]

Hi,

I was going through the implementation of some diffusion models and noticed the flip_sin_to_cos parameter. I understand that standard sinusoidal embeddings typically use alternating sine and cosine functions, but I’m curious about the specific role of this parameter.

• What effect does flipping sine to cosine have on the embeddings?
• Are there any particular papers or references that discuss the reasoning behind this choice?

Would love to understand the intuition behind this!

Thanks!

[ukejeb]

Given the linear nature of the dot product, rearranging the order of cosine and sine components does not affect outcomes, provided that consistency is maintained between training and inference processes.

[ahmedbelgacem]

Yes, exactly so i don't see the point of rearranging the order in the first place.
Thank you for your answer.