It makes more sense to do: self.norm = nn.LayerNorm(dim, elementwise_affine=False) rather than the default `nn.LayerNorm(dim)` which would learn its own γ and β. So one has: gate * layer((1 - mu) * normalize(x) + shift)
It makes more sense to do:
self.norm = nn.LayerNorm(dim, elementwise_affine=False) rather than the default
nn.LayerNorm(dim)which would learn its own γ and β.So one has:
gate * layer((1 - mu) * normalize(x) + shift)