out proj weight typo fix

[louiswang524]

Here is a typo for initialize_parameter() function. The output_proj.weight should be output_proj_weight

CLIP/clip/model.py

Line 321 in dcba3cb

nn.init.normal_(block.attn.out_proj.weight, std=proj_std)

[99991]

What makes you think so?

[louiswang524]

What makes you think so?

@99991 before it is a parameter defined here

CLIP/clip/model.py

Line 85 in dcba3cb

out_proj_weight=self.c_proj.weight,

[99991]

The line from your first message initializes the member block.attn.out_proj.weight of self.transformer.resblocks

• CLIP/clip/model.py

  Line 321 in dcba3cb

  nn.init.normal_(block.attn.out_proj.weight, std=proj_std)

• CLIP/clip/model.py

  Line 319 in dcba3cb

  for block in self.transformer.resblocks:

• CLIP/clip/model.py

  Line 282 in dcba3cb

  self.transformer = Transformer(

• CLIP/clip/model.py

  Line 200 in dcba3cb

  self.resblocks = nn.Sequential(*[ResidualAttentionBlock(width, heads, attn_mask) for _ in range(layers)])

• CLIP/clip/model.py

  Line 175 in dcba3cb

  self.attn = nn.MultiheadAttention(d_model, n_head)

• https://github.com/pytorch/pytorch/blob/3f77002b968c093b2b668fed24d85f6a365d6b3c/torch/nn/modules/activation.py#L1102

But in your last message, you highlight self.c_proj.weight, which is not a member of self.transformer.resblocks. Instead, it is a member of AttentionPool2d, which is only used in ModifiedResNet, i.e. self.visual of CLIP.

• CLIP/clip/model.py

  Line 85 in dcba3cb

  out_proj_weight=self.c_proj.weight,

• https://github.com/openai/CLIP/blob/dcba3cb2e2827b402d2701e7e1c7d9fed8a20ef1/clip/model.py#L58C7-L58C22

• CLIP/clip/model.py

  Line 127 in dcba3cb

  self.attnpool = AttentionPool2d(input_resolution // 32, embed_dim, heads, output_dim)

• CLIP/clip/model.py

  Line 264 in dcba3cb

  self.visual = ModifiedResNet(