Question about the provided pre-trained weights.

[PardoAlejo]

Dear all,

Thanks for your excellent work. I downloaded the Kinetic-400 ViT-Large pre-trained checkpoint. I can load it using my own ViT models. However, I noticed that some keys do not seem to map to anything in the model definition and I would appreciate it if you could explain what "pred_head" refers to.

odict_keys(['pred_head.transforms.0.0.norm1.weight', 'pred_head.transforms.0.0.norm1.bias', 'pred_head.transforms.0.0.attn.q.weight', 'pred_head.transforms.0.0.attn.q.bias', 'pred_head.transforms.0.0.attn.k.weight', 'pred_head.transforms.0.0.attn.k.bias', 'pred_head.transforms.0.0.attn.v.weight', 'pred_head.transforms.0.0.attn.v.bias', 'pred_head.transforms.0.0.attn.proj.weight', 'pred_head.transforms.0.0.attn.proj.bias', 'pred_head.transforms.0.0.norm2.weight', 'pred_head.transforms.0.0.norm2.bias', 'pred_head.transforms.0.0.mlp.fc1.weight', 'pred_head.transforms.0.0.mlp.fc1.bias', 'pred_head.transforms.0.0.mlp.fc2.weight', 'pred_head.transforms.0.0.mlp.fc2.bias', 'pred_head.transforms.0.1.norm1.weight', 'pred_head.transforms.0.1.norm1.bias', 'pred_head.transforms.0.1.attn.q.weight', 'pred_head.transforms.0.1.attn.q.bias', 'pred_head.transforms.0.1.attn.k.weight', 'pred_head.transforms.0.1.attn.k.bias', 'pred_head.transforms.0.1.attn.v.weight', 'pred_head.transforms.0.1.attn.v.bias', 'pred_head.transforms.0.1.attn.proj.weight', 'pred_head.transforms.0.1.attn.proj.bias', 'pred_head.transforms.0.1.norm2.weight', 'pred_head.transforms.0.1.norm2.bias', 'pred_head.transforms.0.1.mlp.fc1.weight', 'pred_head.transforms.0.1.mlp.fc1.bias', 'pred_head.transforms.0.1.mlp.fc2.weight', 'pred_head.transforms.0.1.mlp.fc2.bias', 'pred_head.transforms.0.2.norm1.weight', 'pred_head.transforms.0.2.norm1.bias', 'pred_head.transforms.0.2.attn.q.weight', 'pred_head.transforms.0.2.attn.q.bias', 'pred_head.transforms.0.2.attn.k.weight', 'pred_head.transforms.0.2.attn.k.bias', 'pred_head.transforms.0.2.attn.v.weight', 'pred_head.transforms.0.2.attn.v.bias', 'pred_head.transforms.0.2.attn.proj.weight', 'pred_head.transforms.0.2.attn.proj.bias', 'pred_head.transforms.0.2.norm2.weight', 'pred_head.transforms.0.2.norm2.bias', 'pred_head.transforms.0.2.mlp.fc1.weight', 'pred_head.transforms.0.2.mlp.fc1.bias', 'pred_head.transforms.0.2.mlp.fc2.weight', 'pred_head.transforms.0.2.mlp.fc2.bias', 'pred_head.transforms.0.3.norm1.weight', 'pred_head.transforms.0.3.norm1.bias', 'pred_head.transforms.0.3.attn.q.weight', 'pred_head.transforms.0.3.attn.q.bias', 'pred_head.transforms.0.3.attn.k.weight', 'pred_head.transforms.0.3.attn.k.bias', 'pred_head.transforms.0.3.attn.v.weight', 'pred_head.transforms.0.3.attn.v.bias', 'pred_head.transforms.0.3.attn.proj.weight', 'pred_head.transforms.0.3.attn.proj.bias', 'pred_head.transforms.0.3.norm2.weight', 'pred_head.transforms.0.3.norm2.bias', 'pred_head.transforms.0.3.mlp.fc1.weight', 'pred_head.transforms.0.3.mlp.fc1.bias', 'pred_head.transforms.0.3.mlp.fc2.weight', 'pred_head.transforms.0.3.mlp.fc2.bias', 'pred_head.transforms.0.4.weight', 'pred_head.transforms.0.4.bias', 'pred_head.projections.0.weight', 'pred_head.projections.0.bias',])

[Yiming-M]

I guess these are for the decoding blocks and the final prediction head, but if so then there will only be 5 decoding blocks instead of 8 used in paper.

[Yiming-M]

Hi @PardoAlejo,

Just double-checked the checkpoint. In cfg there is a hyper-parameter decoder_depth under the mask (the starting index is 4429), so I guess the decoder they used in experiment only has 4 blocks instead of 8. Also for pred_head.transforms.0.4.weight and pred_head.transforms.0.4.bias, since there is no other parameters with like attn, norm1, I guess this block pred_head.transforms.0.4 is the layer norm of the decoder.

Best,
Yiming

[tung-nd]

HI @Yiming-M, I followed what you did and was able to load the pred_head weights, but I'm still having some incompatibilities when loading the checkpoints:

_IncompatibleKeys(missing_keys=['decoder_cls_token', 'decoder_pos_embed_spatial', 'decoder_pos_embed_temporal', 'decoder_pos_embed_class'], unexpected_keys=['decoder_pos_embed'])

Did you also have this issue and how did you resolve it?

[innat]

cc. @haooooooqi @weiwangmeta @feichtenhofer
Please address this. Which is correct, the model definition or the pre-trained weights? In code, here, while loading the model weight to arch the strict=False is set. Is it intentional?

# weights
wg = 'mae_pretrain_vit_large_k400.pth'
checkpoint = torch.load(
     wg, map_location="cpu"
)
checkpoint_model = checkpoint["model_state"]

pred_head.transforms.0.0.attn.q.weight torch.Size([512, 512])
pred_head.transforms.0.0.attn.q.bias torch.Size([512])
pred_head.transforms.0.0.attn.k.weight torch.Size([512, 512])
pred_head.transforms.0.0.attn.k.bias torch.Size([512])
pred_head.transforms.0.0.attn.v.weight torch.Size([512, 512])
pred_head.transforms.0.0.attn.v.bias torch.Size([512])
pred_head.transforms.0.0.attn.proj.weight torch.Size([512, 512])
pred_head.transforms.0.0.attn.proj.bias torch.Size([512])
pred_head.transforms.0.0.norm2.weight torch.Size([512])
pred_head.transforms.0.0.norm2.bias torch.Size([512])
pred_head.transforms.0.0.mlp.fc1.weight torch.Size([2048, 512])
pred_head.transforms.0.0.mlp.fc1.bias torch.Size([2048])
pred_head.transforms.0.0.mlp.fc2.weight torch.Size([512, 2048])
pred_head.transforms.0.0.mlp.fc2.bias torch.Size([512])

# model
model_pt = mae_vit_large_patch16()
state_dict = model_pt.state_dict()

decoder_blocks.0.attn.k.bias torch.Size([512])
decoder_blocks.0.attn.k.weight torch.Size([512, 512])
decoder_blocks.0.attn.proj.bias torch.Size([512])
decoder_blocks.0.attn.proj.weight torch.Size([512, 512])
decoder_blocks.0.attn.q.bias torch.Size([512])
decoder_blocks.0.attn.q.weight torch.Size([512, 512])
decoder_blocks.0.attn.v.bias torch.Size([512])
decoder_blocks.0.attn.v.weight torch.Size([512, 512])
decoder_blocks.0.mlp.fc1.bias torch.Size([2048])
decoder_blocks.0.mlp.fc1.weight torch.Size([2048, 512])
decoder_blocks.0.mlp.fc2.bias torch.Size([512])
decoder_blocks.0.mlp.fc2.weight torch.Size([512, 2048])
decoder_blocks.0.norm1.bias torch.Size([512])
decoder_blocks.0.norm1.weight torch.Size([512])
decoder_blocks.0.norm2.bias torch.Size([512])
decoder_blocks.0.norm2.weight torch.Size([512])

[innat]

facebookresearch/SlowFast#668