Custom image size for pretrained models

[mxc19912008]

Hi,

I am trying to use Transformer in transformer model and the default image size is 224, is there a way to use pretrained models but change the image size to 320 or other sizes?

Thanks!

[rwightman]

Yes, you need to pass a different img_size when you create the model, it will interpolate the position embedding when it loads the pretrained weights. It needs to be evenly divisible by the patch size. It should work with the vit, vit_deit, vit_deit_distilled. Has not been implemented for pit, swin, and tnt yet.

However you cannot just change the input image on the fly like other models, there is a bit of loss in the interpolation so it relies on fine-tuning to bring the accuracy up on the different sized model. The current train scripts don't have the img size arg plumbed through (would break the convnets right now), so you if you use the scripts here you need to manually hack the img_size arg in.

>>> m = timm.create_model('vit_deit_small_patch16_224', img_size=256, pretrained=True)
>>> m.pos_embed.shape
torch.Size([1, 257, 384])

[mxc19912008]

Thanks! I was able to change other parameters like patch size to make the image size larger, in this way the parameters of larger image size model will stay the same with those in the checkpoint.

[JohnG0024]

@rwightman Hi, I'd like to use vit_large_patch16_384 but the default image size is 384, while my images are 896. Should I do the same as

>>> m = timm.create_model('vit_large_patch16_384', img_size=896, pretrained=True)
>>> m.pos_embed.shape
torch.Size([1, 3137, 384])

Or is there another pretrained model I should use?

[rwightman]

@JohnG0024 that's the closest one, but big warning, it's going to be insanely expensive to train! these models do not scale well with resolution

[JohnG0024]

@rwightman How expensive is it compared to training without a pre-trained model? Should I try tf_efficientnet_l2_ns instead? Its input shape is (3, 800, 800)

[deshwalmahesh]

Yes, you need to pass a different img_size when you create the model, it will interpolate the position embedding when it loads the pretrained weights. It needs to be evenly divisible by the patch size. It should work with the vit, vit_deit, vit_deit_distilled. Has not been implemented for pit, swin, and tnt yet.

However you cannot just change the input image on the fly like other models, there is a bit of loss in the interpolation so it relies on fine-tuning to bring the accuracy up on the different sized model. The current train scripts don't have the img size arg plumbed through (would break the convnets right now), so you if you use the scripts here you need to manually hack the img_size arg in.

>>> m = timm.create_model('vit_deit_small_patch16_224', img_size=256, pretrained=True)
>>> m.pos_embed.shape
torch.Size([1, 257, 384])

Hi, I am testing the SWIN models on down steam tasks like segmentation and others. Problem is that in their approach for segmentation, they used the UperNet with image size of 512 but here we have a maximum size of 256. Let us suppose I want to use it as a backbone for DeepLabV3+ and need to use bigger size images then what could be the best possible solution doing this? Do I have to just change the argument img_size = 512 or there is some other solution?