Choice of clip In caption_dataset

[kszpxxzmc]

Why feat_caption and token_caption Use caption_clip's seq and caption_cam_clip's token_caption in CLaTr/src/datasets/modalities/caption_dataset.py. Why Not Use caption_cam_clip's seq and token for Both?

[robincourant]

Hi,
Token (i.e. cls token of CLIP) and seq (i.e. sequential tokens of CLIP) are both used, one to filter out captions that are too similar during the contrastive loss calculation and the other as CLaTr's input features.