High-Resolution Imagery, Sub-pixel accuracy, and Flow Covariance

[ChenYutongTHU]

Dear authors,

Great thanks for this wonderful work! May I have two questions?

1. Sub-pixel accuracy.

I see that the model downsamples images to 460x520 by default. This discards high-resolution imagery and may result in suboptimal accuracy.
In fact, in my experiments, it underperforms RoMa a lot, possibly because RoMa could process the native resolution, e.g. Full HD.

I wonder if how I could boost the sub-pixel accuracy. The paper mentions that there is a UFM980 model. Where can I find the weights and inference code of this model?

2. Uncertainty or Flow covariance.

The current code predicts covisibility score. Perceptually, this is not exactly the same as the model's confidence regarding the flow accuracy. I guess the 'flow covariance' seems a more relevant descriptor, but the current model does not contain head to produce 'flow covariance'.

Thanks so much for your help!

[infinity1096]

Thank you for your interest in our work and your effort in comparing it to other baselines! A few clarifications regarding your questions:

Sub-pixel accuracy for 560 model at high resolution input

• If you inference the 560-trained model directly at high resolution, the UFM560 model is indeed not performing particularly well. We observed that it does not extend to significantly different resolutions than those seen in training. To address this, we used a nearest interpolation to scale the flow for demo purposes, which is suboptimal for high-resolution scenarios.
  For this reason, we finetuned it to the 980 model as you mentioned. We are preparing them for release, and we will add them to the repository soon (< 1 month).
• When evaluating UFM’s output, did you exclude the non-covisible regions? UFM560 is not trained on non-covisible regions, so predictions there are essentially arbitrary. (UFM980 is trained to have a valid output everywhere).
• Be sure to use UFM560-Refine for its improved sub-pixel accuracy over the base model.
• Regarding your statement that UFM “underperforms RoMa a lot,” we believe this reflects a misunderstanding. During our evaluation, we also used RoMa’s upsample version, and under comparable conditions, UFM performs on par with or slightly better than RoMa. We do observe that RoMa’s output is more locally consistent, but not necessarily more accurate.

Uncertainty / Flow covariance

Thanks for the question. We actually spent a long time on the covariance head, but it does not work very well yet.

• We consider covisibility to be a more useful quantity in selecting good points(for pose estimation). We did try a version with a covariance head, both with backbone frozen or finetuned, but found it biased toward the center of the image regardless. If you use it to select points, the points will be cluttered and will not provide well-distributed constraints.
• We have integrated UFM with a VO system (MAC-VO) and found that covisibility provides a better estimation of confidence than covariance.
• We would be happy to learn about your application for flow covariance beyond pose estimation and share the covariance head with you once we have it working well.

Thanks again for your interest, and let me know how we can further assist.

[ChenYutongTHU]

Dear authors,

Thanks so much for the detailed reply. This helps me a lot!
I'm looking forward to the release of UFM980 models! 🎉

[infinity1096]

Hi Yutong,

The 980 resolution checkpoints are updated. You can take a look at the demo code. Basically, they have the same interface as before despite they inference at 980 x 644 resolution internally.