Training Code

[Maxye112]

Hi, thank you for your excellent work again. I want to know when will you release the training code?
Or maybe it's like CUT3R's?
Thank you so much!!!!

[LiZizun]

Sorry for the wait! I've been busy and the old code needed cleaning. I just pushed the training code to the train branch.

It might still have small bugs and lacks a README, but you can use it as a reference for now. I'll update it later.

My framework is structurally similar to Pi3 (except for data, model, and loss), so you can check that for reference too. The core training logic is all in the branch now. Let me know if you run into issues!

[runnerhdh]

Sorry for the wait! I've been busy and the old code needed cleaning. I just pushed the training code to the train branch.

It might still have small bugs and lacks a README, but you can use it as a reference for now. I'll update it later.

My framework is structurally similar to Pi3 (except for data, model, and loss), so you can check that for reference too. The core training logic is all in the branch now. Let me know if you run into issues!

Excellent work！My question is, during the second training phase (60-frames), I noticed that the state update didn't use a TBPTT strategy similar to Cut3R. Will directly optimizing the state on the 60-frames cause gradient instability? Looking forward to your reply, thinks :)

[LiZizun]

May I ask what your dataset looks like? I utilized a large amount of data during the first stage, and I only proceed to the second stage using 60-frame sequences once the first stage is well-trained. For the second stage, using a window size of 4 requires the data to maintain some temporal continuity; feeding frames in a completely random order will cause training instability.

[runnerhdh]

May I ask what your dataset looks like? I utilized a large amount of data during the first stage, and I only proceed to the second stage using 60-frame sequences once the first stage is well-trained. For the second stage, using a window size of 4 requires the data to maintain some temporal continuity; feeding frames in a completely random order will cause training instability.

My dataset is the same as cut3r's. Cut3r's code uses tbptt gradient truncation when training sequences exceeding 32 frames, training only the last 16 frames, and cutting the last 16 frames into 4 chunks. The gradients between each chunk are detached. I asked the author about this, and they said that detaching and tbptt are for stable training, especially for dynamic sequences. I don't see similar operations in your code (https://github.com/LiZizun/WinT3R/blob/train/trainers/trainer.py#L76, maybe I missed it.).
So I'm curious: when you train 60 frames in the second stage, are the gradients of the state completely backpropagated from the last frame to the first frame? Does this cause the problem mentioned by the cut3r author?
Looking forward to your reply, thinks :)

[LiZizun]

I didn't use tbptt strategy in all training stages. and the gradients of the state completely backpropagated from the last frame to the first frame. During the first stage of training, to enhance robustness, I used a mixture of temporally random data and sequentially ordered video data. In the second training stage, I exclusively used relatively continuous video data and applied a smaller learning rate. No gradient instability occurred during the second stage.