As the finetune-training progressed, the colors of the generated images became weird

[A-pril]

Hello everyone, I just wanna to finetune the SD1.5 model alike "examples/text_to_image/train_text_to_image.py" does. I have 25k images for finetune, but I just found the color of image will be weird after 10k steps. I don't know whether it means that my code is wrong or the number of training steps mismatches the number of my training data? But the loss is still going down , and I don't think it is overfitting(or maybe it is :(
here is the result in 4k steps

it's the result in 15k steps

[tolgacangoz]

Hello @A-pril. Could you share the training command?

[bghira]

you might want to try training a zero-terminal snr / v-prediction model, as they represent the training data better than traditional epsilon with linear schedule

[tolgacangoz]

How do you define weirdness here, could you elaborate? Also, could you share more information about the datasets you are using?

[A-pril]

you might want to try training a zero-terminal snr / v-prediction model, as they represent the training data better than traditional epsilon with linear schedule

Thank you, I have made a simple experiment with small data, but it seems not work well. I'll run the test on the full data set later.

[A-pril]

How do you define weirdness here, could you elaborate? Also, could you share more information about the datasets you are using?

I just fix this this problem! I used the default scheduler for inference. Now I change to DDPM scheduler and the color is normal!! Really Thanks!!!