Reference Fine-Tuning Code

[Palmik]

Is your feature request related to a problem? Please describe.
I am interested in fine-tuning DeepSeek V3/R1.

Describe the solution you'd like
It would be great to provide the fine-tuning code, even if it's simplistic, it would be invaluable reference for others to build upon.
MoEs have historically been tricky to fine-tune correctly (and in the case of some older MoE models, it took the community months to figure out all the bugs in the HF implementation).

[dinithaw]

Is your feature request related to a problem? Please describe.
Yes, I'm interested in fine-tuning DeepSeek V3/R1, but I’m aware that Mixture of Experts (MoEs) models can be tricky to fine-tune, especially considering the challenges the community has faced with previous models. It would be helpful to have guidance or reference code for fine-tuning to avoid potential pitfalls.

It would be invaluable if there could be fine-tuning code or a basic example provided for DeepSeek V3/R1. Even a simplistic version would go a long way in helping others in the community build upon it. MoE models often require specific adjustments, and having a working starting point or references for fine-tuning could save a lot of time and effort. It’s also worth noting that the community could benefit from any lessons learned regarding issues specific to the MoE structure.

[py-bin]

looking for Fine-Tuning Code +1

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

[LemonWei111]

looking for Fine-Tuning Code +1