How to save a trained model with `ppo_trainer`?

[lucascassiano]

Hi,
Great repository, very minimal and clean indeed. I'm sure many other students will learn immensely from here.

I'm currently running the gpt2 as the active model, with bhadresh-savani/distilbert-base-uncased-emotion as reward model.

How I can save the final trained model to a local directory?
Could it be done similarly to transformers Trainer:

# Huggingface transformers trainer:
from transformers import Trainer
# ...
trainer.train()
trainer.save_model("my-model")

# minRLFH trainer
ppo_trainer.train()
ppo_trainer.save_model("my-model") ## <-- possible? 🤔