[Feature Request] Image Augmentation Is All You Need benchmark

[vmoens]

Nice benchmark to have as it would interact with other torch libraries

http://arxiv.org/abs/2004.13649

[yemaedahrav]

Hi, I would be interested in taking up this issue. Should I assign myself on this?

[vmoens]

That would be lovely!
From what I understood the main technical difficulty is to do transforms in a consistent manner (eg, have an action mapping operator that follows the image/state transform etc).
Not sure what form these will take but happy to help or provide some guidance if needed.
If you want to discuss this not on gh, feel free to join our slack channel or discord (but gh is also perfectly fine).

[yemaedahrav]

I did go through the /benchmarks directory (including the commit history) and take a look at the code there and had read the paper as well. I feel I would still require more clarity on the goals this issue attempts to accomplish. Is it to get a working implementation integrated into the TorchRL library or am I missing something?

[vmoens]

I would say that the primary goal is that if anyone wants to build on top of this work a new research paper or an application (eg apply it to a different domain or else) it should be doable.
The secondary goals are:

• provide the set of primitives that are needed for this. In this case I think these will be transforms to append to the environment.
• provide a script that reproduces some of the paper's result to showcase how those primitives should be used.

This hierarchy of values is important to keep in mind while implementing all this: we don't want to write a new benchmark for the sake of the script, we want the benchmark to show how to use what that paper proposes using torchrl.

I hope that makes sense!

[neo-alex]

I am looking forward to this DrQ implementation!

Just wondering if it wouldn't be better to rather reproduce DrQv2 directly? It should be the same amount of work but the performance is vastly superior, as shown in the first illustration of the paper: https://arxiv.org/abs/2107.09645 (the image augmentation trick is the same but the v2 made a couple of changes wrt. hyperparameters, replacing SAC with DDPG...).

By the way, you can find the official implementation of DrQv2 here as an inspiration:
https://github.com/facebookresearch/drqv2

...and I believe that the core image augmentation trick is implemented here:
https://github.com/facebookresearch/drqv2/blob/main/drqv2.py#L14