[RLlib; Offline RL] Offline performance cleanup. #47731
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…'. This was initialized at each iteration and slowed down our 'OfflineData' sampling. Ina ddition tuned all Offline examples for the changes made. Signed-off-by: simonsays1980 <[email protected]>
…e added an option for users to materialize the dataset if needed and enough memory is available. Signed-off-by: simonsays1980 <[email protected]>
Signed-off-by: simonsays1980 <[email protected]>
Signed-off-by: simonsays1980 <[email protected]>
Signed-off-by: simonsays1980 <[email protected]>
…ialization and reinitialization of iterators in single-learner mode. Furthermore, changed after-mapping batch size to 1 b/c rows are then 'MultiAgentBatches' of 'train_batch_size_per_learner' environment steps each. In addition added two further options to 'AlgorithmConfig' such that users can control memory usage and performance. Signed-off-by: simonsays1980 <[email protected]>
…alue function output from 'forward_train' and did therefore not train the value function. Signed-off-by: simonsays1980 <[email protected]>
…d and is already converted to a generator we need to rebuild it. Signed-off-by: simonsays1980 <[email protected]>
simonsays1980
requested review from
sven1977 and
ArturNiederfahrenhorst
as code owners
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
| @@ -204,6 +204,12 @@ def add_rllib_example_script_args( | |||
| help="How many (tune.Tuner.fit()) experiments to execute - if possible in " | |||
| "parallel.", | |||
| ) | |||
| parser.add_argument( | |||
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
|
|
||
| # Define the config for Behavior Cloning. | ||
| config = ( | ||
| BCConfig() | ||
| .environment( | ||
| env="WrappedALE/Pong-v5", | ||
| # TODO (sven): Does this have any influence in connectors? |
There was a problem hiding this comment.
Great point! You are right and this setting is NOT propagated to the connectors. Not relevant for Pong as its rewards are all 1 anyways, but for other Atari benchmarks, this could matter.
There was a problem hiding this comment.
Thanks for the clarification. There is actually another one:
# TODO (sven): Has this any influence in the connectors?
actions_in_input_normalized=True,Does this have an influence - or should it? It is not recorgnized, yet, in the offline API.
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
sven1977
reviewed
Sep 19, 2024
Co-authored-by: Sven Mika <[email protected]> Signed-off-by: simonsays1980 <[email protected]>
Co-authored-by: Sven Mika <[email protected]> Signed-off-by: simonsays1980 <[email protected]>
Signed-off-by: simonsays1980 <[email protected]>
sven1977
approved these changes
Sep 19, 2024
sven1977
changed the title
Signed-off-by: simonsays1980 <[email protected]>
…ded anonymous filesystem to RLUnplugged example. Signed-off-by: simonsays1980 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
The
map_batchescall in offline RL learning used to be very slow for unknown reasons. This PR proposes multiple changes to the offline data pipeline to boost performance by several multitudes. These changes arematerialize_data(default isFalse) such that users can control memory usage.materialize_mapped_data(default isTrue) such htat users can control memory usage. This materialization applies theOfflinePreLearneron the raw data a priori and can be used by algorithms that do not have connector pipelines (ConnectorV2pipelines) that need up-to-dateRLModuleand/or states (e.g.BCorCQL).map_batchescall because rows contain nowMultiAgentBatches withtrain_batch_size_per_learnerenvironment steps each.In addition, it fixes an important error in
MARWIL's loss which ignored training the value function.These changes lead to enormous performance boosts:
CartPole-v1withBCin single-learner mode below 7 secs (multi-learner mode < 12 secs).CartPole-v1withMARWILin single-learner mode below 50 secs (multi-learner mode < 217 secs)Pendulum-v1" withCQL` in single-learner mode below 311 secs (multi-learner mode < 116 secs)Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.