Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Discontinue support for "hybrid" API stack (using RLModule + Learner, but still on RolloutWorker and Policy) #46085

Merged
merged 51 commits into from
Sep 27, 2024
Merged

[RLlib] Discontinue support for "hybrid" API stack (using RLModule + Learner, but still on RolloutWorker and Policy) #46085

merged 51 commits into from
Sep 27, 2024

Conversation

sven1977
Copy link
Contributor

@sven1977 sven1977 commented Jun 17, 2024 •
edited
Loading

Discontinue support for "hybrid" API stack (using RLModule + Learner, but still on RolloutWorker and Policy)

  • The config setting combination of enable_rl_module_and_learner=True AND enable_env_runner_and_connector_v2=False is no longer allowed and will create an error.

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@sven1977
wip
cd7cf04 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into depr…
c32287e 
…ecate_hybrid_api_stack

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/BUILD
#	rllib/algorithms/impala/tests/test_impala_learner.py
#	rllib/algorithms/impala/tests/test_impala_off_policyness.py
#	rllib/algorithms/ppo/tests/test_ppo_with_env_runner.py
#	rllib/tuned_examples/impala/cartpole-impala.yaml
@sven1977
wip
396ddf7 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
9ff3d6d 
Signed-off-by: sven1977 <[email protected]>
simonsays1980
simonsays1980 approved these changes Jun 20, 2024
View reviewed changes
Copy link
Collaborator

@simonsays1980 simonsays1980 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please check, if we can remove in the same breath the view_requirement from core.models.Catalog and the view_requirements-related methods from RLModule. In addition: Can't we with this change also deprecate TensorFlow in the new stack?

rllib/algorithms/algorithm_config.py
# Disabled hybrid API stack. Now, both `enable_rl_module_and_learner` and
# `enable_env_runner_and_connector_v2` must be True or both False.
if not self.enable_env_runner_and_connector_v2:
raise ValueError(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I smell a new world rising .... :)

rllib/algorithms/bc/tests/test_bc.py Outdated
@@ -51,7 +51,7 @@ def test_bc_compilation_and_learning_from_offline_file(self):
min_return_to_reach = 75.0

# Test for RLModule API and ModelV2.
for rl_modules in [True, False]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yessss

rllib/policy/policy.py
for key, view_req in self.view_requirements.items():
if key not in self._dummy_batch.accessed_keys:
view_req.used_for_compute_actions = False
for key, view_req in self.view_requirements.items():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to remove now also the view_requirements from the Catalog, shouldn't we? In addition remove all the view_requirement methods in the RLModule

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into depr…
30faf81 
…ecate_hybrid_api_stack
@sven1977
wip
1b97844 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
4d2352b 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into depr…
89497fb 
…ecate_hybrid_api_stack

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/algorithms/algorithm.py
#	rllib/algorithms/bc/tests/test_bc.py
#	rllib/algorithms/ppo/ppo.py
#	rllib/algorithms/ppo/tests/test_ppo.py
#	rllib/algorithms/ppo/tests/test_ppo_with_env_runner.py
#	rllib/algorithms/tests/test_algorithm_config.py
#	rllib/core/testing/tests/test_bc_algorithm.py
#	rllib/evaluation/postprocessing.py
#	rllib/evaluation/rollout_worker.py
#	rllib/evaluation/tests/test_trajectory_view_api.py
#	rllib/models/tests/test_preprocessors.py
#	rllib/policy/eager_tf_policy_v2.py
#	rllib/policy/policy.py
#	rllib/policy/tests/test_compute_log_likelihoods.py
#	rllib/policy/tests/test_policy.py
#	rllib/tests/test_algorithm_rl_module_restore.py
#	rllib/utils/exploration/tests/test_explorations.py
@sven1977 sven1977 enabled auto-merge (squash) September 17, 2024 10:13
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Sep 17, 2024
@sven1977
LINT
26b1757 
Signed-off-by: sven1977 <[email protected]>
@sven1977
fix
c92bec8 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
46535ec 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
dd16b54 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into depr…
2fe34e0 
…ecate_hybrid_api_stack

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/algorithms/ppo/ppo.py
#	rllib/algorithms/ppo/tests/test_ppo.py
#	rllib/algorithms/ppo/tests/test_ppo_with_env_runner.py
#	rllib/core/learner/learner_group.py
#	rllib/examples/multi_agent/self_play_league_based_with_open_spiel.py
#	rllib/examples/rl_modules/classes/mobilenet_rlm.py
#	rllib/models/tests/test_preprocessors.py
@sven1977
wip
56564fd 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into swit…
a14dd3e 
…ch_on_new_api_stack_by_default_for_sac_and_dqn

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/algorithms/dqn/tests/test_dqn.py
#	rllib/algorithms/ppo/tests/test_repro_ppo.py
#	rllib/algorithms/sac/sac.py
#	rllib/algorithms/sac/tests/test_rnnsac.py
#	rllib/algorithms/sac/tests/test_sac.py
@sven1977
fix
3b08419 
Signed-off-by: sven1977 <[email protected]>
@sven1977
fix
15ca1cb 
Signed-off-by: sven1977 <[email protected]>
@sven1977
fix
706ec70 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
e1fae2c 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
6f9d09f 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into swit…
a0fb060 
…ch_on_new_api_stack_by_default_for_sac_and_dqn
@sven1977
fix
9674b45 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
f4bbec2 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into depr…
a049042 
…ecate_hybrid_api_stack

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	doc/source/rllib/doc_code/training.py
#	rllib/algorithms/cql/cql.py
#	rllib/algorithms/dqn/dqn.py
#	rllib/algorithms/ppo/tests/test_ppo.py
#	rllib/algorithms/ppo/tests/test_ppo_old_api_stack.py
#	rllib/algorithms/sac/sac.py
@sven1977 sven1977 enabled auto-merge (squash) September 26, 2024 15:49
@sven1977
fixes
e0521a1 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 enabled auto-merge (squash) September 26, 2024 17:23
@sven1977
wip
68e2642 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
7bdc1d6 
Signed-off-by: sven1977 <[email protected]>
@sven1977
fixes
5e2fb85 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
e2226a0 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 merged commit c9fa046 into ray-project:master Sep 27, 2024
5 checks passed
@sven1977 sven1977 deleted the deprecate_hybrid_api_stack branch September 27, 2024 15:32
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
c4ce0f7 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
f33434b 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
b0d2b8c 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
39d34bc 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
344d8dc 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
2c125bc 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
4c5a08f 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
417cdd2 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Discontinue support for "hybrid" API stack (using RLModule + …
5e7601b 
…Learner, but still on RolloutWorker and Policy) (ray-project#46085)

Signed-off-by: ujjawal-khare <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonsays1980 simonsays1980 simonsays1980 approved these changes

@ArturNiederfahrenhorst ArturNiederfahrenhorst Awaiting requested review from ArturNiederfahrenhorst

@maxpumperla maxpumperla Awaiting requested review from maxpumperla maxpumperla is a code owner

Assignees

@simonsays1980 simonsays1980

Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sven1977 @simonsays1980