[Feature] SAC Trainer

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Documentation #3192
• -> [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [CI] LLM tests integration #3216
• [Quality] Fix flaky test #3211

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3191

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures, 1 Unrelated Failure

As of commit 6ad5bc6 with merge base 963fdd4 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t d20ab1860a928af783016771ae06e2ad159afe9359e056fc7cb73656628352a0 /exec failed with exit code 1
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 6a0c1e4e7cf5a0761c8917a23970ea55a699a7d2b88230b67bddef1a72b06ede /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 49c62d0250b22bb1f837bdcaca0136ac259b2a7039a07d43d66b6f196a3e1329 /exec failed with exit code 1
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t ab04c539b2714aee5f43ed90d45657386dff8740e1f62ea18667eddebd5c19b8 /exec failed with exit code 1
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 5c68605e84cec0dc90a3e8bb407d2cfd3fa92f7954ae6abcc9446a3a5d5cea2f /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_configs.py::TestHydraParsing::test_trainer_parsing_with_file
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_configs.py::TestHydraParsing::test_trainer_parsing_with_file
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_configs.py::TestHydraParsing::test_trainer_parsing_with_file
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_configs.py::TestHydraParsing::test_trainer_parsing_with_file
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_configs.py::TestHydraParsing::test_trainer_parsing_with_file

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_collector.py::TestPolicyFactory::test_weight_update

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}20$. Worsened: $\large\color{#d91a1a}14$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	80.6150μs	79.3546μs	12.6017 KOps/s	11.7432 KOps/s	$\textbf{\color{#35bf28}+7.31\%}$
test_tensor_to_bytestream_speed[torch.save]	0.1378ms	0.1365ms	7.3261 KOps/s	7.2128 KOps/s	$\color{#35bf28}+1.57\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1166s	0.1157s	8.6429 Ops/s	8.5856 Ops/s	$\color{#35bf28}+0.67\%$
test_tensor_to_bytestream_speed[numpy]	2.8315μs	2.8269μs	353.7452 KOps/s	361.7272 KOps/s	$\color{#d91a1a}-2.21\%$
test_tensor_to_bytestream_speed[safetensors]	43.3081μs	41.7048μs	23.9781 KOps/s	22.4393 KOps/s	$\textbf{\color{#35bf28}+6.86\%}$
test_simple	0.5441s	0.5422s	1.8445 Ops/s	1.7586 Ops/s	$\color{#35bf28}+4.88\%$
test_transformed	1.0954s	1.0944s	0.9138 Ops/s	0.8952 Ops/s	$\color{#35bf28}+2.07\%$
test_serial	1.6465s	1.6451s	0.6079 Ops/s	0.5986 Ops/s	$\color{#35bf28}+1.55\%$
test_parallel	1.1609s	1.1032s	0.9065 Ops/s	0.9459 Ops/s	$\color{#d91a1a}-4.17\%$
test_step_mdp_speed[True-True-True-True-True]	0.2538ms	44.2856μs	22.5807 KOps/s	22.6495 KOps/s	$\color{#d91a1a}-0.30\%$
test_step_mdp_speed[True-True-True-True-False]	57.3610μs	24.6409μs	40.5829 KOps/s	41.0965 KOps/s	$\color{#d91a1a}-1.25\%$
test_step_mdp_speed[True-True-True-False-True]	64.3510μs	24.6721μs	40.5317 KOps/s	40.0437 KOps/s	$\color{#35bf28}+1.22\%$
test_step_mdp_speed[True-True-True-False-False]	44.4310μs	13.4328μs	74.4444 KOps/s	74.4938 KOps/s	$\color{#d91a1a}-0.07\%$
test_step_mdp_speed[True-True-False-True-True]	0.1274ms	45.6566μs	21.9026 KOps/s	21.6729 KOps/s	$\color{#35bf28}+1.06\%$
test_step_mdp_speed[True-True-False-True-False]	73.5110μs	27.0313μs	36.9942 KOps/s	36.5367 KOps/s	$\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-True-False-False-True]	54.2510μs	27.6180μs	36.2082 KOps/s	36.5833 KOps/s	$\color{#d91a1a}-1.03\%$
test_step_mdp_speed[True-True-False-False-False]	46.9110μs	16.2159μs	61.6679 KOps/s	61.7008 KOps/s	$\color{#d91a1a}-0.05\%$
test_step_mdp_speed[True-False-True-True-True]	89.7010μs	50.1602μs	19.9361 KOps/s	20.3730 KOps/s	$\color{#d91a1a}-2.14\%$
test_step_mdp_speed[True-False-True-True-False]	63.8310μs	30.2499μs	33.0579 KOps/s	33.3587 KOps/s	$\color{#d91a1a}-0.90\%$
test_step_mdp_speed[True-False-True-False-True]	60.0010μs	27.7714μs	36.0082 KOps/s	36.0268 KOps/s	$\color{#d91a1a}-0.05\%$
test_step_mdp_speed[True-False-True-False-False]	46.7610μs	16.3593μs	61.1273 KOps/s	60.7272 KOps/s	$\color{#35bf28}+0.66\%$
test_step_mdp_speed[True-False-False-True-True]	94.4010μs	52.3700μs	19.0949 KOps/s	19.1281 KOps/s	$\color{#d91a1a}-0.17\%$
test_step_mdp_speed[True-False-False-True-False]	96.2820μs	32.3143μs	30.9461 KOps/s	30.5545 KOps/s	$\color{#35bf28}+1.28\%$
test_step_mdp_speed[True-False-False-False-True]	63.5010μs	29.8729μs	33.4752 KOps/s	33.1301 KOps/s	$\color{#35bf28}+1.04\%$
test_step_mdp_speed[True-False-False-False-False]	49.1910μs	18.8462μs	53.0610 KOps/s	51.9387 KOps/s	$\color{#35bf28}+2.16\%$
test_step_mdp_speed[False-True-True-True-True]	78.5310μs	48.9924μs	20.4113 KOps/s	20.2042 KOps/s	$\color{#35bf28}+1.03\%$
test_step_mdp_speed[False-True-True-True-False]	75.2910μs	29.8977μs	33.4473 KOps/s	32.9062 KOps/s	$\color{#35bf28}+1.64\%$
test_step_mdp_speed[False-True-True-False-True]	2.4896ms	31.1489μs	32.1039 KOps/s	31.2668 KOps/s	$\color{#35bf28}+2.68\%$
test_step_mdp_speed[False-True-True-False-False]	48.8410μs	17.9907μs	55.5842 KOps/s	55.3226 KOps/s	$\color{#35bf28}+0.47\%$
test_step_mdp_speed[False-True-False-True-True]	91.4320μs	51.6794μs	19.3501 KOps/s	19.1771 KOps/s	$\color{#35bf28}+0.90\%$
test_step_mdp_speed[False-True-False-True-False]	92.2920μs	32.2688μs	30.9897 KOps/s	30.4110 KOps/s	$\color{#35bf28}+1.90\%$
test_step_mdp_speed[False-True-False-False-True]	75.7710μs	33.8951μs	29.5028 KOps/s	29.5354 KOps/s	$\color{#d91a1a}-0.11\%$
test_step_mdp_speed[False-True-False-False-False]	51.2610μs	20.2441μs	49.3971 KOps/s	48.0262 KOps/s	$\color{#35bf28}+2.85\%$
test_step_mdp_speed[False-False-True-True-True]	94.8820μs	55.9773μs	17.8644 KOps/s	18.2175 KOps/s	$\color{#d91a1a}-1.94\%$
test_step_mdp_speed[False-False-True-True-False]	72.9410μs	35.0774μs	28.5084 KOps/s	28.3194 KOps/s	$\color{#35bf28}+0.67\%$
test_step_mdp_speed[False-False-True-False-True]	63.9910μs	33.1457μs	30.1698 KOps/s	29.3817 KOps/s	$\color{#35bf28}+2.68\%$
test_step_mdp_speed[False-False-True-False-False]	46.6110μs	20.4050μs	49.0077 KOps/s	48.6581 KOps/s	$\color{#35bf28}+0.72\%$
test_step_mdp_speed[False-False-False-True-True]	0.1326ms	56.5171μs	17.6938 KOps/s	17.6004 KOps/s	$\color{#35bf28}+0.53\%$
test_step_mdp_speed[False-False-False-True-False]	65.4010μs	37.1818μs	26.8949 KOps/s	26.0607 KOps/s	$\color{#35bf28}+3.20\%$
test_step_mdp_speed[False-False-False-False-True]	69.9710μs	34.8521μs	28.6927 KOps/s	27.6384 KOps/s	$\color{#35bf28}+3.81\%$
test_step_mdp_speed[False-False-False-False-False]	57.3710μs	22.7043μs	44.0444 KOps/s	43.2870 KOps/s	$\color{#35bf28}+1.75\%$
test_values[generalized_advantage_estimate-True-True]	10.2620ms	9.9395ms	100.6083 Ops/s	101.5799 Ops/s	$\color{#d91a1a}-0.96\%$
test_values[vec_generalized_advantage_estimate-True-True]	19.3502ms	17.5303ms	57.0441 Ops/s	57.3151 Ops/s	$\color{#d91a1a}-0.47\%$
test_values[td0_return_estimate-False-False]	0.2062ms	0.1285ms	7.7796 KOps/s	7.9133 KOps/s	$\color{#d91a1a}-1.69\%$
test_values[td1_return_estimate-False-False]	28.1205ms	27.4427ms	36.4396 Ops/s	36.7048 Ops/s	$\color{#d91a1a}-0.72\%$
test_values[vec_td1_return_estimate-False-False]	18.3547ms	17.6580ms	56.6316 Ops/s	57.1264 Ops/s	$\color{#d91a1a}-0.87\%$
test_values[td_lambda_return_estimate-True-False]	42.0123ms	40.8940ms	24.4534 Ops/s	24.5929 Ops/s	$\color{#d91a1a}-0.57\%$
test_values[vec_td_lambda_return_estimate-True-False]	17.9761ms	17.5760ms	56.8958 Ops/s	57.2960 Ops/s	$\color{#d91a1a}-0.70\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.7013ms	8.5661ms	116.7398 Ops/s	117.1550 Ops/s	$\color{#d91a1a}-0.35\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.7457ms	1.5332ms	652.2342 Ops/s	674.9381 Ops/s	$\color{#d91a1a}-3.36\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.5171ms	0.4173ms	2.3963 KOps/s	2.4175 KOps/s	$\color{#d91a1a}-0.87\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	34.3974ms	33.8566ms	29.5363 Ops/s	29.1311 Ops/s	$\color{#35bf28}+1.39\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	1.9516ms	1.7188ms	581.7873 Ops/s	579.5286 Ops/s	$\color{#35bf28}+0.39\%$
test_dqn_speed[False-None]	6.5660ms	1.4049ms	711.7879 Ops/s	710.5103 Ops/s	$\color{#35bf28}+0.18\%$
test_dqn_speed[False-backward]	1.9499ms	1.8928ms	528.3190 Ops/s	528.5249 Ops/s	$\color{#d91a1a}-0.04\%$
test_dqn_speed[True-None]	0.8957ms	0.4999ms	2.0002 KOps/s	1.9636 KOps/s	$\color{#35bf28}+1.87\%$
test_dqn_speed[True-backward]	0.9788ms	0.9549ms	1.0472 KOps/s	925.3447 Ops/s	$\textbf{\color{#35bf28}+13.17\%}$
test_dqn_speed[reduce-overhead-None]	0.8612ms	0.4920ms	2.0325 KOps/s	1.8884 KOps/s	$\textbf{\color{#35bf28}+7.63\%}$
test_dqn_speed[reduce-overhead-backward]	0.9838ms	0.9449ms	1.0583 KOps/s	962.2671 Ops/s	$\textbf{\color{#35bf28}+9.98\%}$
test_ddpg_speed[False-None]	3.3575ms	2.8445ms	351.5604 Ops/s	352.1373 Ops/s	$\color{#d91a1a}-0.16\%$
test_ddpg_speed[False-backward]	4.2160ms	4.0488ms	246.9894 Ops/s	246.8919 Ops/s	$\color{#35bf28}+0.04\%$
test_ddpg_speed[True-None]	1.4742ms	1.3634ms	733.4517 Ops/s	716.2954 Ops/s	$\color{#35bf28}+2.40\%$
test_ddpg_speed[True-backward]	2.3733ms	2.3352ms	428.2304 Ops/s	357.2481 Ops/s	$\textbf{\color{#35bf28}+19.87\%}$
test_ddpg_speed[reduce-overhead-None]	1.4791ms	1.3473ms	742.2364 Ops/s	732.3459 Ops/s	$\color{#35bf28}+1.35\%$
test_ddpg_speed[reduce-overhead-backward]	2.4125ms	2.3416ms	427.0573 Ops/s	367.2927 Ops/s	$\textbf{\color{#35bf28}+16.27\%}$
test_sac_speed[False-None]	8.3779ms	7.8424ms	127.5126 Ops/s	129.3049 Ops/s	$\color{#d91a1a}-1.39\%$
test_sac_speed[False-backward]	11.6480ms	11.0974ms	90.1108 Ops/s	91.2155 Ops/s	$\color{#d91a1a}-1.21\%$
test_sac_speed[True-None]	2.7957ms	2.0942ms	477.5110 Ops/s	478.7894 Ops/s	$\color{#d91a1a}-0.27\%$
test_sac_speed[True-backward]	4.1303ms	3.9854ms	250.9160 Ops/s	248.8166 Ops/s	$\color{#35bf28}+0.84\%$
test_sac_speed[reduce-overhead-None]	2.4557ms	2.0750ms	481.9279 Ops/s	473.8963 Ops/s	$\color{#35bf28}+1.69\%$
test_sac_speed[reduce-overhead-backward]	4.2742ms	4.0347ms	247.8475 Ops/s	219.8709 Ops/s	$\textbf{\color{#35bf28}+12.72\%}$
test_redq_speed[False-None]	15.3552ms	10.4895ms	95.3334 Ops/s	95.4301 Ops/s	$\color{#d91a1a}-0.10\%$
test_redq_speed[False-backward]	25.0326ms	18.5105ms	54.0235 Ops/s	57.4508 Ops/s	$\textbf{\color{#d91a1a}-5.97\%}$
test_redq_speed[True-None]	6.0777ms	4.4648ms	223.9721 Ops/s	235.5566 Ops/s	$\color{#d91a1a}-4.92\%$
test_redq_speed[True-backward]	9.9567ms	9.7223ms	102.8558 Ops/s	107.7428 Ops/s	$\color{#d91a1a}-4.54\%$
test_redq_speed[reduce-overhead-None]	4.6116ms	4.3224ms	231.3507 Ops/s	220.9435 Ops/s	$\color{#35bf28}+4.71\%$
test_redq_speed[reduce-overhead-backward]	10.0853ms	9.8426ms	101.5992 Ops/s	99.5935 Ops/s	$\color{#35bf28}+2.01\%$
test_redq_deprec_speed[False-None]	11.2068ms	10.8187ms	92.4328 Ops/s	92.0208 Ops/s	$\color{#35bf28}+0.45\%$
test_redq_deprec_speed[False-backward]	16.2127ms	15.5305ms	64.3893 Ops/s	63.6629 Ops/s	$\color{#35bf28}+1.14\%$
test_redq_deprec_speed[True-None]	3.8444ms	3.6026ms	277.5802 Ops/s	270.4391 Ops/s	$\color{#35bf28}+2.64\%$
test_redq_deprec_speed[True-backward]	7.8458ms	7.5095ms	133.1649 Ops/s	117.1483 Ops/s	$\textbf{\color{#35bf28}+13.67\%}$
test_redq_deprec_speed[reduce-overhead-None]	3.7811ms	3.5577ms	281.0826 Ops/s	283.0518 Ops/s	$\color{#d91a1a}-0.70\%$
test_redq_deprec_speed[reduce-overhead-backward]	7.7381ms	7.5191ms	132.9944 Ops/s	127.4055 Ops/s	$\color{#35bf28}+4.39\%$
test_td3_speed[False-None]	8.2686ms	7.8683ms	127.0926 Ops/s	127.7301 Ops/s	$\color{#d91a1a}-0.50\%$
test_td3_speed[False-backward]	11.6347ms	10.8322ms	92.3178 Ops/s	93.6485 Ops/s	$\color{#d91a1a}-1.42\%$
test_td3_speed[True-None]	1.8275ms	1.7549ms	569.8451 Ops/s	559.8546 Ops/s	$\color{#35bf28}+1.78\%$
test_td3_speed[True-backward]	3.7366ms	3.5643ms	280.5581 Ops/s	277.6267 Ops/s	$\color{#35bf28}+1.06\%$
test_td3_speed[reduce-overhead-None]	1.7662ms	1.7265ms	579.2024 Ops/s	575.5131 Ops/s	$\color{#35bf28}+0.64\%$
test_td3_speed[reduce-overhead-backward]	3.6830ms	3.5604ms	280.8699 Ops/s	240.5318 Ops/s	$\textbf{\color{#35bf28}+16.77\%}$
test_cql_speed[False-None]	26.7160ms	25.5279ms	39.1728 Ops/s	39.1726 Ops/s	$+0.00\%$
test_cql_speed[False-backward]	37.9070ms	34.7647ms	28.7648 Ops/s	29.0834 Ops/s	$\color{#d91a1a}-1.10\%$
test_cql_speed[True-None]	12.7372ms	12.1545ms	82.2738 Ops/s	83.3683 Ops/s	$\color{#d91a1a}-1.31\%$
test_cql_speed[True-backward]	18.2418ms	17.8458ms	56.0355 Ops/s	57.8037 Ops/s	$\color{#d91a1a}-3.06\%$
test_cql_speed[reduce-overhead-None]	12.7137ms	12.3970ms	80.6646 Ops/s	82.3048 Ops/s	$\color{#d91a1a}-1.99\%$
test_cql_speed[reduce-overhead-backward]	18.6896ms	18.0779ms	55.3162 Ops/s	58.2380 Ops/s	$\textbf{\color{#d91a1a}-5.02\%}$
test_a2c_speed[False-None]	5.8703ms	5.4468ms	183.5940 Ops/s	189.4871 Ops/s	$\color{#d91a1a}-3.11\%$
test_a2c_speed[False-backward]	12.2108ms	11.7966ms	84.7701 Ops/s	86.4139 Ops/s	$\color{#d91a1a}-1.90\%$
test_a2c_speed[True-None]	4.0387ms	3.6658ms	272.7887 Ops/s	265.7775 Ops/s	$\color{#35bf28}+2.64\%$
test_a2c_speed[True-backward]	8.8670ms	8.6094ms	116.1516 Ops/s	106.5370 Ops/s	$\textbf{\color{#35bf28}+9.02\%}$
test_a2c_speed[reduce-overhead-None]	3.8575ms	3.6934ms	270.7499 Ops/s	270.9100 Ops/s	$\color{#d91a1a}-0.06\%$
test_a2c_speed[reduce-overhead-backward]	9.0198ms	8.7252ms	114.6108 Ops/s	92.6975 Ops/s	$\textbf{\color{#35bf28}+23.64\%}$
test_ppo_speed[False-None]	6.5782ms	5.9144ms	169.0785 Ops/s	165.1559 Ops/s	$\color{#35bf28}+2.38\%$
test_ppo_speed[False-backward]	12.8343ms	12.4025ms	80.6292 Ops/s	81.2465 Ops/s	$\color{#d91a1a}-0.76\%$
test_ppo_speed[True-None]	3.7914ms	3.6279ms	275.6401 Ops/s	267.6812 Ops/s	$\color{#35bf28}+2.97\%$
test_ppo_speed[True-backward]	8.6178ms	8.4009ms	119.0352 Ops/s	113.6109 Ops/s	$\color{#35bf28}+4.77\%$
test_ppo_speed[reduce-overhead-None]	3.9672ms	3.5812ms	279.2387 Ops/s	280.3165 Ops/s	$\color{#d91a1a}-0.38\%$
test_ppo_speed[reduce-overhead-backward]	9.0384ms	8.7580ms	114.1818 Ops/s	110.5215 Ops/s	$\color{#35bf28}+3.31\%$
test_reinforce_speed[False-None]	5.0393ms	4.5707ms	218.7838 Ops/s	221.1937 Ops/s	$\color{#d91a1a}-1.09\%$
test_reinforce_speed[False-backward]	7.4995ms	7.3259ms	136.5021 Ops/s	138.0232 Ops/s	$\color{#d91a1a}-1.10\%$
test_reinforce_speed[True-None]	3.2789ms	2.8308ms	353.2582 Ops/s	345.5881 Ops/s	$\color{#35bf28}+2.22\%$
test_reinforce_speed[True-backward]	7.8714ms	7.6723ms	130.3391 Ops/s	125.5157 Ops/s	$\color{#35bf28}+3.84\%$
test_reinforce_speed[reduce-overhead-None]	3.1679ms	2.8125ms	355.5495 Ops/s	352.6981 Ops/s	$\color{#35bf28}+0.81\%$
test_reinforce_speed[reduce-overhead-backward]	8.2500ms	7.8877ms	126.7795 Ops/s	118.9079 Ops/s	$\textbf{\color{#35bf28}+6.62\%}$
test_iql_speed[False-None]	24.2601ms	19.9942ms	50.0145 Ops/s	50.6550 Ops/s	$\color{#d91a1a}-1.26\%$
test_iql_speed[False-backward]	36.1773ms	30.1691ms	33.1465 Ops/s	33.4264 Ops/s	$\color{#d91a1a}-0.84\%$
test_iql_speed[True-None]	8.7938ms	8.4485ms	118.3647 Ops/s	115.0957 Ops/s	$\color{#35bf28}+2.84\%$
test_iql_speed[True-backward]	17.0302ms	16.5464ms	60.4360 Ops/s	60.1072 Ops/s	$\color{#35bf28}+0.55\%$
test_iql_speed[reduce-overhead-None]	8.9468ms	8.5074ms	117.5446 Ops/s	117.0394 Ops/s	$\color{#35bf28}+0.43\%$
test_iql_speed[reduce-overhead-backward]	17.3291ms	17.0565ms	58.6286 Ops/s	58.8032 Ops/s	$\color{#d91a1a}-0.30\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.3556ms	5.8520ms	170.8832 Ops/s	171.3601 Ops/s	$\color{#d91a1a}-0.28\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6446ms	0.3584ms	2.7904 KOps/s	3.5817 KOps/s	$\textbf{\color{#d91a1a}-22.09\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5526ms	0.3251ms	3.0762 KOps/s	3.6999 KOps/s	$\textbf{\color{#d91a1a}-16.86\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.8358ms	5.6169ms	178.0332 Ops/s	181.5214 Ops/s	$\color{#d91a1a}-1.92\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.0271ms	0.3397ms	2.9442 KOps/s	3.2069 KOps/s	$\textbf{\color{#d91a1a}-8.19\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5284ms	0.3253ms	3.0737 KOps/s	3.7015 KOps/s	$\textbf{\color{#d91a1a}-16.96\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.7435ms	1.3863ms	721.3339 Ops/s	800.9725 Ops/s	$\textbf{\color{#d91a1a}-9.94\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.6634ms	1.2991ms	769.7827 Ops/s	860.5179 Ops/s	$\textbf{\color{#d91a1a}-10.54\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.9287ms	5.6967ms	175.5411 Ops/s	175.2130 Ops/s	$\color{#35bf28}+0.19\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.7703ms	0.4940ms	2.0242 KOps/s	2.2268 KOps/s	$\textbf{\color{#d91a1a}-9.10\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7825ms	0.4778ms	2.0927 KOps/s	2.3490 KOps/s	$\textbf{\color{#d91a1a}-10.91\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.6929ms	5.5740ms	179.4031 Ops/s	181.6264 Ops/s	$\color{#d91a1a}-1.22\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.6736ms	0.3401ms	2.9407 KOps/s	801.2966 Ops/s	$\textbf{\color{#35bf28}+266.99\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6174ms	0.3213ms	3.1125 KOps/s	3.8845 KOps/s	$\textbf{\color{#d91a1a}-19.87\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.8605ms	5.5424ms	180.4273 Ops/s	178.1539 Ops/s	$\color{#35bf28}+1.28\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7291ms	0.3265ms	3.0626 KOps/s	3.0384 KOps/s	$\color{#35bf28}+0.80\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5557ms	0.2914ms	3.4321 KOps/s	2.5354 KOps/s	$\textbf{\color{#35bf28}+35.36\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	9.5150ms	5.8799ms	170.0704 Ops/s	171.9957 Ops/s	$\color{#d91a1a}-1.12\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.7870ms	0.4498ms	2.2233 KOps/s	1.8483 KOps/s	$\textbf{\color{#35bf28}+20.29\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7146ms	0.4622ms	2.1638 KOps/s	1.9666 KOps/s	$\textbf{\color{#35bf28}+10.03\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.5349s	15.5828ms	64.1731 Ops/s	196.8594 Ops/s	$\textbf{\color{#d91a1a}-67.40\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	9.5211ms	2.0326ms	491.9869 Ops/s	435.0916 Ops/s	$\textbf{\color{#35bf28}+13.08\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	7.1665ms	1.1527ms	867.5175 Ops/s	844.4477 Ops/s	$\color{#35bf28}+2.73\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	6.6219ms	5.0271ms	198.9205 Ops/s	55.3084 Ops/s	$\textbf{\color{#35bf28}+259.66\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	9.4640ms	2.0226ms	494.4059 Ops/s	524.1742 Ops/s	$\textbf{\color{#d91a1a}-5.68\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	6.3670ms	1.1631ms	859.7719 Ops/s	867.4385 Ops/s	$\color{#d91a1a}-0.88\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	6.8378ms	5.2334ms	191.0817 Ops/s	187.4821 Ops/s	$\color{#35bf28}+1.92\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.4677s	11.4274ms	87.5094 Ops/s	458.4345 Ops/s	$\textbf{\color{#d91a1a}-80.91\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	1.2822ms	0.9623ms	1.0392 KOps/s	720.3805 Ops/s	$\textbf{\color{#35bf28}+44.26\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	34.3950ms	32.4686ms	30.7990 Ops/s	30.5061 Ops/s	$\color{#35bf28}+0.96\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	19.0833ms	17.3216ms	57.7315 Ops/s	58.4846 Ops/s	$\color{#d91a1a}-1.29\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	35.3050ms	33.2327ms	30.0908 Ops/s	29.7188 Ops/s	$\color{#35bf28}+1.25\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.1667ms	17.4037ms	57.4591 Ops/s	57.4199 Ops/s	$\color{#35bf28}+0.07\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	37.0372ms	35.1647ms	28.4376 Ops/s	28.4830 Ops/s	$\color{#d91a1a}-0.16\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	19.9385ms	18.7937ms	53.2094 Ops/s	52.2417 Ops/s	$\color{#35bf28}+1.85\%$