[Feature] Weight Synchronization Schemes - Core Infrastructure

[vmoens]

Stack from ghstack (oldest at bottom):

• [Quality] Fix flaky test #3211
• [Feature] Documentation #3192
• [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [Feature] Collectors - Weight Sync Scheme Integration #3187
• [Feature] vLLM Weight Synchronization Schemes #3186
• -> [Feature] Weight Synchronization Schemes - Core Infrastructure #3185

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3185

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm][CI] Machines under the label linux.rocm.gpu.2 are undergoing maintenance.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 156. Improved: $\large\color{#35bf28}26$. Worsened: $\large\color{#d91a1a}6$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	83.2112μs	81.5465μs	12.2629 KOps/s	11.5132 KOps/s	$\textbf{\color{#35bf28}+6.51\%}$
test_tensor_to_bytestream_speed[torch.save]	0.1405ms	0.1401ms	7.1388 KOps/s	7.0809 KOps/s	$\color{#35bf28}+0.82\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1164s	0.1155s	8.6571 Ops/s	8.7639 Ops/s	$\color{#d91a1a}-1.22\%$
test_tensor_to_bytestream_speed[numpy]	2.8435μs	2.8274μs	353.6841 KOps/s	361.8523 KOps/s	$\color{#d91a1a}-2.26\%$
test_tensor_to_bytestream_speed[safetensors]	42.4019μs	42.1683μs	23.7145 KOps/s	23.3873 KOps/s	$\color{#35bf28}+1.40\%$
test_simple	0.6581s	0.5689s	1.7578 Ops/s	1.7552 Ops/s	$\color{#35bf28}+0.15\%$
test_transformed	1.2155s	1.1257s	0.8883 Ops/s	0.8861 Ops/s	$\color{#35bf28}+0.25\%$
test_serial	1.7554s	1.6700s	0.5988 Ops/s	0.5961 Ops/s	$\color{#35bf28}+0.45\%$
test_parallel	1.1803s	1.0805s	0.9255 Ops/s	0.9104 Ops/s	$\color{#35bf28}+1.65\%$
test_step_mdp_speed[True-True-True-True-True]	0.1642ms	44.3232μs	22.5616 KOps/s	22.2600 KOps/s	$\color{#35bf28}+1.35\%$
test_step_mdp_speed[True-True-True-True-False]	55.4710μs	24.9751μs	40.0399 KOps/s	39.6582 KOps/s	$\color{#35bf28}+0.96\%$
test_step_mdp_speed[True-True-True-False-True]	76.5120μs	24.9301μs	40.1121 KOps/s	38.8954 KOps/s	$\color{#35bf28}+3.13\%$
test_step_mdp_speed[True-True-True-False-False]	51.8200μs	13.7614μs	72.6671 KOps/s	70.2785 KOps/s	$\color{#35bf28}+3.40\%$
test_step_mdp_speed[True-True-False-True-True]	92.1710μs	47.2844μs	21.1486 KOps/s	21.0601 KOps/s	$\color{#35bf28}+0.42\%$
test_step_mdp_speed[True-True-False-True-False]	57.9010μs	27.6389μs	36.1809 KOps/s	35.6553 KOps/s	$\color{#35bf28}+1.47\%$
test_step_mdp_speed[True-True-False-False-True]	57.9210μs	28.1037μs	35.5825 KOps/s	35.8141 KOps/s	$\color{#d91a1a}-0.65\%$
test_step_mdp_speed[True-True-False-False-False]	53.9210μs	16.7096μs	59.8460 KOps/s	60.1124 KOps/s	$\color{#d91a1a}-0.44\%$
test_step_mdp_speed[True-False-True-True-True]	86.2120μs	50.4571μs	19.8188 KOps/s	19.9441 KOps/s	$\color{#d91a1a}-0.63\%$
test_step_mdp_speed[True-False-True-True-False]	73.3210μs	30.3102μs	32.9922 KOps/s	32.5700 KOps/s	$\color{#35bf28}+1.30\%$
test_step_mdp_speed[True-False-True-False-True]	64.4310μs	28.0652μs	35.6313 KOps/s	35.7415 KOps/s	$\color{#d91a1a}-0.31\%$
test_step_mdp_speed[True-False-True-False-False]	44.9600μs	16.7449μs	59.7197 KOps/s	60.2009 KOps/s	$\color{#d91a1a}-0.80\%$
test_step_mdp_speed[True-False-False-True-True]	86.9210μs	53.0781μs	18.8402 KOps/s	19.2484 KOps/s	$\color{#d91a1a}-2.12\%$
test_step_mdp_speed[True-False-False-True-False]	67.5810μs	33.0594μs	30.2486 KOps/s	30.7725 KOps/s	$\color{#d91a1a}-1.70\%$
test_step_mdp_speed[True-False-False-False-True]	86.2320μs	29.9045μs	33.4398 KOps/s	33.4146 KOps/s	$\color{#35bf28}+0.08\%$
test_step_mdp_speed[True-False-False-False-False]	48.9310μs	19.1331μs	52.2654 KOps/s	51.9868 KOps/s	$\color{#35bf28}+0.54\%$
test_step_mdp_speed[False-True-True-True-True]	91.7010μs	49.4648μs	20.2164 KOps/s	19.8771 KOps/s	$\color{#35bf28}+1.71\%$
test_step_mdp_speed[False-True-True-True-False]	63.9710μs	30.3288μs	32.9720 KOps/s	32.7063 KOps/s	$\color{#35bf28}+0.81\%$
test_step_mdp_speed[False-True-True-False-True]	2.4965ms	31.9555μs	31.2935 KOps/s	31.1825 KOps/s	$\color{#35bf28}+0.36\%$
test_step_mdp_speed[False-True-True-False-False]	48.5100μs	18.1527μs	55.0883 KOps/s	53.3398 KOps/s	$\color{#35bf28}+3.28\%$
test_step_mdp_speed[False-True-False-True-True]	97.2210μs	52.6023μs	19.0106 KOps/s	18.9860 KOps/s	$\color{#35bf28}+0.13\%$
test_step_mdp_speed[False-True-False-True-False]	70.2310μs	33.0346μs	30.2713 KOps/s	30.7562 KOps/s	$\color{#d91a1a}-1.58\%$
test_step_mdp_speed[False-True-False-False-True]	61.4810μs	34.2539μs	29.1938 KOps/s	29.5519 KOps/s	$\color{#d91a1a}-1.21\%$
test_step_mdp_speed[False-True-False-False-False]	61.0010μs	20.5283μs	48.7132 KOps/s	48.1001 KOps/s	$\color{#35bf28}+1.27\%$
test_step_mdp_speed[False-False-True-True-True]	90.0710μs	55.2565μs	18.0974 KOps/s	18.2247 KOps/s	$\color{#d91a1a}-0.70\%$
test_step_mdp_speed[False-False-True-True-False]	0.1032ms	35.7460μs	27.9752 KOps/s	28.3295 KOps/s	$\color{#d91a1a}-1.25\%$
test_step_mdp_speed[False-False-True-False-True]	64.8810μs	33.9857μs	29.4241 KOps/s	29.3500 KOps/s	$\color{#35bf28}+0.25\%$
test_step_mdp_speed[False-False-True-False-False]	62.5010μs	20.4967μs	48.7884 KOps/s	49.0098 KOps/s	$\color{#d91a1a}-0.45\%$
test_step_mdp_speed[False-False-False-True-True]	97.0020μs	57.1513μs	17.4974 KOps/s	17.8081 KOps/s	$\color{#d91a1a}-1.74\%$
test_step_mdp_speed[False-False-False-True-False]	0.1011ms	37.9378μs	26.3589 KOps/s	26.8744 KOps/s	$\color{#d91a1a}-1.92\%$
test_step_mdp_speed[False-False-False-False-True]	78.5110μs	35.9580μs	27.8103 KOps/s	27.7440 KOps/s	$\color{#35bf28}+0.24\%$
test_step_mdp_speed[False-False-False-False-False]	55.9510μs	23.1708μs	43.1577 KOps/s	43.3810 KOps/s	$\color{#d91a1a}-0.51\%$
test_packing[True]	0.6511s	0.6444s	1.5518 Ops/s	1.5451 Ops/s	$\color{#35bf28}+0.43\%$
test_packing[False]	0.7205s	0.7140s	1.4006 Ops/s	1.4003 Ops/s	$\color{#35bf28}+0.03\%$
test_values[generalized_advantage_estimate-True-True]	9.8305ms	9.4760ms	105.5303 Ops/s	106.3115 Ops/s	$\color{#d91a1a}-0.73\%$
test_values[vec_generalized_advantage_estimate-True-True]	12.4397ms	11.1221ms	89.9114 Ops/s	90.4955 Ops/s	$\color{#d91a1a}-0.65\%$
test_values[td0_return_estimate-False-False]	0.2245ms	0.1291ms	7.7485 KOps/s	7.8777 KOps/s	$\color{#d91a1a}-1.64\%$
test_values[td1_return_estimate-False-False]	26.2092ms	25.6683ms	38.9586 Ops/s	40.6708 Ops/s	$\color{#d91a1a}-4.21\%$
test_values[vec_td1_return_estimate-False-False]	11.9250ms	11.0690ms	90.3426 Ops/s	90.0440 Ops/s	$\color{#35bf28}+0.33\%$
test_values[td_lambda_return_estimate-True-False]	38.8877ms	38.1502ms	26.2122 Ops/s	26.9732 Ops/s	$\color{#d91a1a}-2.82\%$
test_values[vec_td_lambda_return_estimate-True-False]	11.2360ms	11.0120ms	90.8104 Ops/s	90.1855 Ops/s	$\color{#35bf28}+0.69\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.4294ms	8.2814ms	120.7530 Ops/s	123.2965 Ops/s	$\color{#d91a1a}-2.06\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.7557ms	1.5069ms	663.6009 Ops/s	656.5625 Ops/s	$\color{#35bf28}+1.07\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4625ms	0.3946ms	2.5340 KOps/s	2.5745 KOps/s	$\color{#d91a1a}-1.57\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	20.1792ms	19.6360ms	50.9268 Ops/s	51.9193 Ops/s	$\color{#d91a1a}-1.91\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	2.0766ms	1.6960ms	589.6089 Ops/s	599.6581 Ops/s	$\color{#d91a1a}-1.68\%$
test_dqn_speed[False-None]	1.7232ms	1.4060ms	711.2282 Ops/s	718.1145 Ops/s	$\color{#d91a1a}-0.96\%$
test_dqn_speed[False-backward]	1.9353ms	1.8854ms	530.3977 Ops/s	541.8830 Ops/s	$\color{#d91a1a}-2.12\%$
test_dqn_speed[True-None]	0.9182ms	0.5232ms	1.9113 KOps/s	1.8895 KOps/s	$\color{#35bf28}+1.16\%$
test_dqn_speed[True-backward]	1.0886ms	0.9549ms	1.0473 KOps/s	904.7086 Ops/s	$\textbf{\color{#35bf28}+15.76\%}$
test_dqn_speed[reduce-overhead-None]	0.8987ms	0.5089ms	1.9649 KOps/s	1.9152 KOps/s	$\color{#35bf28}+2.59\%$
test_dqn_speed[reduce-overhead-backward]	0.9671ms	0.9415ms	1.0622 KOps/s	1.0882 KOps/s	$\color{#d91a1a}-2.39\%$
test_ddpg_speed[False-None]	3.1324ms	2.8072ms	356.2216 Ops/s	353.3936 Ops/s	$\color{#35bf28}+0.80\%$
test_ddpg_speed[False-backward]	4.1652ms	3.9911ms	250.5597 Ops/s	250.4698 Ops/s	$\color{#35bf28}+0.04\%$
test_ddpg_speed[True-None]	1.6066ms	1.3983ms	715.1564 Ops/s	722.2421 Ops/s	$\color{#d91a1a}-0.98\%$
test_ddpg_speed[True-backward]	2.4497ms	2.3486ms	425.7817 Ops/s	384.0401 Ops/s	$\textbf{\color{#35bf28}+10.87\%}$
test_ddpg_speed[reduce-overhead-None]	1.7713ms	1.3815ms	723.8329 Ops/s	707.3817 Ops/s	$\color{#35bf28}+2.33\%$
test_ddpg_speed[reduce-overhead-backward]	2.7267ms	2.3457ms	426.3120 Ops/s	439.1455 Ops/s	$\color{#d91a1a}-2.92\%$
test_sac_speed[False-None]	8.1267ms	7.6698ms	130.3813 Ops/s	98.1099 Ops/s	$\textbf{\color{#35bf28}+32.89\%}$
test_sac_speed[False-backward]	11.2117ms	10.8339ms	92.3026 Ops/s	96.5537 Ops/s	$\color{#d91a1a}-4.40\%$
test_sac_speed[True-None]	2.4739ms	2.0930ms	477.7807 Ops/s	458.1816 Ops/s	$\color{#35bf28}+4.28\%$
test_sac_speed[True-backward]	4.2995ms	3.9741ms	251.6270 Ops/s	252.4788 Ops/s	$\color{#d91a1a}-0.34\%$
test_sac_speed[reduce-overhead-None]	2.5621ms	2.0881ms	478.9014 Ops/s	456.8142 Ops/s	$\color{#35bf28}+4.84\%$
test_sac_speed[reduce-overhead-backward]	4.1156ms	3.9782ms	251.3706 Ops/s	251.7723 Ops/s	$\color{#d91a1a}-0.16\%$
test_redq_speed[False-None]	10.4637ms	9.9782ms	100.2184 Ops/s	100.3476 Ops/s	$\color{#d91a1a}-0.13\%$
test_redq_speed[False-backward]	18.1774ms	17.4137ms	57.4262 Ops/s	59.7198 Ops/s	$\color{#d91a1a}-3.84\%$
test_redq_speed[True-None]	4.5213ms	4.2767ms	233.8275 Ops/s	217.6083 Ops/s	$\textbf{\color{#35bf28}+7.45\%}$
test_redq_speed[True-backward]	9.9952ms	9.5841ms	104.3400 Ops/s	103.6878 Ops/s	$\color{#35bf28}+0.63\%$
test_redq_speed[reduce-overhead-None]	5.3220ms	4.2995ms	232.5841 Ops/s	218.8467 Ops/s	$\textbf{\color{#35bf28}+6.28\%}$
test_redq_speed[reduce-overhead-backward]	10.0377ms	9.7910ms	102.1343 Ops/s	95.1315 Ops/s	$\textbf{\color{#35bf28}+7.36\%}$
test_redq_deprec_speed[False-None]	11.1455ms	10.6551ms	93.8520 Ops/s	93.3734 Ops/s	$\color{#35bf28}+0.51\%$
test_redq_deprec_speed[False-backward]	15.6906ms	15.3185ms	65.2805 Ops/s	66.1866 Ops/s	$\color{#d91a1a}-1.37\%$
test_redq_deprec_speed[True-None]	3.8212ms	3.5481ms	281.8390 Ops/s	277.4548 Ops/s	$\color{#35bf28}+1.58\%$
test_redq_deprec_speed[True-backward]	7.4734ms	7.2066ms	138.7612 Ops/s	130.9055 Ops/s	$\textbf{\color{#35bf28}+6.00\%}$
test_redq_deprec_speed[reduce-overhead-None]	3.8190ms	3.4943ms	286.1818 Ops/s	280.7897 Ops/s	$\color{#35bf28}+1.92\%$
test_redq_deprec_speed[reduce-overhead-backward]	7.3954ms	7.1895ms	139.0922 Ops/s	126.8760 Ops/s	$\textbf{\color{#35bf28}+9.63\%}$
test_td3_speed[False-None]	49.8825ms	8.1358ms	122.9140 Ops/s	132.1126 Ops/s	$\textbf{\color{#d91a1a}-6.96\%}$
test_td3_speed[False-backward]	11.0145ms	10.5036ms	95.2058 Ops/s	98.8626 Ops/s	$\color{#d91a1a}-3.70\%$
test_td3_speed[True-None]	1.8913ms	1.8338ms	545.3167 Ops/s	544.6600 Ops/s	$\color{#35bf28}+0.12\%$
test_td3_speed[True-backward]	3.7783ms	3.5946ms	278.1961 Ops/s	244.3962 Ops/s	$\textbf{\color{#35bf28}+13.83\%}$
test_td3_speed[reduce-overhead-None]	1.8126ms	1.7613ms	567.7679 Ops/s	559.0430 Ops/s	$\color{#35bf28}+1.56\%$
test_td3_speed[reduce-overhead-backward]	3.7497ms	3.5735ms	279.8367 Ops/s	229.2782 Ops/s	$\textbf{\color{#35bf28}+22.05\%}$
test_cql_speed[False-None]	25.4979ms	24.9351ms	40.1041 Ops/s	39.2736 Ops/s	$\color{#35bf28}+2.11\%$
test_cql_speed[False-backward]	34.4123ms	33.8264ms	29.5627 Ops/s	29.7363 Ops/s	$\color{#d91a1a}-0.58\%$
test_cql_speed[True-None]	12.4860ms	12.2969ms	81.3216 Ops/s	80.5860 Ops/s	$\color{#35bf28}+0.91\%$
test_cql_speed[True-backward]	18.4094ms	17.7941ms	56.1983 Ops/s	55.5451 Ops/s	$\color{#35bf28}+1.18\%$
test_cql_speed[reduce-overhead-None]	12.6610ms	12.3801ms	80.7745 Ops/s	79.6460 Ops/s	$\color{#35bf28}+1.42\%$
test_cql_speed[reduce-overhead-backward]	18.2383ms	17.7120ms	56.4589 Ops/s	52.4794 Ops/s	$\textbf{\color{#35bf28}+7.58\%}$
test_a2c_speed[False-None]	5.6236ms	5.3463ms	187.0440 Ops/s	183.0009 Ops/s	$\color{#35bf28}+2.21\%$
test_a2c_speed[False-backward]	12.0040ms	11.6569ms	85.7862 Ops/s	85.1554 Ops/s	$\color{#35bf28}+0.74\%$
test_a2c_speed[True-None]	4.0520ms	3.7144ms	269.2197 Ops/s	257.4899 Ops/s	$\color{#35bf28}+4.56\%$
test_a2c_speed[True-backward]	8.8435ms	8.5254ms	117.2964 Ops/s	116.1268 Ops/s	$\color{#35bf28}+1.01\%$
test_a2c_speed[reduce-overhead-None]	3.8814ms	3.6985ms	270.3774 Ops/s	270.9078 Ops/s	$\color{#d91a1a}-0.20\%$
test_a2c_speed[reduce-overhead-backward]	9.0276ms	8.8523ms	112.9650 Ops/s	110.7843 Ops/s	$\color{#35bf28}+1.97\%$
test_ppo_speed[False-None]	6.2307ms	5.7738ms	173.1961 Ops/s	172.4595 Ops/s	$\color{#35bf28}+0.43\%$
test_ppo_speed[False-backward]	12.7233ms	12.3924ms	80.6949 Ops/s	82.5379 Ops/s	$\color{#d91a1a}-2.23\%$
test_ppo_speed[True-None]	3.8136ms	3.6655ms	272.8117 Ops/s	268.6523 Ops/s	$\color{#35bf28}+1.55\%$
test_ppo_speed[True-backward]	9.2138ms	8.4890ms	117.7989 Ops/s	118.4472 Ops/s	$\color{#d91a1a}-0.55\%$
test_ppo_speed[reduce-overhead-None]	3.7678ms	3.6260ms	275.7848 Ops/s	271.1937 Ops/s	$\color{#35bf28}+1.69\%$
test_ppo_speed[reduce-overhead-backward]	9.1605ms	8.8211ms	113.3647 Ops/s	114.7400 Ops/s	$\color{#d91a1a}-1.20\%$
test_reinforce_speed[False-None]	4.7248ms	4.5146ms	221.5060 Ops/s	218.0367 Ops/s	$\color{#35bf28}+1.59\%$
test_reinforce_speed[False-backward]	7.5655ms	7.2986ms	137.0127 Ops/s	137.0711 Ops/s	$\color{#d91a1a}-0.04\%$
test_reinforce_speed[True-None]	2.9821ms	2.8584ms	349.8462 Ops/s	340.3623 Ops/s	$\color{#35bf28}+2.79\%$
test_reinforce_speed[True-backward]	7.8711ms	7.7065ms	129.7599 Ops/s	129.4067 Ops/s	$\color{#35bf28}+0.27\%$
test_reinforce_speed[reduce-overhead-None]	3.0919ms	2.8387ms	352.2755 Ops/s	348.0377 Ops/s	$\color{#35bf28}+1.22\%$
test_reinforce_speed[reduce-overhead-backward]	8.2309ms	7.8948ms	126.6653 Ops/s	127.1658 Ops/s	$\color{#d91a1a}-0.39\%$
test_iql_speed[False-None]	20.2770ms	19.5887ms	51.0497 Ops/s	51.5156 Ops/s	$\color{#d91a1a}-0.90\%$
test_iql_speed[False-backward]	30.9739ms	30.1153ms	33.2058 Ops/s	34.0148 Ops/s	$\color{#d91a1a}-2.38\%$
test_iql_speed[True-None]	8.7477ms	8.5092ms	117.5194 Ops/s	116.7528 Ops/s	$\color{#35bf28}+0.66\%$
test_iql_speed[True-backward]	17.2419ms	16.7166ms	59.8209 Ops/s	60.1123 Ops/s	$\color{#d91a1a}-0.48\%$
test_iql_speed[reduce-overhead-None]	8.9722ms	8.6404ms	115.7358 Ops/s	115.9384 Ops/s	$\color{#d91a1a}-0.17\%$
test_iql_speed[reduce-overhead-backward]	17.9388ms	17.2241ms	58.0581 Ops/s	58.3289 Ops/s	$\color{#d91a1a}-0.46\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.4658ms	5.9878ms	167.0072 Ops/s	173.9072 Ops/s	$\color{#d91a1a}-3.97\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5265ms	0.2748ms	3.6391 KOps/s	2.9293 KOps/s	$\textbf{\color{#35bf28}+24.23\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5492ms	0.2554ms	3.9149 KOps/s	3.0765 KOps/s	$\textbf{\color{#35bf28}+27.25\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9920ms	5.7561ms	173.7288 Ops/s	181.5816 Ops/s	$\color{#d91a1a}-4.32\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.9529ms	0.2718ms	3.6788 KOps/s	2.8374 KOps/s	$\textbf{\color{#35bf28}+29.65\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5062ms	0.2529ms	3.9537 KOps/s	3.0878 KOps/s	$\textbf{\color{#35bf28}+28.04\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.4427ms	1.2078ms	827.9684 Ops/s	746.7748 Ops/s	$\textbf{\color{#35bf28}+10.87\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.3248ms	1.1232ms	890.2843 Ops/s	795.4779 Ops/s	$\textbf{\color{#35bf28}+11.92\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	10.0551ms	6.0744ms	164.6255 Ops/s	175.0055 Ops/s	$\textbf{\color{#d91a1a}-5.93\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.1191ms	0.4786ms	2.0896 KOps/s	1.9760 KOps/s	$\textbf{\color{#35bf28}+5.75\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7670ms	0.4827ms	2.0717 KOps/s	2.0505 KOps/s	$\color{#35bf28}+1.04\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	6.1206ms	5.8145ms	171.9840 Ops/s	179.8129 Ops/s	$\color{#d91a1a}-4.35\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0475ms	0.2811ms	3.5579 KOps/s	2.5877 KOps/s	$\textbf{\color{#35bf28}+37.49\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.4609ms	0.2612ms	3.8285 KOps/s	2.9100 KOps/s	$\textbf{\color{#35bf28}+31.56\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9762ms	5.7458ms	174.0407 Ops/s	180.0681 Ops/s	$\color{#d91a1a}-3.35\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.9851ms	0.3449ms	2.8997 KOps/s	3.3648 KOps/s	$\textbf{\color{#d91a1a}-13.82\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4744ms	0.2577ms	3.8799 KOps/s	3.9672 KOps/s	$\color{#d91a1a}-2.20\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.9641ms	5.8854ms	169.9107 Ops/s	174.1490 Ops/s	$\color{#d91a1a}-2.43\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.1893ms	0.4263ms	2.3456 KOps/s	2.1981 KOps/s	$\textbf{\color{#35bf28}+6.71\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6327ms	0.4063ms	2.4614 KOps/s	2.2081 KOps/s	$\textbf{\color{#35bf28}+11.47\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.6426ms	5.1210ms	195.2739 Ops/s	54.6879 Ops/s	$\textbf{\color{#35bf28}+257.07\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	9.6291ms	2.3063ms	433.5911 Ops/s	503.2951 Ops/s	$\textbf{\color{#d91a1a}-13.85\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	5.8674ms	1.1733ms	852.2781 Ops/s	823.7733 Ops/s	$\color{#35bf28}+3.46\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.5483s	15.9426ms	62.7251 Ops/s	201.9572 Ops/s	$\textbf{\color{#d91a1a}-68.94\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	3.9127ms	1.7312ms	577.6246 Ops/s	509.1745 Ops/s	$\textbf{\color{#35bf28}+13.44\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	2.1988ms	1.0546ms	948.2400 Ops/s	829.5255 Ops/s	$\textbf{\color{#35bf28}+14.31\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	8.3948ms	5.2777ms	189.4750 Ops/s	195.6707 Ops/s	$\color{#d91a1a}-3.17\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.2844ms	2.2121ms	452.0612 Ops/s	455.6822 Ops/s	$\color{#d91a1a}-0.79\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	8.8560ms	1.3815ms	723.8317 Ops/s	741.4657 Ops/s	$\color{#d91a1a}-2.38\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	34.2828ms	32.0448ms	31.2064 Ops/s	30.6434 Ops/s	$\color{#35bf28}+1.84\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	18.7044ms	16.9680ms	58.9345 Ops/s	58.2293 Ops/s	$\color{#35bf28}+1.21\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	36.1909ms	33.1883ms	30.1311 Ops/s	29.5321 Ops/s	$\color{#35bf28}+2.03\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.6281ms	17.5306ms	57.0430 Ops/s	58.2540 Ops/s	$\color{#d91a1a}-2.08\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	37.4408ms	35.5891ms	28.0985 Ops/s	28.1588 Ops/s	$\color{#d91a1a}-0.21\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	0.4544s	27.7794ms	35.9979 Ops/s	53.7721 Ops/s	$\textbf{\color{#d91a1a}-33.05\%}$