how many hours is the training time for the PPO algorithm?

[davinca]

I have no GPU,the training process is too slow and i find the agent interacting with the environment spends much time.so I wanna know the total training time.
thanks!

[yao-jason]

I modified the agent_pg.py for time recording, and the following is the log of the first 100 training episodes for PPO on the PC with i7-4790 and GTX 1080.
The agent acts with the environment more than 1k times in each episode, and the spent time of each action is about 1.80 ms.
Because the actor is made with some neural networks, such as convolution and dense if you just use CPU to train the actor, the training time consuming of course.
It is better to train this actor using GPU, if you don't have, give it a try for Google Colab.

Best regards,
Jason Yao

Episode: 1, Actions: 1388, reward: -20, Avg. reward: -20.000, Time: 3.12, T/A: 2.25ms
[Save Checkpoint] Avg. reward improved from -87.000000 to -20.000000
Saving checkpoint...
Episode: 2, Actions: 1714, reward: -18, Avg. reward: -19.000, Time: 3.20, T/A: 1.86ms
[Save Checkpoint] Avg. reward improved from -20.000000 to -19.000000
Saving checkpoint...
Episode: 3, Actions: 1389, reward: -20, Avg. reward: -19.333, Time: 2.50, T/A: 1.80ms
Episode: 4, Actions: 1224, reward: -20, Avg. reward: -19.500, Time: 2.22, T/A: 1.81ms
Episode: 5, Actions: 1473, reward: -20, Avg. reward: -19.600, Time: 2.59, T/A: 1.76ms
Episode: 6, Actions: 1295, reward: -20, Avg. reward: -19.667, Time: 2.38, T/A: 1.84ms
Episode: 7, Actions: 1260, reward: -21, Avg. reward: -19.857, Time: 2.26, T/A: 1.79ms
Episode: 8, Actions: 1576, reward: -21, Avg. reward: -20.000, Time: 2.83, T/A: 1.80ms
Episode: 9, Actions: 1306, reward: -20, Avg. reward: -20.000, Time: 2.41, T/A: 1.85ms
Episode: 10, Actions: 1325, reward: -21, Avg. reward: -20.100, Time: 2.44, T/A: 1.84ms
Episode: 11, Actions: 1347, reward: -21, Avg. reward: -20.182, Time: 2.48, T/A: 1.84ms
Episode: 12, Actions: 1515, reward: -19, Avg. reward: -20.083, Time: 2.75, T/A: 1.82ms
Episode: 13, Actions: 2005, reward: -21, Avg. reward: -20.154, Time: 3.64, T/A: 1.82ms
Episode: 14, Actions: 1095, reward: -21, Avg. reward: -20.214, Time: 1.96, T/A: 1.79ms
Episode: 15, Actions: 1343, reward: -21, Avg. reward: -20.267, Time: 2.46, T/A: 1.83ms
Episode: 16, Actions: 1252, reward: -21, Avg. reward: -20.312, Time: 2.23, T/A: 1.78ms
Episode: 17, Actions: 1085, reward: -21, Avg. reward: -20.353, Time: 1.87, T/A: 1.72ms
Episode: 18, Actions: 1344, reward: -21, Avg. reward: -20.389, Time: 2.43, T/A: 1.81ms
Episode: 19, Actions: 1260, reward: -21, Avg. reward: -20.421, Time: 2.27, T/A: 1.80ms
Episode: 20, Actions: 1250, reward: -21, Avg. reward: -20.450, Time: 2.31, T/A: 1.85ms
Episode: 21, Actions: 1314, reward: -20, Avg. reward: -20.429, Time: 2.45, T/A: 1.86ms
Episode: 22, Actions: 1625, reward: -20, Avg. reward: -20.409, Time: 2.91, T/A: 1.79ms
Episode: 23, Actions: 1605, reward: -19, Avg. reward: -20.348, Time: 2.90, T/A: 1.81ms
Episode: 24, Actions: 1555, reward: -18, Avg. reward: -20.250, Time: 2.81, T/A: 1.81ms
Episode: 25, Actions: 1580, reward: -21, Avg. reward: -20.280, Time: 2.81, T/A: 1.78ms
Episode: 26, Actions: 1635, reward: -18, Avg. reward: -20.192, Time: 2.90, T/A: 1.78ms
Episode: 27, Actions: 1845, reward: -19, Avg. reward: -20.148, Time: 3.31, T/A: 1.80ms
Episode: 28, Actions: 1329, reward: -21, Avg. reward: -20.179, Time: 2.32, T/A: 1.75ms
Episode: 29, Actions: 1893, reward: -20, Avg. reward: -20.172, Time: 3.34, T/A: 1.76ms
Episode: 30, Actions: 1103, reward: -21, Avg. reward: -20.200, Time: 1.98, T/A: 1.79ms
Episode: 31, Actions: 1681, reward: -19, Avg. reward: -20.167, Time: 3.03, T/A: 1.80ms
Episode: 32, Actions: 1175, reward: -21, Avg. reward: -20.267, Time: 2.15, T/A: 1.83ms
Episode: 33, Actions: 1755, reward: -19, Avg. reward: -20.233, Time: 3.18, T/A: 1.81ms
Episode: 34, Actions: 1713, reward: -20, Avg. reward: -20.233, Time: 3.03, T/A: 1.77ms
Episode: 35, Actions: 1512, reward: -19, Avg. reward: -20.200, Time: 2.59, T/A: 1.72ms
Episode: 36, Actions: 1644, reward: -20, Avg. reward: -20.200, Time: 2.90, T/A: 1.76ms
Episode: 37, Actions: 1563, reward: -21, Avg. reward: -20.200, Time: 2.84, T/A: 1.82ms
Episode: 38, Actions: 1173, reward: -21, Avg. reward: -20.200, Time: 2.10, T/A: 1.79ms
Episode: 39, Actions: 1586, reward: -19, Avg. reward: -20.167, Time: 2.88, T/A: 1.82ms
Episode: 40, Actions: 1351, reward: -21, Avg. reward: -20.167, Time: 2.43, T/A: 1.80ms
Episode: 41, Actions: 1868, reward: -18, Avg. reward: -20.067, Time: 3.31, T/A: 1.77ms
Episode: 42, Actions: 1627, reward: -20, Avg. reward: -20.100, Time: 2.97, T/A: 1.82ms
Episode: 43, Actions: 1422, reward: -21, Avg. reward: -20.100, Time: 2.49, T/A: 1.75ms
Episode: 44, Actions: 1461, reward: -20, Avg. reward: -20.067, Time: 2.71, T/A: 1.86ms
Episode: 45, Actions: 1803, reward: -20, Avg. reward: -20.033, Time: 3.32, T/A: 1.84ms
Episode: 46, Actions: 1623, reward: -20, Avg. reward: -20.000, Time: 3.01, T/A: 1.85ms
Episode: 47, Actions: 1442, reward: -19, Avg. reward: -19.933, Time: 2.65, T/A: 1.84ms
Episode: 48, Actions: 1639, reward: -21, Avg. reward: -19.933, Time: 2.93, T/A: 1.79ms
Episode: 49, Actions: 1924, reward: -17, Avg. reward: -19.800, Time: 3.48, T/A: 1.81ms
Episode: 50, Actions: 1582, reward: -21, Avg. reward: -19.800, Time: 2.84, T/A: 1.79ms
Episode: 51, Actions: 1823, reward: -21, Avg. reward: -19.833, Time: 3.20, T/A: 1.75ms
Episode: 52, Actions: 1692, reward: -21, Avg. reward: -19.867, Time: 3.09, T/A: 1.83ms
Episode: 53, Actions: 1641, reward: -20, Avg. reward: -19.900, Time: 2.97, T/A: 1.81ms
Episode: 54, Actions: 1967, reward: -20, Avg. reward: -19.967, Time: 3.57, T/A: 1.81ms
Episode: 55, Actions: 1651, reward: -21, Avg. reward: -19.967, Time: 3.02, T/A: 1.83ms
Episode: 56, Actions: 1693, reward: -19, Avg. reward: -20.000, Time: 3.00, T/A: 1.77ms
Episode: 57, Actions: 1483, reward: -20, Avg. reward: -20.033, Time: 2.60, T/A: 1.75ms
Episode: 58, Actions: 1849, reward: -19, Avg. reward: -19.967, Time: 3.27, T/A: 1.77ms
Episode: 59, Actions: 2351, reward: -18, Avg. reward: -19.900, Time: 4.28, T/A: 1.82ms
Episode: 60, Actions: 1738, reward: -21, Avg. reward: -19.900, Time: 3.09, T/A: 1.78ms
Episode: 61, Actions: 1840, reward: -21, Avg. reward: -19.967, Time: 3.30, T/A: 1.79ms
Episode: 62, Actions: 2210, reward: -18, Avg. reward: -19.867, Time: 3.93, T/A: 1.78ms
Episode: 63, Actions: 1683, reward: -21, Avg. reward: -19.933, Time: 2.93, T/A: 1.74ms
Episode: 64, Actions: 2215, reward: -16, Avg. reward: -19.800, Time: 3.98, T/A: 1.80ms
Episode: 65, Actions: 2153, reward: -19, Avg. reward: -19.800, Time: 3.85, T/A: 1.79ms
Episode: 66, Actions: 1495, reward: -21, Avg. reward: -19.833, Time: 2.67, T/A: 1.78ms
Episode: 67, Actions: 2239, reward: -21, Avg. reward: -19.833, Time: 4.05, T/A: 1.81ms
Episode: 68, Actions: 1941, reward: -19, Avg. reward: -19.767, Time: 3.52, T/A: 1.82ms
Episode: 69, Actions: 1512, reward: -19, Avg. reward: -19.767, Time: 2.70, T/A: 1.79ms
Episode: 70, Actions: 1870, reward: -17, Avg. reward: -19.633, Time: 3.34, T/A: 1.78ms
Episode: 71, Actions: 1383, reward: -20, Avg. reward: -19.700, Time: 2.44, T/A: 1.76ms
Episode: 72, Actions: 1796, reward: -20, Avg. reward: -19.700, Time: 3.15, T/A: 1.76ms
Episode: 73, Actions: 2235, reward: -19, Avg. reward: -19.633, Time: 4.03, T/A: 1.80ms
Episode: 74, Actions: 2162, reward: -19, Avg. reward: -19.600, Time: 3.91, T/A: 1.81ms
Episode: 75, Actions: 1542, reward: -20, Avg. reward: -19.600, Time: 2.71, T/A: 1.76ms
Episode: 76, Actions: 1775, reward: -20, Avg. reward: -19.600, Time: 3.15, T/A: 1.77ms
Episode: 77, Actions: 1844, reward: -21, Avg. reward: -19.667, Time: 3.33, T/A: 1.80ms
Episode: 78, Actions: 1852, reward: -20, Avg. reward: -19.633, Time: 3.31, T/A: 1.78ms
Episode: 79, Actions: 2000, reward: -18, Avg. reward: -19.667, Time: 3.60, T/A: 1.80ms
Episode: 80, Actions: 2006, reward: -19, Avg. reward: -19.600, Time: 3.69, T/A: 1.84ms
Episode: 81, Actions: 2069, reward: -19, Avg. reward: -19.533, Time: 3.75, T/A: 1.81ms
Episode: 82, Actions: 1941, reward: -19, Avg. reward: -19.467, Time: 3.46, T/A: 1.78ms
Episode: 83, Actions: 2278, reward: -18, Avg. reward: -19.400, Time: 4.06, T/A: 1.78ms
Episode: 84, Actions: 2112, reward: -20, Avg. reward: -19.400, Time: 3.77, T/A: 1.79ms
Episode: 85, Actions: 2137, reward: -18, Avg. reward: -19.300, Time: 3.75, T/A: 1.75ms
Episode: 86, Actions: 1496, reward: -21, Avg. reward: -19.367, Time: 2.75, T/A: 1.84ms
Episode: 87, Actions: 2297, reward: -18, Avg. reward: -19.300, Time: 4.17, T/A: 1.81ms
Episode: 88, Actions: 2127, reward: -18, Avg. reward: -19.267, Time: 3.75, T/A: 1.76ms
Episode: 89, Actions: 2240, reward: -21, Avg. reward: -19.367, Time: 4.02, T/A: 1.79ms
Episode: 90, Actions: 2128, reward: -20, Avg. reward: -19.333, Time: 3.77, T/A: 1.77ms
Episode: 91, Actions: 2323, reward: -19, Avg. reward: -19.267, Time: 4.13, T/A: 1.78ms
Episode: 92, Actions: 2474, reward: -17, Avg. reward: -19.233, Time: 4.53, T/A: 1.83ms
Episode: 93, Actions: 1743, reward: -21, Avg. reward: -19.233, Time: 3.12, T/A: 1.79ms
Episode: 94, Actions: 1872, reward: -20, Avg. reward: -19.367, Time: 3.34, T/A: 1.78ms
Episode: 95, Actions: 1924, reward: -19, Avg. reward: -19.367, Time: 3.48, T/A: 1.81ms
Episode: 96, Actions: 2065, reward: -19, Avg. reward: -19.300, Time: 3.76, T/A: 1.82ms
Episode: 97, Actions: 1897, reward: -21, Avg. reward: -19.300, Time: 3.44, T/A: 1.81ms
Episode: 98, Actions: 2220, reward: -18, Avg. reward: -19.267, Time: 3.92, T/A: 1.77ms
Episode: 99, Actions: 2586, reward: -17, Avg. reward: -19.200, Time: 4.62, T/A: 1.78ms
Episode: 100, Actions: 1505, reward: -21, Avg. reward: -19.333, Time: 2.70, T/A: 1.80ms