It may forget pertinent information about positions that it no longer visits.

[apollo-time]

I see my model don't be improved anymore.
Moreover I found "It may forget pertinent information about positions that it no longer visits" as ThomasWAnthony's when opinion select action unusually.
@mokemokechicken, @gooooloo How about it?

[mokemokechicken]

@apollo-time

I think that there is that's possibility,
and if we want to improve the model more and more, we need larger sim_per_move and self-play dataset.

I have a simple hypothesis that

• upper performace(=strength) of model is decided by sim_per_move.
• speed of changing(≒improvement) is decided by speed of generating self-play data and size of self-play dataset(small size is faster).
• generalization performace of model is decided by size of self-play dataset(large size is more general).

so, I feel that increasing sim_per_move and dataset size gradually is effective.
(I think that Human also do that to become professional.)

[apollo-time]

I think larger slim_per_move and self-play dataset can't resolve no longer visits problem, because the unusually positions can't be selected by self-play MCTS.
So I try select fully random action sometimes in self-play, and ignore previous history of the random action.

[AranKomat]

@mokemokechicken I asked @gooooloo a similar question in other thread, but what is the default ratio of the number of games per gradient update ratio of your algorithm? I guess the ratio is important for the performance, since it behaves like sims/move, which is undoubtedly important.

[mokemokechicken]

@AranKomat

what is the default ratio of the number of games per gradient update ratio of your algorithm?

I do not know which number to answer concretely, but the resulting speed is as follows.

setting

• batch size: 256
• sim per move: 400
• (nb_game_in_file, max_file_num): (5, 300)

speed

• 80 seconds per 1 self-play game
• 400 positions per 1 self-play game
• 150 seconds per 200 steps(bs=256) -> 150 seconds per 200*256 positions

so

• Training: 341 positions / seconds (=200*256/160)
• SelfPlay: 5 positions / seconds (=400 / 80)
• Training/SelfPlay Ratio: 68 (=341/5)

Maybe, it means that 1 position is learned 68 times regardless (nb_game_in_file, max_file_num).

[AranKomat]

Thanks for your answer. In the case of Go with AlphaZero, 700k minibatches (2048 positions each) and 21 million self-play games were performed. Assuming that each game ended with 150 stones (positions) placed, 700k x 2048/(21m x 150)=0.44 [trained position]/[self-play-generated position], which is much less than 68. So, I guess you can improve your performance with more self-plays per update. Maybe the performance gain by increasing the sims/move from 100 to 800 was because you had a small self-play/training ratio, that is, you had too little exploration. Since having more games generated means more diverse data than having more sims/move, so spending more time on self-play may be more beneficial than more sims/move. But in practice, since your alg doesn't allow multi-processing (of multiple games) as done by Akababa, my suggestion may be not useful. But this may be useful for @gooooloo .

[mokemokechicken]

@AranKomat

I guess you can improve your performance with more self-plays per update.

I think so too.
In my environment, although GPU usage is already 100%(by self-play and training),
implementing multiprocess self-play will increase self-play games per training.

So I am planing to implement multiprocess self-play,
However, it is under consideration whether or not it really works with the present method.

[mokemokechicken]

I am testing on feature/multiprocess_selfplay,

when 16 parallel in self-play,

• 580 seconds per 1 self-play game (16 parallel) -> 36 seconds per self-play game
• 400 positions per 1 self-play game
• 225 seconds per 200 steps(bs=256) -> 225 seconds per 200*256 positions

so

• Training: 228 positions / seconds (=200*256/225)
• SelfPlay: 11 positions / seconds (=400 / 36)
• Training/SelfPlay Ratio: 21 (=228/11)

[AranKomat]

Cool. So, multi-processing successfully decreased the ratio and achieved 36s per game under 400 sims/move. Now, it suffices to elucidate the trade-off between training/selfplay ratio and sims/move. I'm excited for your subsequent announcements!

[mokemokechicken]

I also added wait to optimizer to change the ratio.

Now,

• 164 self-play game per 1 hour -> 22(=3600/164) seconds per self-play game
• 400 positions per 1 self-play game
• 225 * 2 seconds per 200 steps(bs=256) -> 450 seconds per 200 * 256 positions

so

• Training: 113 positions / seconds (=200*256/450)
• SelfPlay: 18 positions / seconds (= 400 / 22)
• Training/SelfPlay Ratio: 6.2 (=113/18)

[gooooloo]

@AranKomat

Mine is:

• 30 processes for self-play, about 150 seconds per game per process, gives 5 seconds per game in average.
• about 12 minutes per 100 steps training, batch size = 3072, gives 426 positions per second (=3072*100/12/60)

I actually don't understand below number @mokemokechicken mentioned:

400 positions per 1 self-play game

But if I just use this number, then I have self-play speed: 80 positions per second (=400/5).
Then Training/SelfPlay Ration: 5.3 (=426/80)

[gooooloo]

Thanks for your answer. In the case of Go with AlphaZero, 700k minibatches (2048 positions each) and 21 million self-play games were performed. Assuming that each game ended with 150 stones (positions) placed, 700k x 2048/(21m x 150)=0.44 [trained position]/[self-play-generated position], which is much less than 68. So, I guess you can improve your performance with more self-plays per update. Maybe the performance gain by increasing the sims/move from 100 to 800 was because you had a small self-play/training ratio, that is, you had too little exploration. Since having more games generated means more diverse data than having more sims/move, so spending more time on self-play may be more beneficial than more sims/move. But in practice, since your alg doesn't allow multi-processing (of multiple games) as done by Akababa, my suggestion may be not useful. But this may be useful for @gooooloo .

Thanks @AranKomat . I didn't see this post until just now...

I guess you can improve your performance with more self-plays per update

Yes, I also think so. Deepmind uses 2000+ or 4000+ TPU for selfplay (as Aja Huang says in a post, I just can't remember the link). We can see the self play performance is important.

Maybe the performance gain by increasing the sims/move from 100 to 800 was because you had a small self-play/training ratio, that is, you had too little exploration.

Actually I was getting an smaller selfplay/training ratio when increasing sims/move from 100 to 800. Although I also introduced multi process implementation at that time, the overall self play game speed is a little bit slower than before. Yet I observe the AI strength improvement.

[AranKomat]

@gooooloo In AlphaZero, staggering 5000 TPUs were used, so I totally agree. It's weird but nice that increased sims/move resulted in a smaller ratio. Hopefully, @mokemokechicken and others will observe a similar phenomena.

[mokemokechicken]

400 positions per 1 self-play game

Note:
I used (nb_game_in_file, max_file_num)=(5, 300), so the number of total games in training data was 1500 (games).
My training dataset size was about 600k (positions).
So, 600k / 1500 = 400 (position/game).

[gooooloo]

I used (nb_game_in_file, max_file_num)=(5, 300), so the number of total games in training data was 1500 (games).
My training dataset size was about 600k (positions).
So, 600k / 1500 = 400 (position/game).

But a reversi game has up to 60 position to move, isn't it? Event with up to 5 "PASS" move, it is 65. Then even with game state flip and rotation, it is at most 260.

UPDATE:
Oh my fault, "flip and rotation" gives a x8 multiplication, not x4. Then it makes sense. 400/8=50, you are playing 50 moves per game, giving you have a resignation mechanism.

[gooooloo]

... had a small self-play/training ratio

It's weird ... that increased sims/move resulted in a smaller ratio

The ratio is # of selp play moves / # of trained moves. I increased # sims per move, then self play got slower, then # of self play moves smaller. But training module not changed. So the total ratio got smaller. Isn't it?