Skip to content

giannis_5_5_2017_part2

Ioannis Paraskevakos edited this page May 5, 2017 · 1 revision

Dask Throughput experiments Comet 20 cores vs Comet 24 cores.

The experiments were run 10 times for every node count and process count. The experiments were run on Comet and it compares Dask's throughput from the 20 core case to the 24 core case. This will show us how much do the extra 4 cores per node change the execution. The upper two figures on each node count show the average throughput of the 10 runs along with their error bars. The two bottom figure show one instance of the experiment so that is more clear what is happening on one run.

1 node

2 nodes

3 nodes

4 nodes

Conclusions

There is a small change because of the extra 4 cores but not very significant that will show a major difference. We also see here that above 2 nodes there is not much of a gain on Dask's throughput. This means that the scheduler starts to spend more time in the logistics of the execution rather than scheduling tasks. It is somewhat expected since it has to communicate with more worker over the network.

Clone this wiki locally