DDSim proiling

[mgoliyad]

We are working with Rishabh and Mason to build MD use case to test new ddmd v2 framework.
The use case will be similar to "DeepDriveMD-pipeline" (https://github.com/DeepDriveMD/DeepDriveMD-pipeline) but adjusted to new AI steering algorithm.

[mgoliyad]

Rishabh's first draft of pipeline design:

https://docs.google.com/presentation/d/1TE1Y0gKPhd-YH4GPvGohDj8164L9OwjFTp6NhG7C_fk/edit?usp=sharing

[mgoliyad]

Add use case for mini apps

[mgoliyad]

Miniapp Pipeline on anvil CPU node:

[mgoliyad]

New plots from CPU compute node with partition=wholenode

[mgoliyad]

run with miniapps execs - slurm time 23m
run with noop func - slurm time 58s

[mgoliyad]

[mgoliyad]

above plots for run with noop func

[mgoliyad]

sessions are saved on anvil in /anvil/scratch/x-mgoliyad1/ddmd/miniapps/DeepDriveMD/examples/miniapps_pipeline/session_for_plots

[mgoliyad]

Ran more test to make noop and exec workflow as close as possible (the adaptivity of workflow makes it difficult to initiate/cancel/complete the same number of simulations/train/predict).

Here is the summary of the run with minapp exec:
There are 1030 tasks in this session

TOTAL RUNTIME: 1346s | RCT OVH: 27s

Following rates are based on all tasks

scheduling rate: 0.7815915325934958
launching rate: 1.5631568851881843

Here is the summary of the run with noop func:
There are 1013 tasks in this session

TOTAL RUNTIME: 57s | RCT OVH: 24s

Following rates are based on all tasks

scheduling rate: 31.27648271018146
launching rate: 59.03540254858082

[mgoliyad]