Pause-Resume run of Epoch

[DanRRRR]

Any thoughts?
AI says
"Yes, it is possible to pause an MPI run, but the method depends on the context and the tools being used.

For performance analysis, Intel VTune Amplifier XE provides a Pause/Resume API that can be used within MPI programs to control sampling data collection.
This API can be called from code running in different processes, as each process operates in its own memory space and the APIs do not interact with each other.
However, when using certain collection types like "advanced-hotspots" with mpirun, using the Pause/Resume API can lead to unexpected results because multiple processes sending pause/resume requests can confuse the VTune drivers.

For programmatic control during execution, there is no direct "pause" function in MPI. Instead, a process can be made to wait for a specific duration using a loop that checks the elapsed time, for example, by using MPI_WTIME() to simulate a sleep function.
To stop all MPI processes after a fixed time, one process (often rank 0) can monitor the elapsed time and signal other processes to terminate using MPI_Send.
This requires careful coordination to ensure all processes are unblocked from any pending communication before they terminate.

If a program is running in an infinite loop and needs to be stopped externally, standard methods like pressing Ctrl+C in the command prompt usually work.
If that fails, the mpirun process can be killed from the task manager.
For a more graceful shutdown, a file-based signal can be used where processes periodically check for a specific file's existence and exit if found"