Small CorrDiff fixes

[swbg]

PhysicsNeMo Pull Request

Description

Three small fixes:

• For n output timestamps, generate.py always wrote the first n timestamps of the dataset to the output file. It now uses the sample indices for indexing the list of times instead.
• The checkpoint cleanup procedure was potentially called on multiple ranks, which can cause race conditions. It is now only called after saving a checkpoint (on rank 0).
• Write the input data only once to the output NetCDF4 file when using generate.py.

I also changed io_syncronous to io_synchronous.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• The CHANGELOG.md is up to date with these changes.
• An issue is linked to this pull request.

Dependencies