Proposed - Move temDir and chunkDir to scratch space

[milechin]

Hi @seamore-z

You might get some speed up by moving the tempDir and chunkDir off the file system and onto the compute node's scratch space. Doing a lot of I/O operations on the local scratch space will be faster and more consistent than to access the chunks via file system. The change will need to be made in MSLSP_runTile_SCC.sh in the code block below:

MSLSP/SCC/MSLSP_runTile_SCC.sh

Lines 46 to 48 in 46ed940

	tempDir="${workDir}${tile}/temp/"
	chunkDir="${workDir}${tile}/imageChunks/"
	phenDir="${workDir}${tile}/phenoMetrics/"

The updated code will look like this, notice $workDIR is replaced with $TMPDIR`:

tempDir="${TMPDIR}/${tile}/temp/"	
chunkDir="${TMPDIR}/${tile}/imageChunks/"
phenDir="${workDir}${tile}/phenoMetrics/"

Couple things to note:

1. These files will be deleted when the job finishes, so if you want to save them, you will need to copy them to the file system before the job finishes. Since these seem like temporary files, I am assuming it is OK for them to be deleted.
2. There is a limit on how much storage is available on node's local scratch disk. Many of nodes have 885GB dedicated to the scratch storage. How much storage do you estimate will be required for tempDir and chunkDir for a single tile run?

[seamore-z]

I see... Yes, these are not needed once the final product is made. I just checked a couple of tiles, and it seems like up to around 40 GB for chunkDir and 21 GB for tempDir. Will that be ok? Am I sharing the scratch space with other users on the node/what would happen if it exceeds the scratch storage space?

[milechin]

That should be OK. Based on the parameter file, you are requesting 28 cores, so you are getting a whole node to yourself. In fact the "geo" queue has fifteen 32 core nodes, so you can request 32 cores and get 4 additional workers to work on the tile.

MSLSP/MSLSP_Parameters.json

Lines 49 to 51 in 46ed940

	"numCores":28,
	"numChunks":196,
	"runS10":false

There could be files leftover from other jobs in the scratch space, but if the scratch space is getting filled up, there is an auto cleanup process that will delete the oldest files not currently used for a job. So you should be OK.

[seamore-z]

Sounds good, I'll give this a try on the next set of tiles I run. Thanks!