Generating datasets and running the pipeline

[sidgairo18]

Hi @KupynOrest ,

Thanks for the work.

I had a question regarding generating the datasets - so from the pipeline it seems like each sample is generated individually (sequentially) and the images cannot be batched (i might be wrong about this). And each sample can take a few minutes to generate a new augmentation; even on GPUs.

How does one manage to parallelise this, to generate datasets faster? because otherwise even for generating ~10k images it can take over a day (or few days).

Would be grateful for your response.