Roadmap for StepUp-queue (and Core)?

[lcdumort]

Although I only have experience with stepup-reprep, I am considering using stepup-core and stepup-queue to manage my computations and results on a SLURM cluster. Previously, I used parman, which has since been archived.

Before diving in and introducing these packages at DLR, I was wondering if there are plans to further develop and support them for general use, or if the focus will shift primarily toward reprep support.

[tovrstra]

I plan to maintain StepUp Queue for the near future (at least). I'll try to explain a bit more its current scope. We've been testing this package in several scenarios and it seems to work reasonably well for the goal of setting up customized medium-size workflows, i.e. up to 10k SLURM jobs. That upper limit mainly stems from two things:

1. The overhead of StepUp Core. It stores its state in an SQLite database, so it can always restart, even after being killed in unfavorable conditions, and pick up where it was stopped. This type of persistence comes with some cost.

2. We don't use pilot jobs, which means that every computation is its own SLURM job. That is intuitive, but comes with some overhead. Another common strategy is to work with pilot jobs, which get submitted upfront to reserve resources, and which perform work when requested. One could extend StepUp Queue with this feature, but I don't have concrete plans for it. Pilot jobs are mainly useful when you have a huge amount (> 10M) of tiny compute tasks (few seconds), which I don't expect to be doing anytime soon.

There will be optimizations in future that may stretch the upper limit by one order of magnitude, but that is not a priority.

As far StepUp RepRep goes, the main need is good documentation and better templates. One code change I anticipate for RepRep is reliable Typst integration. (I'm waiting for Typst 0.15.) Another nice-to-have would be integration with FigShare, in the same way it currently works with Zenodo.

[lcdumort]

Thanks for the reply! Good to hear that StepUp-core and StepUp Queue are here to stay, at least for a while. I've been enjoying it quite a bit once you get the hang of it.
I'll continue using it for my workflows and will introduce it to the research group here as well, as the upper limit seems more than enough for what we intend to use it for.