Link root-painter to a working HPC enviroment running slurm

[paubach711]

Hi, I installed root-painter in one of my HPC servers running slurm, its seems to work but i dont really know how can i link the root-painter trainer to the server.
Usually the user im helping runs this software with google collab but we want to run this on more powerful machines and locally

[Abe404]

Hi,

I'm not sure exactly what you mean by 'link root-painter trainer to the server'. Could you provide a little more detail?

The way RootPainter works is there is a client and a server component. They communicate via a shared folder. If your slurm nodes have access to a shared, synced or mounted drive that the user can also mount/sync on their desktop / workstation where they run the client, then it should be possible to connect the client and server. The key is you have the root_painter_sync directory specified inside this shared drive/folder for both client and server. The server asks for you to specify a root_painter_sync directory when you first start it (it is subsequently stored in a root_painter_settings.json file in your home folder. The client asks for you to specify this directory when you first run it (and it is also subsequently stored in a root_painter_settings.json file in your home folder). Updating or locating the sync directory with the client GUI is also possible via options in the extras menu.

I usually use sshfs for mounting drives on a server, but perhaps your university already has some type of shared drive/folder system that would work for this particular user?

With colab the shared folder is facilitated with google drive, but I suggest finding a better solution for your own HPC setup as google drive is slow to sync data and can cause data sharing concerns for many universities (depending on their policy).

Another challenge with slurm and why I don't personally use it, is that it often assumes jobs can be run in the 'background' at some time in the future. RootPainter facilitates a real-time interactive process for model training that often doesn't fit the way some universities (or other organisations) like to have their slurm jobs scheduled.

Kind regards,
Abraham

[Abe404]

FYI I'm converting this issue to a discussion as it's not clear to me that there is an issue here.

[Arianui]

Hi Abe!
I work with Pau, who has been very helpful in setting up RootPainter on ICM server - as I am not an IT person, and never used RT on a institutional server before. Let me guide you through the steps that we followed. We set up RootPainter following the sshfs instructions, with these main steps:

1. I log in to the server via ssh on Ubuntu.
2. Cloned the RootPainter from github.
3. We struggled to install the dependencies in the virtual environment, but finally we succeeded.
4. Run RootPainter and synchronized the directory to a folder located in the Desktop. Maybe this could be the issue?

To mount the sync directory from the local machine, we installed a sshfs, as it is mentioned in the sshfs instructions. When I open RT, I see the folders, and I even created a project with some trial images. However, RT shuts down as I start training, so we suspect RT is not fully linked to the server. I personally haven’t understood very well where exactly do I have to load the images (in the PC, or the sync folder in the server). I think we are very close to setting it up, but not there yet!

Thank you very much for your help!
Best,
Ari

[Abe404]

Thanks for the details!

"RT shuts down as I start training" -- you mean the RootPainter client shuts down? I mean the GUI painter application? Do you mean it crashes? So it just suddenly closes? If the client is crashing, it makes me think there could be some kind of bug somewhere.

It maybe be worth checking the file ''client_exceptions.txt' in your sync folder, that could include some information on what went wrong if the client crashed.

The server link is really just a shared folder. You can test this works by dropping files in there (not using RootPainter) and seeing if they show up in the server folder and vice versa i.e put a file in there which on the server and see if it shows up inside your desktop folder.

Kind regards,
Abraham