sharing DB

[EricDeveaud]

Hello,

silly and dumb question (parece loco pero no temo nada) can we share the BDD accros various user. is the BioData DB used read only by fantasia(I did not crawl all the code in order to check it)

lets imagine we setup 2 BDDs with ecah embeddings on a shared space available accros our cluster

would it means that our user can run directly fantasia run ?

regards

Eric

[frapercan]

Hello Eric,

Ask freely, no question is absurd.

Yes, BioData DB is read-only for FANTASIA.

The idea is not really to maintain two separate databases, but one shared database that can be used by different FANTASIA clients. PostgreSQL handles concurrent reads well, and FANTASIA writes its results directly to disk. That said, duplicating the database is not such a big problem either: running one Docker container per machine is perfectly possible, and I do not think the whole setup would exceed 10 GB on disk.

In theory, RabbitMQ was meant to allow workload distribution across different hosts. In practice, FANTASIA ended up halfway between a sequential CLI tool and a distributed system, so this part is not fully adapted or documented.

The main issue is that fantasia run clears the queues at the beginning. There is also a config parameter related to that behaviour, so parallel runs could interfere with each other depending on how it is set. If the queues are preserved, then in principle two hosts could run in parallel and write to a shared output location.

I have not tested this myself, because I have not had the resources or the need to do so.

So, if you have multiple machines, the simplest options are:

• one shared instance with shared queues and shared output
• or separate independent instances

In HPC, a practical option is simply to rebuild the database from scratch for each job.

Best regards,
Francisco