KLIFS multiprocessing #74
Assignees
Labels
module-databases-klifs
Concerns opencadd.databases.klifs module
Comments
|
Is this related? Yelp/bravado#471 |
dominiquesydow
added
the
module-databases-klifs
|
@schallerdavid, I don't know why but I missed this issue. If fixed, can you please close this issue? |
|
@dominiquesydow I can still observe this behavior. I get the following error message when running two processes in parallel with the above mentioned script: May this also be an intended behavior from the KLIFS side to prevent it from being overloaded with requests from a single IP? |
|
Thanks, @schallerdavid, could you please provide here a minimum example that reproduces this error? Happy to look into this. |
|
Ah, I see now that you have done that in the first message, thanks! :) |
|
I cannot figure out right now why For For your example it looks like this (the job is to subset structures by input kinase): from itertools import repeat
import multiprocessing
import sys
from opencadd.databases.klifs import setup_remote
number_processes = int(sys.argv[1])
def run_job(kinase, klifs_session):
structures = klifs_session.structures.all_structures()
print(
f"All structures: {len(structures)}\n"
f"{kinase} structures: {len(structures[structures['kinase.klifs_name'] == kinase])}"
)
if __name__ == "__main__":
remote = setup_remote()
kinases = ["EGFR", "BRAF", "KIT", "LOK"]
pool = multiprocessing.Pool(processes=number_processes)
pool.starmap(run_job, zip(kinases, repeat(remote)))
pool.close()
pool.join()Output: Would that work for you - or do you need to pass the process IDs explicitly as you did in your script? |
|
This looks like a great solution. Thanks @dominiquesydow |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey there,
since my KinoML code is getting more and more hungry I already played a bit with multiprocessing. However, when using the
multiprocessingfrom python I receive abravado.http_future.RequestsFutureAdapterConnectionErrorfrom opencadd's KLIFS module. This only happens if more than 1 process is using the KLIFS module at the same time. Would there be a way to use the KLIFS module in a multiprocess fashion?I attached a script (klifs_mp.zip) to reproduce the error. It only requires a current
opencaddinstallation.# run 1 job at the same timepython klifs_mp.py 1# run 2 jobs at the timepython klifs_mp.py 2Thanks in advance!
The text was updated successfully, but these errors were encountered: