Can't open magi_workflow.py when running the script run_full_workflow_test.sh

[vitorlimac2]

Hi.

First of all, congrats for your really nice and important work.

I have tried to run the magi test script (run_full_workflow_test.sh), but it prints a error about the script magi_workflow.py:

python: can't open file '../../workflow/magi_workflow.py': [Errno 2] No such file or directory

In fact, we can't see this script in the workflow directory. How can I fix this?

Sincerely

[ampend]

Hi,

I am also receiving the same error as above. Any updates?

Thank you

[benbowen]

It's a problem with the python path not getting setup right during the installation. Can you try either setting it in the python code with sys.path.insert() or in shell with PYTHONPATH env variable? You will need the paths for running BLAST, accessing the packages, and accessing the databases.

[ampend]

Thank you for getting back!

Two questions, the first is in continuation of the first comment in this thread, there is nomagi_workflow.py in the GitHub workflow directory (https://github.com/biorack/magi/tree/master/workflow). Is it supposed to unpack from somewhere else?

Secondly, to alter the $PYTHONPATH, where should I be setting the variable? You have mentioned 'in the python code', but in which script's code should I include this?

Thank you,
Amanda

[benbowen]

Sorry I was confused, but I see the problem now. That test is obsolete and should be removed. We are slowly migrating from what we call MAGI v1 towards a new one called MAGI v2. There is an intermediate version we have that is a gap in between these. The original MAGI v1 commands (like run_full_workflow_test.sh) will be broken until we get things sorted out and the repo cleaned up.

the original workflow wasn't cleanly broken up into individual steps and the version current on the main branch is.

Something similar could be recreated from this pattern:

https://github.com/biorack/magi/blob/master/run_magi.sh#L26

but using the test data for coelicolor as predefined inputs.

[ampend]

I see, I'm excited to try v2 when it comes out.

I was able to get the workflow shell script, as is on the current GitHub version (downloaded 2021-02-17), to go through all steps in your link above (https://github.com/biorack/magi/blob/master/run_magi.sh#L26) except the last step: python $magi_path/workflow/magi_workflow_scoring.py.

At this last step, the job just ends, and nothing is reported to the error log. Many of the .pkl outputs are generated, but the final scoring does not complete to create the .csv, and I'm eager to see how well your program can make sense of our metabolite data.

As I think this may be because I'm working with v2 in progress sets of scripts, for now, is there an archived version here on GitHub of the MAGI v1 that can be accessed while the transition to MAGI v2 is complete? If so, I can pull and older push with v1 only?

Thank you again for your help.

[benbowen]

I wish we had done that, but we forgot. Are you running with features (e.g. m/z values only) or inchi keys? If you are willing to try a small test, you can make a tiny input and paste a link to the data and we will try to check it out.

MAGI 2 uses SMILES as input instead of inchi keys. It can search for compounds that are not in our compound to reaction database using reaction patterns from retrorules. https://retrorules.org/

It has precomputed compound to reactions for all our SMILES strings, but if you hit it with a totally new molecule, the job takes a bit longer since it has to make the compound to reaction links.

I think the code is there so you can try it. Its run_magi2.sh

@hleegwater might be able to help. She wrote the MAGI2 implementation, but is now in a grad program doing something different. @greensii has recently used it; so she might be able to take a look too. I am a bit out of touch with the exact details of this vintage of MAGI, but am happy to take a look and test it. Sorry for all the troubles!

It goes without saying, but we desperately need to build a full stack suite of tests.

[hleegwater]

Thanks for the tag Ben! I completely forgot that I broke the test when I split the MAGI workflow into sub-workflows (and apparently I also switched off my GitHub notifications). I can already say that if you follow the tutorial magi/docs/tutorials/MAGI tutorial.ipynb successfully, MAGI1 is probably working just fine, but I should fix the test run_full_workflow_test.sh as well.

[hleegwater]

Hi @ampend, I just updated the run_full_workflow_test.sh and it works for me (so I hope it will work for you too!)

The scoring script consists of three parts:

1. Merge all information from the previous gene-to-reaction, compound-to-reaction and reaction-to-gene searches. If there is a file merged_before_score.h5, this part was run successfully.
2. Calculate MAGI scores. This should have run successfully when you see !@# Scoring done in minutes.
3. Format data tables to add metadata and writing output to file. This should create all the files that you are interested in.

I don't exactly know what happens in your case, but it might be that MAGI cannot write the h5 file because the package Tables is missing, or that pandas has trouble sorting the dataframe.

If you have the h5 file, you can type the following commands in a python console to run MAGI slightly different:
import sys
import os
import pandas as pd
# Maybe some other ones as well

magi_path = "path_to_local_magi_folder/magi"
sys.path.insert(0,magi_path)
from workflow import magi_workflow_scoring as mg

merged_dataframe = pd.read_hdf(path_to_h5_file)

merged_dataframe = mg.calculate_scores(merged_dataframe,
reciprocal_closeness = 0.75,
final_weights = [1.0, 1.0, 1.0, 1.0],
chemnet_penalty = 4,
start_time = time.time())

After this, you have a draft dataframe with scores. The next chunk of MAGI converts gene IDs to strings, sorts some columns, drops duplicates, adds interpretable reaction IDs and renames some columns.

merged_dataframe, time_start = mg.format_table(merged_dataframe)

Finally, to create the MAGI compound or the MAGI gene table, the save outputs function can be run:
mg.save_outputs(merged_dataframe, time_start, output_dir_to_store_the_files, full_path_to_intermediate_files_dir)

Good luck! :D