Making Matlab Runtime work with ridge extraction #3
Comments
|
Sorry about this, I forgot it was a bit poorly documented! I'll see what I can do to improve it. The MATLAB-packaged libraries (like WT) are currently bundled with the PyMODA source code, even if you're using PyMODAlib without PyMODA. You just need to download the PyMODA repository as a zip file (and extract it) or clone the repo with the terminal command: git clone https://github.com/luphysics/PyMODA.gitThen you can open a terminal in the main PyMODA folder, and install the required packages with the following command: # Note: try replacing 'python3' in the command with 'python' or 'python3.7' if it doesn't work.
python3 packages/install.py -ymu After running the command, the error with your script should go away and you can delete the PyMODA folder. As a sidenote: I've noticed an error in PyMODAlib which occurs with the example script for ridge extraction; it might also affect your script. I'll comment when I've fixed it. |
|
I've fixed the exception which was raised when running python3 -m pip install --upgrade pymodalib(You may need to use |
|
Thank you for the quick reply. I cloned the repo inside my current working folder, and ran the upgrade command as you suggested. I have a new error, which is progress! I can confirm that the file matlabruntimeforpython3_7.so is in the right place inside |
|
Looks similar to an issue someone had here. Could you try this terminal command: find /Users/macintosh/.pyenv/versions/3.7.3/lib/ -name "libpython3.7m.dylib"And reply with the output? Then we should be able to symlink the file so the Matlab Runtime can find it. |
|
There is no output from that command, but there is a file called |
|
Hmm, could you try adding this line to your script (just before the line which starts with os.environ["LD_LIBRARY_PATH"] = f"/Users/macintosh/.pyenv/versions/3.7.3/lib:{os.environ['LD_LIBRARY_PATH']}"And see if it works? If it still doesn't work, try adding this as well (just after the line added above): os.environ["PATH"] = f"{os.environ['LD_LIBRARY_PATH']}:{os.environ['PATH']}" |
|
Curious, these both fail as LD_LIBRARY_PATH raises a KeyError. I checked os.environ.keys() and the (edited) output is: Perhaps to add |
|
Ah, maybe just try adding these lines instead (I'm using diff syntax to make it clearer which lines have been added, so don't include the leading import pymodalib
import pandas as pd
import csv
import numpy as np
from scipy.io import loadmat
import h5py
import os
import sys
from matplotlib import pyplot as plt
+os.environ["LD_LIBRARY_PATH"] = f"/Users/macintosh/.pyenv/versions/3.7.3/lib"
+os.environ["PATH"] = f"{os.environ['LD_LIBRARY_PATH']}:{os.environ['PATH']}"
os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
+os.environ["DYLD_LIBRARY_PATH"] = f'{os.environ["LD_LIBRARY_PATH"]}:{os.environ["DYLD_LIBRARY_PATH"]})'
patient="x"
samplerate = 500.0
fmax_arg = 1.2
fmin_arg = 0.8
file = "path/to/file"
with h5py.File(file, 'r') as f:
dset = f[patient]
data = f.get(patient)
signal = np.array(data)
times = pymodalib.generate_times(signal, samplerate)
wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
iamp,iphi,ifreq = pymodalib.ridge_extraction(tfr=wt,frequencies=freq,fs=float(samplerate),method='direct',wopt=wtdict,implementation='matlab')If that works then we'll know whether we should edit |
|
Unfortunately it is still failing to find this libpython3.7m.dylib file: I searched through my user directory and I can't see a libpython.*.dylib file anywhere. |
|
The pyenv/versions/3.7.3/lib directory contains:
The pyenv/versions/3.7.3/include directory contains a dir called python3.7m which contains a number of .h files. |
|
Would you recommend to uninstall pyenv and install 3.7.3 globally? |
Yup, I think uninstalling pyenv and using Homebrew to install Python 3.7 is the easiest solution. I've only ever tested it with Homebrew. I'm not sure which command would be used to install Python 3.7 with Homebrew (rather than a newer version) though - maybe |
|
Rrrrr... now scipy is playing up: Have installed pymodalib-0.11.3b1 and python 3.7.3 directly from python.org (taking care to follow uninstall pyenv instructions here |
|
Could you try updating python3 -m pip install --user --upgrade pipAnd then install these particular numpy and scipy versions: python3 -m pip install --force-reinstall scipy==1.4.1 numpy==1.18.3Edit: Added |
|
If it's still failing after that (check the edit in my previous comment if you haven't seen it), check for the location of the fortran find ~ -name "libgfortran.3.dylib" If nothing appears from that command, try: sudo find /Library/Frameworks/ -name "libgfortran.3.dylib" |
|
The force reinstalls result in the same error message as previously. |
|
Ok, could you try adding this line: import pymodalib
import pandas as pd
import csv
import numpy as np
from scipy.io import loadmat
import h5py
import os
import sys
from matplotlib import pyplot as plt
os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
+os.environ["DYLD_LIBRARY_PATH"] = f'/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/.dylibs:/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/.dylibs:{os.environ["DYLD_LIBRARY_PATH"]})'
patient="x"
samplerate = 500.0
fmax_arg = 1.2
fmin_arg = 0.8
file = "path/to/file"
with h5py.File(file, 'r') as f:
dset = f[patient]
data = f.get(patient)
signal = np.array(data)
times = pymodalib.generate_times(signal, samplerate)
wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
iamp,iphi,ifreq = pymodalib.ridge_extraction(tfr=wt,frequencies=freq,fs=float(samplerate),method='direct',wopt=wtdict,implementation='matlab') |
|
If that doesn't work, try removing the line from the previous comment and then creating this symlink: sudo ln -s /Library/Frameworks//Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/.dylibs/libgfortran.3.dylib /Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64 |
|
These both give the same error message as above. Output for the symlink above is |
|
Hmm, it would be interesting to see what the error message is when you insert these lines here: +import os
+if "DYLD_LIBRARY_PATH" in os.environ:
+ del os.environ["DYLD_LIBRARY_PATH"]
+from scipy.stats import ranksums
import pymodalib
import pandas as pd
import csv
import numpy as np
from scipy.io import loadmat
import h5py
import os
import sys
from matplotlib import pyplot as plt
os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
patient="x"
samplerate = 500.0
fmax_arg = 1.2
fmin_arg = 0.8
file = "path/to/file"
with h5py.File(file, 'r') as f:
dset = f[patient]
data = f.get(patient)
signal = np.array(data)
times = pymodalib.generate_times(signal, samplerate)
wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
iamp,iphi,ifreq = pymodalib.ridge_extraction(tfr=wt,frequencies=freq,fs=float(samplerate),method='direct',wopt=wtdict,implementation='matlab')This might help us to find a workaround. |
|
Relevant os.environ output is: The error message after adding the above code is: |
|
Perhaps this all means that I need to get scipy working as a first step. I'll search for some options. |
|
On digging a bit, I see that I don;t have a PYTHONPATH variable set, perhaps this is the issue. Will look into it further this evening, thanks so much for your help so far. |
|
No worries, and sorry I can't help more effectively - unfortunately it's not really possible for me to get macOS running in a VM for testing. The weird part is that it seems to be trying to find the It looks to me like there's an environment variable which is causing Scipy to try and use an incompatible I'd suggest trying to remove the import os
assert not "DYLD_LIBRARY_PATH" in os.environ, "Env variable should not be set"
from scipy.stats import ranksums
print("Success")If it doesn't work, we should know it's definitely a problem with Scipy. You could also try setting the PYTHONPATH to |
|
The script above works, so it seems that it is not a problem with scipy. The good news is that pymodalib is loading as before when I change implementation to "python", the bad news is that the implementation="matlab" is still causing the error:
Is there any way I might try to export ridge_extraction from PYMODA and use that function directly? Another alternative might be to try a different Python VM like anaconda and see if that runs into similar issues? |
I think this might be happening because you reinstalled Python; have you tried following the instructions in this comment again? Hopefully that will fix this issue and we'll be back to the more recent problem. I should be able to implement a workaround for it today or tomorrow. |
|
Thank you. I'm travelling today but should be able to try the reinstall
tomorrow. I'll let you know as soon as it's done.
…On Sat, 21 Aug 2021, 16:26 spmccormack, ***@***.***> wrote:
The script above works, so it seems that it is not a problem with scipy.
The good news is that pymodalib is loading as before when I change
implementation to "python", the bad news is that the
implementation="matlab" is still causing the error:
pymodalib.utils.matlab_runtime.MatlabLibraryException: The MATLAB-packaged
library 'WT' is not installed. Please install PyMODA
I think this might be happening because you reinstalled Python; have you
tried following this comment again
<#3 (comment)>?
Hopefully that will fix this issue and we'll be back to the more recent
problem. I should be able to implement a workaround for it today or
tomorrow.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJZFSUZQA52WYF2PILFIJLT56ZPXANCNFSM5CE7ZUFQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
|
|
By the way, I've pushed a change which may fix the issue. Could you try installing from the dev branch using: python -m pip install git+https://github.com/luphysics/PyMODAlib.git@dev --upgradeEdit: There was a small mistake in that version; in the unlikely event that you'd already installed it, just re-run the command above!
And then run your script, ensuring that the -os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
+from pymodalib.utils import macos
+
+macos.configure_dyld_library_path(
+ "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:"
+ "/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:"
+ "/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:"
+ "/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
+)Here's an easier-to-copy block (just make sure you remember to remove the old from pymodalib.utils import macos
macos.configure_dyld_library_path(
"/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:"
"/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:"
"/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:"
"/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
) |
|
Thanks very much. I was unexpectedly away from the comp over the last week
but back on it tomorrow, I let you know how it goes!
…On Mon, 30 Aug 2021, 23:15 spmccormack, ***@***.***> wrote:
By the way, I've pushed a change which may fix the issue. Could you try
installing from the dev branch using:
python -m pip install ***@***.***
And then run your script, ensuring that this line is still present near
the start:
os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJZFSWDBM34UXKDMIYMR4TT7PYHVANCNFSM5CE7ZUFQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
I followed the above suggestions (reinstalling PYMODA and then the pymodalib fix as above. Now I have the following ImportError:
The wavelet transform with python implementation is working as intended. I think we're close! |
|
Sorry for the slow reply; I've pushed a fix for the import issue you were experiencing, could you try uninstalling your PyMODAlib: python -m pip uninstall pymodalib -yAnd then following the instructions from this comment again? Hopefully we'll get some useful results this time! |
|
I've followed the uninstall and reinstall as you suggested. When I either try to run Verbose output is as follows: |
Sorry, could you clarify which error message you get? I would expect it to hang for a while; it's hard to judge how long it will take for it to run, but the example script from this repository (which uses a signal of around 10,000 samples) will take around 1-2 minutes on a Macbook, so you could expect substantially longer calculations if your signal is larger. There's also a long delay on the first run, I think caused by the Matlab Runtime building some caches. There's no real way to monitor its progress and it does normally appear to hang, but you can watch the CPU and RAM usage to see if it's doing anything. This part is actually normal, so nothing to worry about: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymodalib/utils/decorators.py:96: RuntimeWarning: Trying to set MATLAB Runtime variables...
"Trying to set MATLAB Runtime variables...", RuntimeWarningJust to check, was the verbose output caused by stopping the program with Ctrl+C, or did it happen without intervention? |
|
Sorry for the confusion: I mean that I get the above error message both when I run I'm fairly sure that the script is hanging (for both I'll leave the EDIT: unfortunately no progress after leaving it all night |
|
Sorry for the late reply! Could you see if there's any difference in behaviour if you put everything inside an if __name__ == "__main__":
# Note: This is based on the code from your first post, just as an example. (Use your newer code)
import pymodalib
import pandas as pd
import csv
import numpy as np
from scipy.io import loadmat
import h5py
import os
import sys
from matplotlib import pyplot as plt
os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
patient="x"
samplerate = 500.0
fmax_arg = 1.2
fmin_arg = 0.8
file = "path/to/file"
with h5py.File(file, 'r') as f:
dset = f[patient]
data = f.get(patient)
signal = np.array(data)
times = pymodalib.generate_times(signal, samplerate)
wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
iamp,iphi,ifreq = pymodalib.ridge_extraction(tfr=wt,frequencies=freq,fs=float(samplerate),method='direct',wopt=wtdict,implementation='matlab')Also, if that doesn't make any difference, could you try adding the following lines at the start while retaining the new if __name__ == "__main__":
### These lines added ###
import multiprocess
import multiprocessing
start_method = "spawn"
multiprocess.set_start_method(start_method)
multiprocessing.set_start_method(start_method)
### End of added code ###
# Rest of code continues here
import pymodalib
import pandas as pd
# ... |
Hi there,
I'm collaborating on an EEG project in Croatia, and am having difficulties porting the ridge extraction from Matlab to python using pymodalib.
I'm running OSX 15.7.1, currently using Python 3.7.3 and have installed Matlab Runtime Engine 9.6. I don't have a separate Matlab license.
I am getting a MatlabLibraryException when I try to implement wavelet transform or ridge extraction using matlab (implementation="python" works fine for wavelet transform), but as ridge_extraction requires Matlab, I can't get past the error. The DYLD_LIBRARY_PATH seems to be correct, have added it to both .bash_profile and as an environment variable in the script.
Here is my script:
Traceback is as follows:
Many thanks for your help in advance.
The text was updated successfully, but these errors were encountered: