Making Matlab Runtime work with ridge extraction

[Lemez]

Hi there,
I'm collaborating on an EEG project in Croatia, and am having difficulties porting the ridge extraction from Matlab to python using pymodalib.
I'm running OSX 15.7.1, currently using Python 3.7.3 and have installed Matlab Runtime Engine 9.6. I don't have a separate Matlab license.

I am getting a MatlabLibraryException when I try to implement wavelet transform or ridge extraction using matlab (implementation="python" works fine for wavelet transform), but as ridge_extraction requires Matlab, I can't get past the error. The DYLD_LIBRARY_PATH seems to be correct, have added it to both .bash_profile and as an environment variable in the script.

Here is my script:

import pymodalib
import pandas as pd
import csv 
import numpy as np
from scipy.io import loadmat
import h5py
import os
import sys
from matplotlib import pyplot as plt

os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"

patient="x"
samplerate = 500.0
fmax_arg = 1.2
fmin_arg = 0.8
file = "path/to/file"

with h5py.File(file, 'r') as f:
  dset = f[patient]
  data = f.get(patient)
  signal = np.array(data)
  times = pymodalib.generate_times(signal, samplerate)
  wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
  iamp,iphi,ifreq = pymodalib.ridge_extraction(tfr=wt,frequencies=freq,fs=float(samplerate),method='direct',wopt=wtdict,implementation='matlab')

Traceback is as follows:

Traceback (most recent call last):
  File "moda.py", line 189, in <module>
    wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/algorithms/wavelet.py", line 191, in wavelet_transform
    **kwargs,
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/utils/decorators.py", line 72, in wrapper
    f"The MATLAB-packaged library '{module}' is not installed. Please install "
pymodalib.utils.matlab_runtime.MatlabLibraryException: The MATLAB-packaged library 'WT' is not installed. Please install PyMODA, which supplies the MATLAB-packaged libraries, or check if there is a pure-Python implementation by passing 'implementation="python"' to the function.

Many thanks for your help in advance.

[spmccormack]

Sorry about this, I forgot it was a bit poorly documented! I'll see what I can do to improve it.

The MATLAB-packaged libraries (like WT) are currently bundled with the PyMODA source code, even if you're using PyMODAlib without PyMODA. You just need to download the PyMODA repository as a zip file (and extract it) or clone the repo with the terminal command:

git clone https://github.com/luphysics/PyMODA.git

Then you can open a terminal in the main PyMODA folder, and install the required packages with the following command:

# Note: try replacing 'python3' in the command with 'python' or 'python3.7' if it doesn't work.
python3 packages/install.py -ymu 

After running the command, the error with your script should go away and you can delete the PyMODA folder.

As a sidenote: I've noticed an error in PyMODAlib which occurs with the example script for ridge extraction; it might also affect your script. I'll comment when I've fixed it.

[spmccormack]

I've fixed the exception which was raised when running ridge_extraction.py from examples/ridge_extraction. If you experience it too, you can upgrade your PyMODAlib with the command:

python3 -m pip install --upgrade pymodalib

(You may need to use python or python3.7 in the command instead.)

[Lemez]

Thank you for the quick reply. I cloned the repo inside my current working folder, and ran the upgrade command as you suggested. I have a new error, which is progress! I can confirm that the file matlabruntimeforpython3_7.so is in the right place inside /Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64.

Exception caught during initialization of Python interface. Details: dlopen(/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so, 2): Library not loaded: @rpath/libpython3.7m.dylib
  Referenced from: /Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so
  Reason: image not found
Traceback (most recent call last):
  File "moda.py", line 189, in <module>
    wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/algorithms/wavelet.py", line 191, in wavelet_transform
    **kwargs,
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/utils/decorators.py", line 104, in wrapper
    return func(*args, **kwargs)
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/implementations/matlab/wavelet/wavelet_transform.py", line 61, in wavelet_transform
    import WT
  File "/Users/macintosh/.local/lib/python3.7/site-packages/WT/__init__.py", line 283, in <module>
    _pir.import_cppext()
  File "/Users/macintosh/.local/lib/python3.7/site-packages/WT/__init__.py", line 276, in import_cppext
    self.cppext_handle = importlib.import_module("matlabruntimeforpython" + self.interpreter_version)
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: dlopen(/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so, 2): Library not loaded: @rpath/libpython3.7m.dylib
  Referenced from: /Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so
  Reason: image not found

[spmccormack]

Looks similar to an issue someone had here.

Could you try this terminal command:

find /Users/macintosh/.pyenv/versions/3.7.3/lib/ -name "libpython3.7m.dylib"

And reply with the output? Then we should be able to symlink the file so the Matlab Runtime can find it.

[Lemez]

There is no output from that command, but there is a file called libpython3.7m.a in /Users/macintosh/.pyenv/versions/3.7.3/lib

[spmccormack]

Hmm, could you try adding this line to your script (just before the line which starts with os.environ["DYLD_LIBRARY_PATH"]):

os.environ["LD_LIBRARY_PATH"] = f"/Users/macintosh/.pyenv/versions/3.7.3/lib:{os.environ['LD_LIBRARY_PATH']}"

And see if it works?

If it still doesn't work, try adding this as well (just after the line added above):

os.environ["PATH"] = f"{os.environ['LD_LIBRARY_PATH']}:{os.environ['PATH']}"

[Lemez]

Curious, these both fail as LD_LIBRARY_PATH raises a KeyError. I checked os.environ.keys() and the (edited) output is:

KeysView(environ({'SHELL': '/usr/local/bin/bash', 
'PYENV_HOOK_PATH': '/Users/macintosh/.pyenv/pyenv.d
:/usr/local/Cellar/pyenv/1.2.21/pyenv.d
:/usr/local/etc/pyenv.d
:/etc/pyenv.d
:/usr/lib/pyenv/hooks', 
'PYENV_SHELL': 'bash', 'XPC_FLAGS': '0x0',  'HOME': '/Users/Macintosh',
  'PYENV_DIR': '/Users/macintosh/Documents/_localDev/xs',  'USER': 'macintosh', 'COLORFGBG': '7;0', 'LC_TERMINAL_VERSION': '3.4.8', 'SHLVL': '1', 'XPC_SERVICE_NAME': '0',
 'PYENV_ROOT': '/Users/macintosh/.pyenv',
 'PATH': '/Users/macintosh/.pyenv/versions/3.7.3/bin
:/usr/local/Cellar/pyenv/1.2.21/libexec
:/usr/local/Cellar/pyenv/1.2.21/plugins/python-build/bin
:/usr/local/sbin:/Users/macintosh/.pyenv/shims
:/Users/macintosh/.gem/ruby/2.7.2/bin
:/Users/macintosh/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin
:/Users/macintosh/.rubies/ruby-2.7.2/bin
:/Users/macintosh/.bin:/Users/macintosh/.pyenv/shims:/Users/macintosh/.pyenv/shims
:/Users/macintosh/.pyenv/shims:/usr/local/sbin
:/Users/macintosh/.pyenv/shims
:/Users/macintosh/.bin:/usr/local/sbin:/Users/macintosh/.pyenv/shims
:/Users/macintosh/.bin:/usr/local/sbin:/Users/macintosh/.pyenv/shims
:/Users/macintosh/.bin
:/usr/local/bin
:/usr/bin
:/bin
:/usr/sbin
:/sbin',
 'OLDPWD': '/Users/macintosh/Documents/_localDev/xs/pymoda', }))

Perhaps to add LD_LIBRARY_PATH = /Users/macintosh/.pyenv/versions/3.7.3/lib/libpython3.7m.a to my .bash_profile?

[spmccormack]

Ah, maybe just try adding these lines instead (I'm using diff syntax to make it clearer which lines have been added, so don't include the leading + symbol):

import pymodalib
import pandas as pd
import csv 
import numpy as np
from scipy.io import loadmat
import h5py
import os
import sys
from matplotlib import pyplot as plt

+os.environ["LD_LIBRARY_PATH"] = f"/Users/macintosh/.pyenv/versions/3.7.3/lib"
+os.environ["PATH"] = f"{os.environ['LD_LIBRARY_PATH']}:{os.environ['PATH']}"

os.environ["DYLD_LIBRARY_PATH"] = "/Applications/MATLAB/MATLAB_Runtime/v96/runtime/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64:/Applications/MATLAB/MATLAB_Runtime/v96/extern/bin/maci64"
+os.environ["DYLD_LIBRARY_PATH"] = f'{os.environ["LD_LIBRARY_PATH"]}:{os.environ["DYLD_LIBRARY_PATH"]})'

patient="x"
samplerate = 500.0
fmax_arg = 1.2
fmin_arg = 0.8
file = "path/to/file"

with h5py.File(file, 'r') as f:
  dset = f[patient]
  data = f.get(patient)
  signal = np.array(data)
  times = pymodalib.generate_times(signal, samplerate)
  wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
  iamp,iphi,ifreq = pymodalib.ridge_extraction(tfr=wt,frequencies=freq,fs=float(samplerate),method='direct',wopt=wtdict,implementation='matlab')

If that works then we'll know whether we should edit ~/.bash_profile.

[Lemez]

Unfortunately it is still failing to find this libpython3.7m.dylib file:

Exception caught during initialization of Python interface. Details: dlopen(/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so, 2): Library not loaded: @rpath/libpython3.7m.dylib
  Referenced from: /Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so
  Reason: image not found
Traceback (most recent call last):
  File "moda.py", line 191, in <module>
    wt, freq,wtdict = pymodalib.wavelet_transform(signal, samplerate, resolution=wt_resolution,cut_edges=False,fmax=fmax_arg,fmin=fmin_arg,wavelet=wavelet,return_opt=True,implementation='matlab')
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/algorithms/wavelet.py", line 191, in wavelet_transform
    **kwargs,
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/utils/decorators.py", line 104, in wrapper
    return func(*args, **kwargs)
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pymodalib/implementations/matlab/wavelet/wavelet_transform.py", line 61, in wavelet_transform
    import WT
  File "/Users/macintosh/.local/lib/python3.7/site-packages/WT/__init__.py", line 283, in <module>
    _pir.import_cppext()
  File "/Users/macintosh/.local/lib/python3.7/site-packages/WT/__init__.py", line 276, in import_cppext
    self.cppext_handle = importlib.import_module("matlabruntimeforpython" + self.interpreter_version)
  File "/Users/macintosh/.pyenv/versions/3.7.3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: dlopen(/Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so, 2): Library not loaded: @rpath/libpython3.7m.dylib
  Referenced from: /Applications/MATLAB/MATLAB_Runtime/v96/bin/maci64/matlabruntimeforpython3_7.so
  Reason: image not found

I searched through my user directory and I can't see a libpython.*.dylib file anywhere.

[Lemez]

The pyenv/versions/3.7.3/lib directory contains:

python3.7 directory
pkgconfig directory
libpython3.7m.a

The pyenv/versions/3.7.3/include directory contains a dir called python3.7m which contains a number of .h files.

[Lemez]

Would you recommend to uninstall pyenv and install 3.7.3 globally?

[spmccormack]

Would you recommend to uninstall pyenv and install 3.7.3 globally?

Yup, I think uninstalling pyenv and using Homebrew to install Python 3.7 is the easiest solution. I've only ever tested it with Homebrew. I'm not sure which command would be used to install Python 3.7 with Homebrew (rather than a newer version) though - maybe brew install python@3.7 or brew install python@3.7.3 etc.

[Lemez]

Rrrrr... now scipy is playing up:

Traceback (most recent call last):
  File "moda.py", line 2, in <module>
    import pymodalib
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymodalib/__init__.py", line 19, in <module>
    from pymodalib.algorithms.group_coherence import group_coherence, dual_group_coherence
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymodalib/algorithms/group_coherence.py", line 25, in <module>
    from scipy.stats import ranksums
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/stats/__init__.py", line 441, in <module>
    from .stats import *
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/stats/stats.py", line 37, in <module>
    from scipy.spatial.distance import cdist
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/spatial/__init__.py", line 102, in <module>
    from ._geometric_slerp import geometric_slerp
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/spatial/_geometric_slerp.py", line 8, in <module>
    from scipy.spatial.distance import euclidean
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/spatial/distance.py", line 124, in <module>
    from ..special import rel_entr
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/special/__init__.py", line 643, in <module>
    from . import _ufuncs
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/special/_ufuncs.cpython-37m-darwin.so, 2): Symbol not found: __gfortran_stop_numeric_f08
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/special/_ufuncs.cpython-37m-darwin.so
  Expected in: /Applications/MATLAB/MATLAB_Runtime/v96/sys/os/maci64/libgfortran.3.dylib
 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/special/_ufuncs.cpython-37m-darwin.so

Have installed pymodalib-0.11.3b1 and python 3.7.3 directly from python.org (taking care to follow uninstall pyenv instructions here

[spmccormack]

Could you try updating pip:

python3 -m pip install --user --upgrade pip

And then install these particular numpy and scipy versions:

python3 -m pip install --force-reinstall scipy==1.4.1 numpy==1.18.3

Edit: Added --force-reinstall parameter to the second command, run the edited command if you've already tried the unedited one and it didn't work.

[spmccormack]

If it's still failing after that (check the edit in my previous comment if you haven't seen it), check for the location of the fortran dylib:

find ~ -name "libgfortran.3.dylib" 

If nothing appears from that command, try:

sudo find /Library/Frameworks/ -name "libgfortran.3.dylib"