Install revscoring on OS X

[bkowshik]

1. Per documentation on README

$ brew install aspell --with-all-languages
$ brew install enchant

$ mkvirtualenv wikiai_revscoring
pip install --no-binary pyenchant revscoring
...
Collecting scipy<0.17.999,>=0.13.3 (from revscoring)
  Using cached scipy-0.17.1.tar.gz
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/numpy/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
    Couldn't find index page for 'numpy' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
    No local packages or working download links found for numpy>=1.6.2
...

2. This looks like a known issue!

• pip install pandas fails to download numpy dependency

Per comment here, installing numpy first and then installing revscoring is the solution.

$ pip install numpy

$ pip install --no-binary pyenchant revscoring
Running setup.py bdist_wheel for scikit-learn ... error
Complete output from command /Users/bkowshik/.virtualenvs/wikiai_revscoring/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/_0/75g04h290zz51r_l4k_f22zr0000gn/T/pip-build-r55j2ov3/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/_0/75g04h290zz51r_l4k_f22zr0000gn/T/tmpdj729dstpip-wheel- --python-tag cp36:
Partial import of sklearn during the build process.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/folders/_0/75g04h290zz51r_l4k_f22zr0000gn/T/pip-build-r55j2ov3/scikit-learn/setup.py", line 247, in <module>
    setup_package()
  File "/private/var/folders/_0/75g04h290zz51r_l4k_f22zr0000gn/T/pip-build-r55j2ov3/scikit-learn/setup.py", line 237, in setup_package
    .format(scipy_req_str, instructions))
ImportError: Scientific Python (SciPy) is not installed.
scikit-learn requires SciPy >= 0.9.
Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html

3. Should scipy be installated seperately as well?

$ pip install scipy

$ pip install --no-binary pyenchant revscoring
...
----------------------------------------
Failed building wheel for scikit-learn
Running setup.py clean for scikit-learn
...
  Running setup.py install for pyenchant ... error
    Complete output from command /Users/bkowshik/.virtualenvs/wikiai_revscoring/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/_0/75g04h290zz51r_l4k_f22zr0000gn/T/pip-build-w7tlixyc/pyenchant/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/_0/75g04h290zz51r_l4k_f22zr0000gn/T/pip-vtwens7f-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/bkowshik/.virtualenvs/wikiai_revscoring/bin/../include/site/python3.6/pyenchant:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/bkowshik/.virtualenvs/wikiai_revscoring/lib/python3.6/site-packages/setuptools/__init__.py", line 11, in <module>
        from setuptools.extension import Extension
      File "/Users/bkowshik/.virtualenvs/wikiai_revscoring/lib/python3.6/site-packages/setuptools/extension.py", line 8, in <module>
        from .dist import _get_unpatched
      File "/Users/bkowshik/.virtualenvs/wikiai_revscoring/lib/python3.6/site-packages/setuptools/dist.py", line 19, in <module>
        import pkg_resources
      File "/Users/bkowshik/.virtualenvs/wikiai_revscoring/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1724, in <module>
        register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
    AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'

4. What if I remove --no-binary?

$ pip install pyenchant revscoring
...
Successfully installed pyenchant-1.6.8 revscoring-1.3.10

Questions

• Is the ^ procesure, (step 4) the right way to install revscoring?
• Should we use the Docker image instead?

---

cc: @halfak @geohacker

[halfak]

I've no way to test Mac OS. If your strategy works, then I'm a fan of making that the recommended installation strategy in the README.

Also, the numpy/scipy bug is still a thing!? Arg.

[bkowshik]

Also, the numpy/scipy bug is still a thing!? Arg.

I don't think it is now. The following is what I did and it looks like it works.

# Create a new virtual environment for revscoring.
$ mkvirtualenv wikiai_revscoring

# Install revscoring.
$ pip install pyenchant revscoring

[halfak]

do you have to include pyenchant in the install call? It's one of our requirements, so it should work without installing that separately.

[paulkernfeld]

Ah, hey, just saw this. Installing scipy and scikit-learn on MacOS has always been tough. The right installation method depends on a few different factors:

• The version of MacOS
• The versions of scipy and scikit-learn
• The version of Python

In my opinion, the ideal setup would be to just use wheels. That way, the user doesn't need to install and set up gfortran and gcc, which can be pretty involved.

The trick with wheels is finding out which ones are available for which versions of Python. Looking at the release for scipy 1.2.3, they offer MacOS wheels for Python 3.4, 3.5, 3.6, and 3.7. Similarly, the release for scikit-learn 0.22.1 has wheels for Python 3.5, 3.6, 3.7, and 3.8.

So, with the current versions of scipy and scikit-learn, maybe we should recommend that MacOS users use Python 3.5, 3.6, or 3.7? I was able to get all the dependencies installed with Python 3.7 so I can at least verify that that works.

The alternative would be to actually build these libraries from source. For example, scipy has a GitHub Actions build for MacOS, so you should be able to build Scipy by basically copying what their build does. However, it's pretty complex (about 50 lines of shell script) and it might cause other problems because it relies on an older version of GCC and it requires modifying some global settings. So I wouldn't recommend doing this unless there's a strong-ish reason to.

[Simonmaignan]

Hi there, not sure if this is the right place, but I just noticed that the aspel installation on OS X 12.1 via homebrew doesn't recognize the --with-all-languages.
Installing the package without option seems to work

Error: invalid option: --with-all-languages
(wikimedia-ml) simonmaignan@Simons-MacBook-Pro wikimedia % brew install aspell                     
==> Downloading https://ghcr.io/v2/homebrew/core/aspell/manifests/0.60.8-1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/aspell/blobs/sha256:d75d3b4e2929123244786e9adb7ff335ec75943617e7dbd855d71c7ce
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:d75d3b4e2929123244786e9adb7ff335ec7594361
######################################################################## 100.0%
==> Pouring aspell--0.60.8.arm64_monterey.bottle.1.tar.gz
🍺  /opt/homebrew/Cellar/aspell/0.60.8: 767 files, 322.8MB
==> Running `brew cleanup aspell`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

Should we create a new issue for updating the documentation or does someone want to reproduce the issue?

[halfak]

I'd welcome an update to the documentation. The OSX install is rarely used, but I'd be very happy to have it improved.

[Simonmaignan]

@halfak , I am having issues installing pyenchant (see comment).
I will update the documentations once I am able to fully use revscoring on my MacBook M1.

[Simonmaignan]

I have a few questions.
Here is my configuration

• MacBook Pro M1
• MacOS Monterey 12.1

Can someone explain me which version is installed when, as per documentation, I install

pip install --no-binary pyenchant revscoring

From pip list, I have version 0.2.1

─❯ pip list | grep revscoring
revscoring           0.2.1

From pip index versions revscoring and from the changelog, it seems that 0.2.1 is one of the first version and is very old.
Does it mean that there is no newer version compatible with Mac?

Did someone manage to install the latest version of revscoring on a Mac? @paulkernfeld , @bkowshik ?

[halfak]

pip should have pulled in 2.11. There should be no architectural limitation on where you can install revscoring.

https://pypi.org/project/revscoring/

I don't know why you would end up with such an old version.

[Simonmaignan]

Hi everyone
Thanks @halfak for your answer.
I was finally able to install the revscoring in my Apple silicon python environment.

Configuration

• MacBook M1 Pro
• MacOS Monterey 12.1

Prerequesite

I was not able to install the revscoring dependencies with the Apple Silicon native (arm64) packages.
I didn't dig deep why it didn't work with the native arm64 packages because I was able to install revscoring and its dependencies in the simulated -x86_64 environment using Roseta2.
For those interested by documentations about managing parallel arm64 and -x86_64 using conda, I refer you to another comment from me.

As @paulkernfeld mentioned above, some of the dependencies only have wheels for python 3.7 max so I used Python 3.7.12.

Installation instruction

Aspell

The instruction for the aspell installation on Mac mention a --with-all-languages which doesn't exist anymore.

brew install aspell

worked fine for me.

Enchant

Please refer to this issue for the enchant installation on Apple Silicon.

Python packages

pip install --no-binary pyenchant revscoring

didn't work for me. This command raised a few errors and install the version 0.2.1 of revscoring.

pip install revscoring

installed revscoring 2.11 and all its dependencies without error and I was able to run some test code without issue.

Can someone try to reproduce the installation step on a Mac?

@halfak , should we update the documentation?

[halfak]

What a fantastic writeup! Thank you for your work! I'd welcome an update to the documentation. Any documentation of your process would be an improvement to what we have right now -- which apparently was not working for folks.

[Simonmaignan]

@paulkernfeld , @bkowshik , could you try to reproduce the installation on MacOS with my change from #511 ?