Unable to run local server on an M1 Mac

[hydrosquall]

Thank you for developing this software! I had tried the public cloud instance before, and am keen to try to run it locally.

I ran into some difficulties with running the docker images locally on my Mac laptop using the latest commit 590a262 and the wiki guide . I am sharing here in case others may have ideas for a workaround or how to debug.

I've grouped the error messages into two separate sections

Installing google-re2

This error is thrown first while installing django-requirements-frozen.txt, but if we comment out this line it'll throw in every *-frozen.txt file .

38.28     Running setup.py install for google-re2: finished with status 'error'
38.28     ERROR: Command errored out with exit status 1:
38.28      command: /opt/venv/django/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7wn1ub26/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7wn1ub26/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-b4hq8o2f/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/django/include/site/python3.8/google-re2
38.28          cwd: /tmp/pip-install-7wn1ub26/google-re2/
38.28     Complete output (15 lines):
38.28     running install
38.28     running build
38.28     running build_py
38.28     creating build
38.28     creating build/lib.linux-aarch64-3.8
38.28     copying re2.py -> build/lib.linux-aarch64-3.8
38.28     running build_ext
38.28     building '_re2' extension
38.28     creating build/temp.linux-aarch64-3.8
38.28     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/venv/django/include -I/usr/local/include/python3.8 -c _re2.cc -o build/temp.linux-aarch64-3.8/_re2.o -fvisibility=hidden
38.28     _re2.cc:10:10: fatal error: pybind11/pybind11.h: No such file or directory
38.28      #include <pybind11/pybind11.h>
38.28               ^~~~~~~~~~~~~~~~~~~~~
38.28     compilation terminated.
38.28     error: command 'gcc' failed with exit status 1
38.28     ----------------------------------------
38.28 ERROR: Command errored out with exit status 1: /opt/venv/django/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7wn1ub26/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7wn1ub26/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-b4hq8o2f/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/django/include/site/python3.8/google-re2 Check the logs for full command output.
38.43 WARNING: You are using pip version 20.2.3; however, version 24.3.1 is available.
38.43 You should consider upgrading via the '/opt/venv/django/bin/python -m pip install --upgrade pip' command.
------
failed to solve: process "/bin/sh -c python -m venv --copies /opt/venv/django     && /opt/venv/django/bin/python -m pip install --no-deps --no-cache-dir -r /app/venv/django-requirements-frozen.txt     && find /opt/venv/django -name __pycache__ -depth -exec rm -r {} +" did not complete successfully: exit code: 1

If we add a pybind11 dependency to the lockfile, we get a new error from the same line

37.07     Running setup.py install for google-re2: finished with status 'error'
37.07     ERROR: Command errored out with exit status 1:
37.07      command: /opt/venv/django/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d2jyh_x2/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-d2jyh_x2/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9ydqmswb/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/django/include/site/python3.8/google-re2
37.07          cwd: /tmp/pip-install-d2jyh_x2/google-re2/
37.07     Complete output (15 lines):
37.07     running install
37.07     running build
37.07     running build_py
37.07     creating build
37.07     creating build/lib.linux-aarch64-3.8
37.07     copying re2.py -> build/lib.linux-aarch64-3.8
37.07     running build_ext
37.07     building '_re2' extension
37.07     creating build/temp.linux-aarch64-3.8
37.07     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/venv/django/lib/python3.8/site-packages/pybind11/include -I/opt/venv/django/include -I/usr/local/include/python3.8 -c _re2.cc -o build/temp.linux-aarch64-3.8/_re2.o -fvisibility=hidden
37.07     _re2.cc:12:10: fatal error: re2/stringpiece.h: No such file or directory
37.07      #include "re2/stringpiece.h"
37.07               ^~~~~~~~~~~~~~~~~~~
37.07     compilation terminated.
37.07     error: command 'gcc' failed with exit status 1
37.07     ----------------------------------------
37.07 ERROR: Command errored out with exit status 1: /opt/venv/django/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d2jyh_x2/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-d2jyh_x2/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9ydqmswb/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/django/include/site/python3.8/google-re2 Check the logs for full command output.
37.21 WARNING: You are using pip version 20.2.3; however, version 24.3.1 is available.
37.21 You should consider upgrading via the '/opt/venv/django/bin/python -m pip install --upgrade pip' command.
------
failed to solve: process "/bin/sh -c python -m venv --copies /opt/venv/django     && /opt/venv/django/bin/python -m pip install --no-deps --no-cache-dir -r /app/venv/django-requirements-frozen.txt     && find /opt/venv/django -name __pycache__ -depth -exec rm -r {} +" did not complete successfully: exit code: 1

This error seems attributable to the lack of libre2 in the Docker image per the google-re2 wiki. I try both libre2-dev (current latest version) and libre2-5 (a recent version in 2020) to the base image pybase-venv, and get a new error

37.95     Running setup.py install for google-re2: finished with status 'error'
37.95     ERROR: Command errored out with exit status 1:
37.95      command: /opt/venv/django/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wd8myd8r/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wd8myd8r/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0v1jrzs8/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/django/include/site/python3.8/google-re2
37.95          cwd: /tmp/pip-install-wd8myd8r/google-re2/
37.95     Complete output (31 lines):
37.95     running install
37.95     running build
37.95     running build_py
37.95     creating build
37.95     creating build/lib.linux-aarch64-3.8
37.95     copying re2.py -> build/lib.linux-aarch64-3.8
37.95     running build_ext
37.95     building '_re2' extension
37.95     creating build/temp.linux-aarch64-3.8
37.95     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/venv/django/lib/python3.8/site-packages/pybind11/include -I/opt/venv/django/include -I/usr/local/include/python3.8 -c _re2.cc -o build/temp.linux-aarch64-3.8/_re2.o -fvisibility=hidden
37.95     _re2.cc: In function ‘std::vector<int> re2_python::RE2ProgramFanoutShim(const re2::RE2&)’:
37.95     _re2.cc:91:32: error: no matching function for call to ‘re2::RE2::ProgramFanout(std::vector<int>*) const’
37.95        self.ProgramFanout(&histogram);
37.95                                     ^
37.95     In file included from /usr/include/re2/filtered_re2.h:27,
37.95                      from _re2.cc:13:
37.95     /usr/include/re2/re2.h:296:7: note: candidate: ‘int re2::RE2::ProgramFanout(std::map<int, int>*) const’
37.95        int ProgramFanout(std::map<int, int>* histogram) const;
37.95            ^~~~~~~~~~~~~
37.95     /usr/include/re2/re2.h:296:7: note:   no known conversion for argument 1 from ‘std::vector<int>*’ to ‘std::map<int, int>*’
37.95     _re2.cc: In function ‘std::vector<int> re2_python::RE2ReverseProgramFanoutShim(const re2::RE2&)’:
37.95     _re2.cc:97:39: error: no matching function for call to ‘re2::RE2::ReverseProgramFanout(std::vector<int>*) const’
37.95        self.ReverseProgramFanout(&histogram);
37.95                                            ^
37.95     In file included from /usr/include/re2/filtered_re2.h:27,
37.95                      from _re2.cc:13:
37.95     /usr/include/re2/re2.h:297:7: note: candidate: ‘int re2::RE2::ReverseProgramFanout(std::map<int, int>*) const’
37.95        int ReverseProgramFanout(std::map<int, int>* histogram) const;
37.95            ^~~~~~~~~~~~~~~~~~~~
37.95     /usr/include/re2/re2.h:297:7: note:   no known conversion for argument 1 from ‘std::vector<int>*’ to ‘std::map<int, int>*’
37.95     error: command 'gcc' failed with exit status 1
37.95     ----------------------------------------
37.96 ERROR: Command errored out with exit status 1: /opt/venv/django/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wd8myd8r/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wd8myd8r/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0v1jrzs8/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/django/include/site/python3.8/google-re2 Check the logs for full command output.
38.11 WARNING: You are using pip version 20.2.3; however, version 24.3.1 is available.
38.11 You should consider upgrading via the '/opt/venv/django/bin/python -m pip install --upgrade pip' command.
------
failed to solve: process "/bin/sh -c python -m venv --copies /opt/venv/django     && /opt/venv/django/bin/python -m pip install --no-deps --no-cache-dir -r /app/venv/django-requirements-frozen.txt     && find /opt/venv/django -name __pycache__ -depth -exec rm -r {} +" did not complete successfully: exit code: 1

Searching suggests running the pip install with C flags ( https://github.com/google/re2/issues/453#issuecomment-1963194606 ) may fix the issue when installing the library on a Mac ( google/re2#453 (comment) ), but it had no effect for me when I just added those flags to the Dockerfile.

At this point, I tried a new option of commenting out google-re2 in every dependency file, which unlocked a different class of errors (next section).

Installing pandas

I've tried going back to the git hash from when pyarrow was upgraded, to the last hash of 2020 - I haven't been able to get something to work. The error first appears when installing cjwkernel-requirements-frozen.txt .

The error is too verbose to reproduce in full, but I include about 1 page of the output below as a sample.

354.4     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/venv/cjwkernel/lib/python3.8/site-packages/numpy/core/include -I/opt/venv/cjwkernel/include -I/usr/local/include/python3.8 -c pandas/io/sas/sas.c -o build/temp.linux-aarch64-3.8/pandas/io/sas/sas.o -Wno-unused-function
354.4     pandas/io/sas/sas.c: In function ‘__pyx_f_6pandas_2io_3sas_4_sas_rdc_decompress’:
354.4     pandas/io/sas/sas.c:3717:65: warning: ‘__pyx_v_ctrl_bits’ may be used uninitialized in this function [-Wmaybe-uninitialized]
354.4          __pyx_t_8 = (((__pyx_v_ctrl_bits & __pyx_v_ctrl_mask) == 0) != 0);
354.4                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~
354.4     gcc -pthread -shared -Wl,--strip-all build/temp.linux-aarch64-3.8/pandas/io/sas/sas.o -L/usr/local/lib -o build/lib.linux-aarch64-3.8/pandas/io/sas/_sas.cpython-38-aarch64-linux-gnu.so
354.4     building 'pandas.io.msgpack._packer' extension
354.4     creating build/temp.linux-aarch64-3.8/pandas/io/msgpack
354.4     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -D__LITTLE_ENDIAN__=1 -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/msgpack -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/venv/cjwkernel/lib/python3.8/site-packages/numpy/core/include -I/opt/venv/cjwkernel/include -I/usr/local/include/python3.8 -c pandas/io/msgpack/_packer.cpp -o build/temp.linux-aarch64-3.8/pandas/io/msgpack/_packer.o -Wno-unused-function
354.4     g++ -pthread -shared -Wl,--strip-all build/temp.linux-aarch64-3.8/pandas/io/msgpack/_packer.o -L/usr/local/lib -o build/lib.linux-aarch64-3.8/pandas/io/msgpack/_packer.cpython-38-aarch64-linux-gnu.so
354.4     building 'pandas.io.msgpack._unpacker' extension
354.4     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -D__LITTLE_ENDIAN__=1 -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/msgpack -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/venv/cjwkernel/lib/python3.8/site-packages/numpy/core/include -I/opt/venv/cjwkernel/include -I/usr/local/include/python3.8 -c pandas/io/msgpack/_unpacker.cpp -o build/temp.linux-aarch64-3.8/pandas/io/msgpack/_unpacker.o -Wno-unused-function
354.4     g++ -pthread -shared -Wl,--strip-all build/temp.linux-aarch64-3.8/pandas/io/msgpack/_unpacker.o -L/usr/local/lib -o build/lib.linux-aarch64-3.8/pandas/io/msgpack/_unpacker.cpython-38-aarch64-linux-gnu.so
354.4     building 'pandas._libs.json' extension
354.4     creating build/temp.linux-aarch64-3.8/pandas/_libs/src/ujson
354.4     creating build/temp.linux-aarch64-3.8/pandas/_libs/src/ujson/python
354.4     creating build/temp.linux-aarch64-3.8/pandas/_libs/src/ujson/lib
354.4     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/ujson/python -Ipandas/_libs/src/ujson/lib -Ipandas/_libs/src/datetime -I/opt/venv/cjwkernel/lib/python3.8/site-packages/numpy/core/include -I/opt/venv/cjwkernel/include -I/usr/local/include/python3.8 -c pandas/_libs/src/ujson/python/ujson.c -o build/temp.linux-aarch64-3.8/pandas/_libs/src/ujson/python/ujson.o -D_GNU_SOURCE -Wno-unused-function
354.4     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/ujson/python -Ipandas/_libs/src/ujson/lib -Ipandas/_libs/src/datetime -I/opt/venv/cjwkernel/lib/python3.8/site-packages/numpy/core/include -I/opt/venv/cjwkernel/include -I/usr/local/include/python3.8 -c pandas/_libs/src/ujson/python/objToJSON.c -o build/temp.linux-aarch64-3.8/pandas/_libs/src/ujson/python/objToJSON.o -D_GNU_SOURCE -Wno-unused-function
354.4     pandas/_libs/src/ujson/python/objToJSON.c: In function ‘initObjToJSON’:
354.4     pandas/_libs/src/ujson/python/objToJSON.c:181:12: error: ‘NUMPY_IMPORT_ARRAY_RETVAL’ undeclared (first use in this function); did you mean ‘NPY_INOUT_ARRAY’?
354.4          return NUMPY_IMPORT_ARRAY_RETVAL;
354.4                 ^~~~~~~~~~~~~~~~~~~~~~~~~
354.4                 NPY_INOUT_ARRAY
354.4     pandas/_libs/src/ujson/python/objToJSON.c:181:12: note: each undeclared identifier is reported only once for each function it appears in
354.4     pandas/_libs/src/ujson/python/objToJSON.c:182:1: warning: control reaches end of non-void function [-Wreturn-type]
354.4      }
354.4      ^
354.4     error: command 'gcc' failed with exit status 1
354.4     ----------------------------------------
354.4 ERROR: Command errored out with exit status 1: /opt/venv/cjwkernel/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8jjdt6ih/pandas/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8jjdt6ih/pandas/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-mi_0qp98/install-record.txt --single-version-externally-managed --compile --install-headers /opt/venv/cjwkernel/include/site/python3.8/pandas Check the logs for full command output.
354.5 WARNING: You are using pip version 20.2.3; however, version 24.3.1 is available.
354.5 You should consider upgrading via the '/opt/venv/cjwkernel/bin/python -m pip install --upgrade pip' command.
------
failed to solve: process "/bin/sh -c python -m venv --copies /opt/venv/cjwkernel     && /opt/venv/cjwkernel/bin/python -m pip install --no-deps --no-cache-dir -r /app/venv/cjwkernel-requirements-frozen.txt" did not complete successfully: exit code: 1

Some search suggests the ‘NPY_INOUT_ARRAY’ flavor of error happens if pandas and numpy versions are incompatible. I tried downgrading numpy to various versions (0.18.5 was known to work with pandas 0.25.3 but is incompatible with this python 3.8 base image), the various versions in between I tried (0.19.x, 0.21.x) didn't have any better luck

Theories for what might not be working

• Some of these dependencies cannot be built on an M1 Mac laptop without further changes
• There are some dependencies that are not accounted for in the requirements.txt frozen files, such as changes to the base docker images and/or linux packages
• Something is configured differently in my system (Env variables, cache, etc) that is quietly interfering with the build

I'm ok to leave the docker image as-is if there might be a way to run the frontend app inside a local web server outside of Docker. Wanted to post mainly to see if other Mac users are running into the same issue, or if it is just me.

[hydrosquall]

It looks like this is an issue specific to M1 (arm64) users - I was able to get the site to build seamlessly on an x86-64 computer.

I suspect what is happening is the current docker images reference prebuilt binaries for the libraries that are throwing errors. The other machine isn't running those "build from source" steps, it's probably bypassing them.

To address this issue, each of those dependencies would either need to also offer prebuilt arm64 binaries, or we'd need to make changes so that the new binaries from source were able to build.