Installing awkward-numba in usermode breaks awkward

[bloer]

If I have awkward installed in my regular site packages, and then try to install awkard-numba in user mode, I get

In [1]: import awkward
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-77000f7762b3> in <module>()
----> 1 import awkward

~/.local/lib/python3.6/site-packages/awkward/__init__.py in <module>()
      9     raise ImportError("Numpy 1.13.1 or later required")
     10 
---> 11 from awkward.array.base import AwkwardArray
     12 from awkward.array.chunked import ChunkedArray, AppendableArray
     13 from awkward.array.indexed import IndexedArray, SparseArray

ModuleNotFoundError: No module named 'awkward.array'

The awkward directory under system site-packages looks like

$ ls /usr/local/lib/python3.6/site-packages/awkward
array     generate.py  pandas.py   __pycache__  util.py
arrow.py  __init__.py  persist.py  type.py      version.py

while in my user site I have

$ ls $HOME/.local/lib/python3.6/site-packages/awkward/
__init__.py  numba  __pycache__

Pip detects that the dependency on awkward is already satisfied so doesn't install into user site, but then awkward-numba goes and installs a partial and broken version.

[jpivarski]

Uninstall awkward-numba using pip's "remove" command. You might need to do it several times if there are several versions installed. There needs to be some warning here. This project was never completed, and the difficulties it revealed led to a redesign.

In scikit-hep/awkward-1.0, Numba support is fully integrated and complete. There's a "from_awkward0" function for converting from old-style to new-style arrays. If you're interested in using Numba with Awkward Array, I highly recommend that.

The reference documentation is complete, but the tutorials aren't yet (that's the only part of the new library that isn't done: the tutorials). To get started, look in the docs-demo-notebooks for a demo with tomorrow's date. If you're interested in attending, I can get you connection information (Coffea Developer's meeting).

[bloer]

Can awkward and awkward1 coexist? We're using a few packages that have awkward as dependencies

[jpivarski]

Yes, they can coexist. There's a transition plan. For the next few months (time period to be determined),

• the pip/import name "awkward" corresponds to the original library
• the pip/import name "awkward1" corresponds to the new library

so as far as both PyPI and Python are concerned, they are two independent libraries.

After the switch,

• the pip/import name "awkward0" will correspond to the original library
• the pip/import name "awkward" will correspond to the new library

so at that time, a backward incompatibility will be introduced, but there's a user action that can revert to the old library if needed (import awkward0 as awkward in the relevant namespace).

The same thing is going to happen to Uproot as it transitions from the current Uproot version 3 (which uses Awkward 0) and the future Uproot version 4 (which will use Awkward 1). The transition for both Awkward and Uproot will happen at the same time, and the Uproot you're using already has an explicit pip requirement that its Awkward must be < version 1.0. (I put that in early last year, knowing that this would be coming up.)

Arrays can be converted between Awkward 0 and Awkward 1 using Awkward 1's ak.from_awkward0 and ak.to_awkward0 functions. The function call must be explicit so that you know which is which, and there's no significant (only an O(1)) performance cost to switching between formats.