manylinux docker image producing broken/misnamed wheels?

[jamadden]

Possibly related to #29 (?), gevent got a report in gevent/gevent#789 about bad unicode symbols from the wheel downloaded from PyPI. The 1.1.0 wheels worked, but the 1.1.1 wheels, built on an image that supported both ABIs, didn't. I'm not enough of an ABI expert to know exactly what's going on at first glance.

Since this wheel was just built and uploaded with the default manylinux docker image, I'm hoping somebody here might have an idea. Thanks!

CC @woozyking

[jamadden]

Argh. This could be entirely my fault. It's possible that I built both the 'm' and 'mu' wheels in the same directory without deleting the build directory first (gevent has some build steps that take a long time so in the past I've not cleaned out the build directory between builds). Are architecture tags like that taken into account in the temp-directory names underneath build, or might existing .so files silently (and wrongly) get re-used?

[rmcgibbo]

To some degree, this is a bug in auditwheel, or at least a feature request that's on my todolist.

For pythons (py2) that have the narrow/wide distinction, auditwheel can and should check whether the wheel contains the appropriate narrow/wide symbols based on its declared arch tag, and error out if they don't match.

[matthew-brett]

It would be very good to apply that, otherwise it will be fairly easy to upload the wrong wheels and give manylinux a bad rep.

[jamadden]

So probably sequential builds in the same directory simply resulted in a bad wheel, and nothing in the toolchain detected that until runtime?

I guess I should go remove that wheel from PyPi then.

[ogrisel]

I guess I should go remove that wheel from PyPi then.

Unfortunately you will have to re-upload with a new (higher) version number as updating an existing file (same filename but different contents) is not possible (deletion is possible).

[rmcgibbo]

So probably sequential builds in the same directory simply resulted in a bad wheel, and nothing in the toolchain detected that until runtime?

Well, this is a plausible explanation, but I'm not sure. If you try re-doing the build while making sure to clean out the environment in between builds, and the problem does in fact go away, then it becomes pretty certain that this was the root cause.

[ogrisel]

I think this is an important bug to fix and we should start doing official (versioned or timestamped) releases of the manylinux docker images along with a changelog to document when those bugs are fixed.

[jamadden]

I just confirmed that if I do /opt/2.7m/bin/python ./setup.py bdist_wheel followed by /opt/2.7mu/bin/python ./setup.py bdist_wheel, the same intermediate build directories are re-used (they don't contain a wideness designation, just arch: lib.linux-x86_64-2.7/) and an invalid mu wheel is produced (that the current version of auditwheel doesn't notice).

(I likely didn't notice the build being so quick the second time around because I did all the mu versions before starting on the m version. The broken wheel also installs and works just fine until a particular extension is used because the main extension has an automatic fallback.)

[njsmith]

Note that wheels do have a rarely used "build tag" field, if you want to avoid bumping your overall version number: https://www.python.org/dev/peps/pep-0427/#file-name-convention

Of course, when things are going wrong may not be the best moment to experiment with new features...

@ogrisel: I agree that starting to keep a change log might be a good idea. OTOH I'm not sure how version numbers work with docker, esp. given our auto deploy stuff, because I am ignorant of most things docker :-). Is there a simple explanation?

[ogrisel]

We can push tags to the quay repo. The default tag is 'latest' and is updated by default each time you push a new image with default tag. Users can pull a specific tag or pull latest if no specific tag is requested.