More documentation of download / upload procedure

Author: matthew-brett

README.rst

@@ -25,16 +25,15 @@ The wheel-building repository:
   (Manylinux1_).  ``delocate`` and ``auditwheel`` copy the required dynamic
   libraries into the wheel and relinks the extension modules against the
   copied libraries;
-* uploads the built wheels to http://wheels.scipy.org (a Rackspace container
-  kindly donated by Rackspace to scikit-learn).
+* uploads the built wheels to http://anaconda.org/nipy/nipy
 
 The resulting wheels are therefore self-contained and do not need any external
 dynamic libraries apart from those provided as standard by OSX / Linux as
 defined by the manylinux1 standard.
 
 The ``.travis.yml`` file in this repository has a line containing the API key
-for the Rackspace container encrypted with an RSA key that is unique to the
-repository - see http://docs.travis-ci.com/user/encryption-keys.  This
+for the Anaconda.org organization encrypted with an RSA key that is unique to
+the repository - see http://docs.travis-ci.com/user/encryption-keys.  This
 encrypted key gives the travis build permission to upload to the Rackspace
 directory pointed to by http://wheels.scipy.org.
 
@@ -70,67 +69,32 @@ hash.
 Uploading the built wheels to pypi
 ==================================
 
-Be careful, http://wheels.scipy.org points to a container on a distributed
-content delivery network.  It can take up to 15 minutes for the new wheel file
-to get updated into the container at http://wheels.scipy.org.
-
-The same contents appear at
-https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com;
-you might prefer this address because it is https.
-
 When the wheels are updated, you can download them to your machine manually,
-and then upload them manually to pypi, or by using twine_.  You can also use a
-script for doing this, housed at :
-https://github.com/MacPython/terryfy/blob/master/wheel-uploader
+and then upload them manually to pypi, or by using twine_.
 
-For the ``wheel-uploader`` script, you'll need twine and `beautiful soup 4
-<bs4>`_.
+To download, use something like::
 
-You will typically have a directory on your machine where you store wheels,
-called a `wheelhouse`.   The typical call for `wheel-uploader` would then
-be something like::
+    python tools/download-wheels.py 0.5.0 --staging-url=https://anaconda.org/nipy/nipy --prefix=nipy -w wheelhouse
 
-    VERSION=0.4.0
-    CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com
-    wheel-uploader -r warehouse -u $CDN_URL -s -v -w ~/wheelhouse -t macosx nipy $VERSION
-    wheel-uploader -r warehouse -u $CDN_URL -s -v -w ~/wheelhouse -t manylinux1 nipy $VERSION
-    wheel-uploader -r warehouse -u $CDN_URL -s -v -w ~/wheelhouse -t win nipy $VERSION
+where `0.5.0` is the release version.
 
-where:
+You may want to add the `sdist` to the `wheelhouse`.
 
-* ``-r warehouse`` uses the upcoming Warehouse PyPI server (it is more
-  reliable than the current PyPI service for uploads);
-* ``-u`` gives the URL from which to fetch the wheels, here the https address,
-  for some extra security;
-* ``-s`` causes twine to sign the wheels with your GPG key;
-* ``-v`` means give verbose messages;
-* ``-w ~/wheelhouse`` means download the wheels from to the local directory
-  ``~/wheelhouse``.
+Then::
 
-``nipy`` is the root name of the wheel(s) to download / upload, and ``0.4.0``
-is the version to download / upload.
+    twine upload --sign wheelhouse/*
 
-In order to use the Warehouse PyPI server, you will need something like this
-in your ``~/.pypirc`` file::
+In order to use Twine, you will need something like this in your ``~/.pypirc``
+file::
 
     [distutils]
     index-servers =
         pypi
-        warehouse
 
     [pypi]
     username:your_user_name
     password:your_password
 
-    [warehouse]
-    repository: https://upload.pypi.io/legacy/
-    username: your_user_name
-    password: your_password
-
-So, in this case, ``wheel-uploader`` will download all wheels starting with
-``nipy-0.4.0-`` from http://wheels.scipy.org to ``~/wheelhouse``, then upload
-them to PyPI.
-
 Of course, you will need permissions to upload to PyPI, for this to work.
 
 .. _manylinux1: https://www.python.org/dev/peps/pep-0513

nipy

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""
+Download wheels from Anaconda staging area.
+"""
+
+import os
+import re
+import shutil
+import argparse
+
+import urllib3
+from bs4 import BeautifulSoup
+
+__version__ = '0.2'
+
+
+def get_wheel_names(version, staging_url, prefix):
+    """ Get wheel names from Anaconda HTML directory.
+
+    This looks in the Anaconda multibuild-wheels-staging page and
+    parses the HTML to get all the wheel names for a release version.
+
+    Parameters
+    ----------
+    version : str
+        The release version. For instance, "1.5.0".
+    staging_url : str
+        URL at which to find packages
+    prefix : str
+        Prefix for wheels to download.
+    """
+    http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED')
+    tmpl = re.compile(rf"^.*{prefix}-{version}-.*\.whl$")
+    index_url = f"{staging_url}/files"
+    index_html = http.request('GET', index_url)
+    soup = BeautifulSoup(index_html.data, 'html.parser')
+    return soup.findAll(text=tmpl)
+
+
+def download_wheels(version, wheelhouse, staging_url, prefix):
+    """Download release wheels.
+
+    The release wheels for the given package version are downloaded
+    into the given directory.
+
+    Parameters
+    ----------
+    version : str
+        The release version. For instance, "1.5.0".
+    wheelhouse : str
+        Directory in which to download the wheels.
+    staging_url : str
+        URL at which to find packages
+    prefix : str
+        Prefix for wheels to download.
+    """
+    http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED')
+    wheel_names = get_wheel_names(version, staging_url, prefix)
+
+    for i, wheel_name in enumerate(wheel_names):
+        wheel_url = f"{staging_url}/{version}/download/{wheel_name}"
+        wheel_path = os.path.join(wheelhouse, wheel_name)
+        with open(wheel_path, 'wb') as f:
+            with http.request('GET', wheel_url, preload_content=False,) as r:
+                print(f"{i + 1:<4}{wheel_name}")
+                shutil.copyfileobj(r, f)
+    print(f"\nTotal files downloaded: {len(wheel_names)}")
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "version",
+        help="Package version to download.")
+    parser.add_argument(
+        "--staging-url",
+        default='https://anaconda.org/multibuild-wheels-staging/scipy',
+        help="URL at which to find packages")
+    parser.add_argument(
+        "--prefix",
+        default='scipy',
+        help="Prefix for wheels (e.g 'scipy'")
+    parser.add_argument(
+        "-w", "--wheelhouse",
+        default=os.path.join(os.getcwd(), "release", "installers"),
+        help="Directory in which to store downloaded wheels\n"
+             "[defaults to <cwd>/release/installers]")
+
+    args = parser.parse_args()
+
+    wheelhouse = os.path.expanduser(args.wheelhouse)
+    if not os.path.isdir(wheelhouse):
+        raise RuntimeError(
+            f"{wheelhouse} wheelhouse directory is not present."
+            " Perhaps you need to use the '-w' flag to specify one.")
+
+    download_wheels(args.version, wheelhouse, args.staging_url, args.prefix)

tools/download-wheels.py

@@ -0,0 +1 @@
+*