Allow firewalls for Pants PEXs (#261)

This change extends the `PANTS_BOOTSTRAP_URLS` functionality to the
fetched Pants PEX.

```console
josh@cephandrius:~/work/scie-pants$ PANTS_BOOTSTRAP_URLS=urls.json  PANTS_VERSION=2.22.0 dist/scie-pants-linux-x86_64
Bootstrapping Pants 2.22.0 using cpython 3.9.16
Installing pantsbuild.pants==2.22.0 into a virtual environment at /home/josh/.cache/nce/67dd6374fe21b02fa9e1be3a775826ef7fcac9791c6052276d2b3972a363d49f/bindings/venvs/2.22.0
Failed to fetch https://poop.com: [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)
Wasn't able to fetch the Pants PEX at https://poop.com.
...
```

README.md

@@ -62,34 +62,46 @@ provides the following:
 
 + Partial support for firewalls:
 
-  Currently, you can only re-direct the URLs scie-pants uses to fetch [Python Build Standalone](
-  https://python-build-standalone.readthedocs.io/en/latest/) CPython distributions it uses to
-  bootstrap Pants with. This is done by exporting a `PANTS_BOOTSTRAP_URLS` environment variable
+  Currently, you can re-direct the URLs used to fetch:
+
+    + [Python Build Standalone](https://python-build-standalone.readthedocs.io/en/latest/) CPython
+      distributions used to bootstrap Pants.
+    + Pants PEX release assets which contain Pants as a single-file application.
+
+  This is done by exporting a `PANTS_BOOTSTRAP_URLS` environment variable
   specifying the path to a JSON file containing a mapping of file names to URLS to fetch them from
   under a top-level `"ptex"` key. For example:
   ```json
   {
     "ptex": {
-      "cpython-3.8.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.8.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz",
-      "cpython-3.8.16+20230507-aarch64-apple-darwin-install_only.tar.gz": "https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.8.16%2B20230507-aarch64-apple-darwin-install_only.tar.gz",
-      "cpython-3.9.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.9.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz",
-      "cpython-3.9.16+20230507-aarch64-apple-darwin-install_only.tar.gz": "https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.9.16%2B20230507-aarch64-apple-darwin-install_only.tar.gz",
+      "cpython-3.8.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "https://example.com/cpython-3.8.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz",
+      "cpython-3.8.16+20230507-aarch64-apple-darwin-install_only.tar.gz": "https://example.com/cpython-3.8.16%2B20230507-aarch64-apple-darwin-install_only.tar.gz",
+      "cpython-3.9.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "https://example.com/cpython-3.9.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz",
+      "cpython-3.9.16+20230507-aarch64-apple-darwin-install_only.tar.gz": "https://example.com/cpython-3.9.16%2B20230507-aarch64-apple-darwin-install_only.tar.gz",
+      "pants.2.18.0-cp9-linux-x86_64.pex": "https://example.com/pants.2.18.0-cp9-linux-x86_64.pex",
+      ...
     }
   }
   ```
-  To see the current mapping used by your version of `scie-pants` you can run:
+
+  For keys that are "embedded" into `scie-pants` itself (such as Python Build Standalone), you can run:
   ```
   $ SCIE=inspect scie-pants | jq .ptex
   ```
   You'll need to run this once for each platform you use `scie-pants` on to gather all mappings
   you'll need; e.g.: once for Linux x86_64 and once for Mac ARM.
 
-  The keys in your re-mapping must match, but the URLs, of course will be different; presumably from
-  a private network server or file share. The full output of the inspect command can be used to 
-  examine the expected file size and hash of each of these distributions. Your re-directed URLs must
-  provide the same content; if the hashes of downloaded files do not match those recorded in
-  scie-pants, install will fail fast and let you know about the hash mismatch. Once Pants itself
-  starts shipping scies, those will also be able to redirected using the same file.
+  The embedded artifact references also contain expected hashes of the downloaded content. Your
+  re-directed URLs must provide the same content as the canonical URLs; if the hashes of downloaded
+  files do not match those recorded in `scie-pants`, install will fail fast and let you know about
+  the hash mismatch.
+
+  For other keys that aren't embedded, and are generated on-the-fly (such as the Pants PEX), there
+  is no single source of truth that can be easily scraped out. For the Pants PEX, the key is the versioned
+  PEX name (E.g. `pants.<version>-<python>-<plat>-<machine>.pex`). These can be found on the relevant
+  GitHub Release page's Assets (e.g. https://github.com/pantsbuild/pants/releases/tag/release_2.18.0a0).
+  (Note that for 2.18.x, PEX exist versioned and unversioned. `scie-pants` only uses the versioned
+  name as the key).
 
 ## Caveats
 

package/scie-pants.toml

@@ -218,6 +218,8 @@ args = [
     "{scie.env.PANTS_DEBUG}",
     "--debugpy-requirement",
     "{scie.env.PANTS_DEBUGPY_VERSION}",
+    "--pants-bootstrap-urls",
+    "{scie.env.PANTS_BOOTSTRAP_URLS}",
     "--ptex-path",
     "{ptex}",
     "{scie.bindings}",

tools/src/scie_pants/install_pants.py

@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import json
 import logging
 import os
 import stat
@@ -78,22 +79,36 @@ def install_pants_from_pex(
     ptex: Ptex,
     extra_requirements: Iterable[str],
     find_links: str | None,
+    bootstrap_urls_path: str | None,
 ) -> None:
     """Installs Pants into the venv using the platform-specific pre-built PEX."""
     uname = os.uname()
-    major, minor = sys.version_info[:2]
-    pex_name = (
-        f"pants.{version}-cp{major}{minor}-{uname.sysname.lower()}_{uname.machine.lower()}.pex"
-    )
+    pex_name = f"pants.{version}-cp39-{uname.sysname.lower()}_{uname.machine.lower()}.pex"
+
     pex_url = f"https://github.com/pantsbuild/pants/releases/download/release_{version}/{pex_name}"
+    if bootstrap_urls_path:
+        bootstrap_urls = json.loads(Path(bootstrap_urls_path).read_text())
+        urls_info = bootstrap_urls["ptex"]
+        pex_url = urls_info.get(pex_name)
+        if pex_url is None:
+            raise ValueError(
+                f"Couldn't find '{pex_name}' in PANTS_BOOTSTRAP_URLS file: '{bootstrap_urls_path}' "
+                "under the 'ptex' key."
+            )
+        if not isinstance(pex_url, str):
+            raise TypeError(
+                f"The value for the key '{pex_name}' in PANTS_BOOTSTRAP_URLS file: '{bootstrap_urls_path}' "
+                f"under the 'ptex' key was expected to be a string. Got a {type(pex_url).__name__}"
+            )
+
     with tempfile.NamedTemporaryFile(suffix=".pex") as pants_pex:
         try:
             ptex.fetch_to_fp(pex_url, pants_pex.file)
         except subprocess.CalledProcessError as e:
             fatal(
                 f"Wasn't able to fetch the Pants PEX at {pex_url}.\n\n"
                 "Check to see if the URL is reachable (i.e. GitHub isn't down) and if"
-                f" {pex_name} asset exists within the release: https://github.com/pantsbuild/pants/releases/release_{version}."
+                f" {pex_name} asset exists within the release."
                 " If the asset doesn't exist it may be that this platform isn't yet supported."
                 " If that's the case, please reach out on Slack: https://www.pantsbuild.org/docs/getting-help#slack"
                 " or file an issue on GitHub: https://github.com/pantsbuild/pants/issues/new/choose.\n\n"
@@ -138,6 +153,11 @@ def main() -> NoReturn:
         type=str,
         help="The find links repo pointing to Pants pre-built wheels for the given Pants version",
     )
+    parser.add_argument(
+        "--pants-bootstrap-urls",
+        type=str,
+        help="The path to the JSON file containing alternate URLs for downloaded artifacts.",
+    )
     parser.add_argument("--debug", type=bool, help="Install with debug capabilities.")
     parser.add_argument("--debugpy-requirement", help="The debugpy requirement to install")
     parser.add_argument("base_dir", nargs=1, help="The base directory to create Pants venvs in.")
@@ -180,6 +200,7 @@ def main() -> NoReturn:
             ptex=ptex,
             extra_requirements=extra_requirements,
             find_links=options.find_links,
+            bootstrap_urls_path=options.pants_bootstrap_urls,
         )
     else:
         install_pants_from_req(