shared library paths

[dazza-codes]

Tracking some issues on other projects here:

• Support pip binary wheel manylinux installations OSGeo/gdal#3060
• https://github.com/matthew-brett/multibuild/issues/365

More refs:

• https://docs.python.org/3/distutils/setupscript.html#library-options
• https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh
• https://github.com/pypa/wheel-builders/blob/f9da350f65be9bb7006cfa3c7388e19d1727a362/pynativelib-proposal.rst
• Campaign to get people Publishing Wheels pypa/packaging-problems#25 (comment)
  • auditwheel rewrites vendored DSOs to give them unique SONAMEs on Linux
• https://pypi.org/project/auditwheel/
• https://github.com/conda-incubator/conda-press

For https://github.com/dazza-codes/aws-lambda-layer-packing/blob/master/layer_create_zip.sh#L108 - this project could hack something to "undo" auditwheel and use the AWS Lambda LD_LIBRARY_PATH.

Is there a way to use conda packaging for a conda-env without too much conda cruft in a way that a .zip package would work OK without conda in the runtime? Try - https://github.com/conda-incubator/conda-press in a build container and extract a .zip archive for a lambda layer?

What are the origins and differences, if any, between lambda docker images and manylinux_2014 (CentOS 6, UBI) images? The short story is that amazonlinux is based on an Ubuntu/Debian OS with apt packaging whereas manylinux* are based on CentOS with yum packaging, so it's tricky to trace the core library versions used and any issues of binary compatibility.

• https://github.com/lambci/docker-lambda/blob/master/base/Dockerfile
  • FROM amazonlinux:1
  • LD_LIBRARY_PATH has /var/task/lib:/opt/lib among others before that
• amazonlinux heritage is hard to trace:
  • https://github.com/docker-library/official-images/blob/master/Dockerfile
  • https://github.com/docker-library/bashbrew/blob/master/Dockerfile
  • https://hub.docker.com/r/tianon/docker-tianon/dockerfile
    • FROM debian:stretch-slim
• also review https://github.com/aws/aws-lambda-builders

The python manylinux wheels are based on CentOS (maybe RH UBI sometime)

• https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
• https://developers.redhat.com/products/rhel/ubi
• https://discuss.python.org/t/using-ubi7-images-instead-on-centos7-for-future-manylinux-images/1709

Less than a full operating system, UBI is three things:
- A set of three base images (ubi, ubi-minimal, ubi-init)
- A set of language runtime images (nodejs, ruby, python, php, perl, etc.)
- A set of associated packages in a YUM repository which satisfy common application dependencies

podman pull registry.access.redhat.com/ubi8/ubi
podman pull registry.access.redhat.com/ubi8/ubi-minimal
podman pull registry.access.redhat.com/ubi8/ubi-init

AWS Lambda ENV

AWS_EXECUTION_ENV=AWS_Lambda_python3.7

PWD=/var/task
LAMBDA_TASK_ROOT=/var/task
LAMBDA_RUNTIME_DIR=/var/runtime

PATH=/var/lang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib
PKG_CONFIG_PATH=/var/lang/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig