Hash validation with python dependencies from private git repo failing #21145

jake-normal · 2024-07-08T21:05:55Z

jake-normal
Jul 8, 2024

I'm running into an issue where hash validation of python dependencies is failing with a message similar to:

stderr:
There was 1 error downloading required artifacts:
1. unstructured 0.14.10.dev12 from git+https://****@github.com/johnathanchiu/unstructured@6a70fc4caab489d88ad1613af9f4a53581d8ffb3
    Expected sha256 hash of 0ceabe966af2f0d63bf4a893c0a510306a460b8037e671e56198ea07483aa638 when downloading unstructured but hashed to d8a7a4be6417b577279adc868641530ed1771790e95f1b610425a55c6ea38114

It looks like another user has run into a similar issue here.

I've done my best to create a minimal reproduction of the issue by copying the __run.sh script out of the sandbox and trimming it down a bit.

Every time I run the is script, the hash of unstructured changes:

#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
env \
  -i \
  CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= \
  PATH=$'/opt/homebrew/lib/ruby/gems/3.3.0/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin' \
  PEX_IGNORE_RCFILES=true \
  PEX_PYTHON=/Users/jake/Library/Caches/nce/d52b8d4e72c7ef97d778057e19dcd2dcb2c048c9ac07385c5ea9cc2954831d4b/bindings/venvs/2.21.0/bin/python3.9 \
  PEX_ROOT=.cache/pex_root \
  PEX_SCRIPT=pex3 \
  /Users/jake/Library/Caches/nce/d52b8d4e72c7ef97d778057e19dcd2dcb2c048c9ac07385c5ea9cc2954831d4b/bindings/venvs/2.21.0/bin/python3.9 \
  ./pex lock create --tmpdir .tmp --no-emit-warnings \
  --python-path $'/opt/homebrew/lib/ruby/gems/3.3.0/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin' \
  $'--output=lock.json' \
  $'--style=universal' \
  --pip-version \
  24.0 \
  --resolver-version pip-2020-resolver \
  --target-system linux \
  --target-system mac \
  $'--indent=2' \
  --no-pypi \
  $'--index=https://pypi.org/simple/' \
  --manylinux manylinux2014 \
  --interpreter-constraint $'CPython==3.12.*' $'unstructured[pdf]@ git+ssh://[email protected]/jake-normal/unstructured@e1a652ad06f8fc1d819ceb2ace73bf0ee285cf9e'

Find the hash with:

cat lock.json | jq '.locked_resolves | map(.locked_requirements) | first | map(select(.project_name == "unstructured")) | first | .artifacts | first | .hash'

Any idea what might be going on?

(Note: I've been testing this with a coworker on two forks, johnathanchiu and jake-normal, which is the reason for the .)

Answered by jsirois

Jul 9, 2024

This one is pretty easy by inspection of that project's setup.py.

In short, if a project's build is not reproducible, Pex can't help you - you must fix non-reproducible builds out of band and point Pex at those. If you can't for some reason, you must pre-build wheels and point Pants at an --index or --find-links repo with those fixed wheels.

That said, here you go - let me know if you need more help, but this should be enough for you to solve the problem:

# Get the repo:
git clone https://github.com/johnathanchiu/unstructured
cd unstructured
git reset --hard 6a70fc4caab489d88ad1613af9f4a53581d8ffb3

# Setup a build env:
python -mvenv /tmp/sdist-build.venv
/tmp/sdist-build.venv/bin/pip -q …

View full answer

jsirois · 2024-07-09T03:24:57Z

jsirois
Jul 9, 2024

This one is pretty easy by inspection of that project's setup.py.

In short, if a project's build is not reproducible, Pex can't help you - you must fix non-reproducible builds out of band and point Pex at those. If you can't for some reason, you must pre-build wheels and point Pants at an --index or --find-links repo with those fixed wheels.

That said, here you go - let me know if you need more help, but this should be enough for you to solve the problem:

# Get the repo:
git clone https://github.com/johnathanchiu/unstructured
cd unstructured
git reset --hard 6a70fc4caab489d88ad1613af9f4a53581d8ffb3

# Setup a build env:
python -mvenv /tmp/sdist-build.venv
/tmp/sdist-build.venv/bin/pip -q install -U setuptools

# Build the sdist 2x:
/tmp/sdist-build.venv/bin/python setup.py sdist --dist-dir /tmp/round1
git clean -fdx
/tmp/sdist-build.venv/bin/python setup.py sdist --dist-dir /tmp/round2

# Compare:
tar -xzf /tmp/round1/unstructured-0.14.10.dev12.tar.gz -C /tmp/round1/
tar -xzf /tmp/round2/unstructured-0.14.10.dev12.tar.gz -C /tmp/round2/
diff -ru /tmp/round1/unstructured-0.14.10.dev12 /tmp/round2/unstructured-0.14.10.dev12

And here's that diff:

diff -ru /tmp/round1/unstructured-0.14.10.dev12 /tmp/round2/unstructured-0.14.10.dev12
diff -ru /tmp/round1/unstructured-0.14.10.dev12/PKG-INFO /tmp/round2/unstructured-0.14.10.dev12/PKG-INFO
--- /tmp/round1/unstructured-0.14.10.dev12/PKG-INFO     2024-07-08 20:18:13.733394600 -0700
+++ /tmp/round2/unstructured-0.14.10.dev12/PKG-INFO     2024-07-08 20:18:22.773394300 -0700
@@ -44,24 +44,24 @@
 Requires-Dist: psutil
 Provides-Extra: all-docs
 Requires-Dist: markdown; extra == "all-docs"
-Requires-Dist: pypdf; extra == "all-docs"
-Requires-Dist: python-docx>=1.1.2; extra == "all-docs"
-Requires-Dist: openpyxl; extra == "all-docs"
 Requires-Dist: pdfminer.six; extra == "all-docs"
-Requires-Dist: google-cloud-vision; extra == "all-docs"
+Requires-Dist: effdet; extra == "all-docs"
 Requires-Dist: onnx; extra == "all-docs"
+Requires-Dist: networkx; extra == "all-docs"
 Requires-Dist: pytesseract; extra == "all-docs"
+Requires-Dist: pandas; extra == "all-docs"
+Requires-Dist: pypdf; extra == "all-docs"
 Requires-Dist: xlrd; extra == "all-docs"
-Requires-Dist: python-oxmsg; extra == "all-docs"
-Requires-Dist: unstructured-inference==0.7.36; extra == "all-docs"
-Requires-Dist: networkx; extra == "all-docs"
-Requires-Dist: effdet; extra == "all-docs"
 Requires-Dist: python-pptx<=0.6.23; extra == "all-docs"
-Requires-Dist: pdf2image; extra == "all-docs"
-Requires-Dist: pandas; extra == "all-docs"
+Requires-Dist: pillow_heif; extra == "all-docs"
+Requires-Dist: google-cloud-vision; extra == "all-docs"
+Requires-Dist: python-docx>=1.1.2; extra == "all-docs"
 Requires-Dist: pikepdf; extra == "all-docs"
+Requires-Dist: unstructured-inference==0.7.36; extra == "all-docs"
 Requires-Dist: pypandoc; extra == "all-docs"
-Requires-Dist: pillow_heif; extra == "all-docs"
+Requires-Dist: python-oxmsg; extra == "all-docs"
+Requires-Dist: openpyxl; extra == "all-docs"
+Requires-Dist: pdf2image; extra == "all-docs"
 Provides-Extra: csv
 Requires-Dist: pandas; extra == "csv"
 Provides-Extra: doc
@@ -212,24 +212,24 @@
 Requires-Dist: transformers; extra == "huggingface"
 Provides-Extra: local-inference
 Requires-Dist: markdown; extra == "local-inference"
-Requires-Dist: pypdf; extra == "local-inference"
-Requires-Dist: python-docx>=1.1.2; extra == "local-inference"
-Requires-Dist: openpyxl; extra == "local-inference"
 Requires-Dist: pdfminer.six; extra == "local-inference"
-Requires-Dist: google-cloud-vision; extra == "local-inference"
+Requires-Dist: effdet; extra == "local-inference"
 Requires-Dist: onnx; extra == "local-inference"
+Requires-Dist: networkx; extra == "local-inference"
 Requires-Dist: pytesseract; extra == "local-inference"
+Requires-Dist: pandas; extra == "local-inference"
+Requires-Dist: pypdf; extra == "local-inference"
 Requires-Dist: xlrd; extra == "local-inference"
-Requires-Dist: python-oxmsg; extra == "local-inference"
-Requires-Dist: unstructured-inference==0.7.36; extra == "local-inference"
-Requires-Dist: networkx; extra == "local-inference"
-Requires-Dist: effdet; extra == "local-inference"
 Requires-Dist: python-pptx<=0.6.23; extra == "local-inference"
-Requires-Dist: pdf2image; extra == "local-inference"
-Requires-Dist: pandas; extra == "local-inference"
+Requires-Dist: pillow_heif; extra == "local-inference"
+Requires-Dist: google-cloud-vision; extra == "local-inference"
+Requires-Dist: python-docx>=1.1.2; extra == "local-inference"
 Requires-Dist: pikepdf; extra == "local-inference"
+Requires-Dist: unstructured-inference==0.7.36; extra == "local-inference"
 Requires-Dist: pypandoc; extra == "local-inference"
-Requires-Dist: pillow_heif; extra == "local-inference"
+Requires-Dist: python-oxmsg; extra == "local-inference"
+Requires-Dist: openpyxl; extra == "local-inference"
+Requires-Dist: pdf2image; extra == "local-inference"
 Provides-Extra: paddleocr
 Requires-Dist: unstructured.paddleocr==2.6.1.3; extra == "paddleocr"
 Provides-Extra: embed-huggingface
diff -ru /tmp/round1/unstructured-0.14.10.dev12/unstructured.egg-info/PKG-INFO /tmp/round2/unstructured-0.14.10.dev12/unstructured.egg-info/PKG-INFO
--- /tmp/round1/unstructured-0.14.10.dev12/unstructured.egg-info/PKG-INFO       2024-07-08 20:18:13.000000000 -0700
+++ /tmp/round2/unstructured-0.14.10.dev12/unstructured.egg-info/PKG-INFO       2024-07-08 20:18:22.000000000 -0700
@@ -44,24 +44,24 @@
 Requires-Dist: psutil
 Provides-Extra: all-docs
 Requires-Dist: markdown; extra == "all-docs"
-Requires-Dist: pypdf; extra == "all-docs"
-Requires-Dist: python-docx>=1.1.2; extra == "all-docs"
-Requires-Dist: openpyxl; extra == "all-docs"
 Requires-Dist: pdfminer.six; extra == "all-docs"
-Requires-Dist: google-cloud-vision; extra == "all-docs"
+Requires-Dist: effdet; extra == "all-docs"
 Requires-Dist: onnx; extra == "all-docs"
+Requires-Dist: networkx; extra == "all-docs"
 Requires-Dist: pytesseract; extra == "all-docs"
+Requires-Dist: pandas; extra == "all-docs"
+Requires-Dist: pypdf; extra == "all-docs"
 Requires-Dist: xlrd; extra == "all-docs"
-Requires-Dist: python-oxmsg; extra == "all-docs"
-Requires-Dist: unstructured-inference==0.7.36; extra == "all-docs"
-Requires-Dist: networkx; extra == "all-docs"
-Requires-Dist: effdet; extra == "all-docs"
 Requires-Dist: python-pptx<=0.6.23; extra == "all-docs"
-Requires-Dist: pdf2image; extra == "all-docs"
-Requires-Dist: pandas; extra == "all-docs"
+Requires-Dist: pillow_heif; extra == "all-docs"
+Requires-Dist: google-cloud-vision; extra == "all-docs"
+Requires-Dist: python-docx>=1.1.2; extra == "all-docs"
 Requires-Dist: pikepdf; extra == "all-docs"
+Requires-Dist: unstructured-inference==0.7.36; extra == "all-docs"
 Requires-Dist: pypandoc; extra == "all-docs"
-Requires-Dist: pillow_heif; extra == "all-docs"
+Requires-Dist: python-oxmsg; extra == "all-docs"
+Requires-Dist: openpyxl; extra == "all-docs"
+Requires-Dist: pdf2image; extra == "all-docs"
 Provides-Extra: csv
 Requires-Dist: pandas; extra == "csv"
 Provides-Extra: doc
@@ -212,24 +212,24 @@
 Requires-Dist: transformers; extra == "huggingface"
 Provides-Extra: local-inference
 Requires-Dist: markdown; extra == "local-inference"
-Requires-Dist: pypdf; extra == "local-inference"
-Requires-Dist: python-docx>=1.1.2; extra == "local-inference"
-Requires-Dist: openpyxl; extra == "local-inference"
 Requires-Dist: pdfminer.six; extra == "local-inference"
-Requires-Dist: google-cloud-vision; extra == "local-inference"
+Requires-Dist: effdet; extra == "local-inference"
 Requires-Dist: onnx; extra == "local-inference"
+Requires-Dist: networkx; extra == "local-inference"
 Requires-Dist: pytesseract; extra == "local-inference"
+Requires-Dist: pandas; extra == "local-inference"
+Requires-Dist: pypdf; extra == "local-inference"
 Requires-Dist: xlrd; extra == "local-inference"
-Requires-Dist: python-oxmsg; extra == "local-inference"
-Requires-Dist: unstructured-inference==0.7.36; extra == "local-inference"
-Requires-Dist: networkx; extra == "local-inference"
-Requires-Dist: effdet; extra == "local-inference"
 Requires-Dist: python-pptx<=0.6.23; extra == "local-inference"
-Requires-Dist: pdf2image; extra == "local-inference"
-Requires-Dist: pandas; extra == "local-inference"
+Requires-Dist: pillow_heif; extra == "local-inference"
+Requires-Dist: google-cloud-vision; extra == "local-inference"
+Requires-Dist: python-docx>=1.1.2; extra == "local-inference"
 Requires-Dist: pikepdf; extra == "local-inference"
+Requires-Dist: unstructured-inference==0.7.36; extra == "local-inference"
 Requires-Dist: pypandoc; extra == "local-inference"
-Requires-Dist: pillow_heif; extra == "local-inference"
+Requires-Dist: python-oxmsg; extra == "local-inference"
+Requires-Dist: openpyxl; extra == "local-inference"
+Requires-Dist: pdf2image; extra == "local-inference"
 Provides-Extra: paddleocr
 Requires-Dist: unstructured.paddleocr==2.6.1.3; extra == "paddleocr"
 Provides-Extra: embed-huggingface
diff -ru /tmp/round1/unstructured-0.14.10.dev12/unstructured.egg-info/requires.txt /tmp/round2/unstructured-0.14.10.dev12/unstructured.egg-info/requires.txt
--- /tmp/round1/unstructured-0.14.10.dev12/unstructured.egg-info/requires.txt   2024-07-08 20:18:13.000000000 -0700
+++ /tmp/round2/unstructured-0.14.10.dev12/unstructured.egg-info/requires.txt   2024-07-08 20:18:22.000000000 -0700
@@ -24,24 +24,24 @@

 [all-docs]
 markdown
-pypdf
-python-docx>=1.1.2
-openpyxl
 pdfminer.six
-google-cloud-vision
+effdet
 onnx
+networkx
 pytesseract
+pandas
+pypdf
 xlrd
-python-oxmsg
-unstructured-inference==0.7.36
-networkx
-effdet
 python-pptx<=0.6.23
-pdf2image
-pandas
+pillow_heif
+google-cloud-vision
+python-docx>=1.1.2
 pikepdf
+unstructured-inference==0.7.36
 pypandoc
-pillow_heif
+python-oxmsg
+openpyxl
+pdf2image

 [astra]
 astrapy
@@ -167,24 +167,24 @@

 [local-inference]
 markdown
-pypdf
-python-docx>=1.1.2
-openpyxl
 pdfminer.six
-google-cloud-vision
+effdet
 onnx
+networkx
 pytesseract
+pandas
+pypdf
 xlrd
-python-oxmsg
-unstructured-inference==0.7.36
-networkx
-effdet
 python-pptx<=0.6.23
-pdf2image
-pandas
+pillow_heif
+google-cloud-vision
+python-docx>=1.1.2
 pikepdf
+unstructured-inference==0.7.36
 pypandoc
-pillow_heif
+python-oxmsg
+openpyxl
+pdf2image

 [md]
 markdown

2 replies

jake-normal Jul 9, 2024
Author

Ah, thanks, this is hugely helpful.

jsirois Oct 9, 2024

I'll note that with pex-tool/pex#2554 Pex 2.20.3 now can lock the "unstructured" project reproducibly despite it using a set in formulating some of its extras_require.

I'll also note that I've sent off a fix to that project here: Unstructured-IO/unstructured#3712
Hopefully they accept.

benjyw · 2024-10-17T13:54:22Z

benjyw
Oct 17, 2024
Maintainer Sponsor

Note that the next release in the Pants 2.24.x series will default to Pex 2.20.3, which includes the relevant change.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hash validation with python dependencies from private git repo failing #21145

{{title}}

Replies: 2 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Hash validation with python dependencies from private git repo failing #21145

jake-normal Jul 8, 2024

Replies: 2 comments · 2 replies

jsirois Jul 9, 2024

jake-normal Jul 9, 2024 Author

jsirois Oct 9, 2024

benjyw Oct 17, 2024 Maintainer Sponsor

jake-normal
Jul 8, 2024

Replies: 2 comments 2 replies

jsirois
Jul 9, 2024

jake-normal Jul 9, 2024
Author

benjyw
Oct 17, 2024
Maintainer Sponsor