The description failed to render in the default format of reStructuredText

[SimonBaron]

Since the #5835 commit (I think) our deployments to pypi have been failing with this error:

HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://test.pypi.org/help/#description-content-type for more information. for url: https://test.pypi.org/legacy/

as previously, our setup.py specifies long_description_content_type="text/markdown"

looking at previous issues, I have tried using twine to verify it locally and it seems perfectly happy with the readme and setup.

i'm using the most recent versions of setuptools and wheel

Expected behavior
If there is a legitimate problem with our description, surely it should be that the markdown can't be read?

To Reproduce

My Platform
This is happening through a bitbucket deployment pipeline

---

i'm sure i've missed some vital info, but hopefully i can answer any questions later. Thanks.

[jamadden]

The source code or the built artifact you're trying to deploy could be useful so we can examine the metadata.

[SimonBaron]

this is the project in question: https://bitbucket.org/dativa4data/dativatools/
dativatools-2.12....tar.gz

this is the tar - github won't let me add the wheel, but you'll probably want to build it yourself anyway to check im not being an idiot :)

This is the deployment that is failing:
https://bitbucket.org/dativa4data/dativatools/addon/pipelines/home#!/results/295

[jamadden]

(There seems to be a problem with your MANIFEST.in: setup.py tries to read a file called readme.md --- all lower case --- but that file isn't included in the tarball. The log file warns about that: warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md. The version.txt file is also missing.)

The sdist you linked to isn't valid, as ... can't be part of a version number and uploading it produced HTTPError: 400 Client Error: '2.12...' is an invalid value for Version. So I unpacked it, changed the directory name and edited the PKG-INFO to have the right version and a distinct project name, and repacked it:

$ diff -u ./dativatools-2.12.../PKG-INFO ./dativatools-2.12.195/PKG-INFO
--- ./dativatools-2.12.../PKG-INFO	2019-05-15 10:06:23.000000000 -0500
+++ ./dativatools-2.12.195/PKG-INFO	2019-05-16 04:42:54.287139204 -0500
@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
-Name: dativatools
-Version: 2.12...
+Name: dativatools-jm
+Version: 2.12.195
 Summary: A selection of tools for easier processing of data using Pandas and AWS
 Home-page: https://bitbucket.org/dativa4data/dativatools/
 Author: Dativa

I then uploaded it to https://test.pypi.org/project/dativatools-jm/ successfully:

$ twine upload --verbose ./dativatools_jm-2.12.195.tar.gz -r testpypi
Uploading distributions to https://test.pypi.org/legacy/
Uploading dativatools_jm-2.12.195.tar.gz
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36.4M/36.4M [00:15<00:00, 2.44MB/s]

The markdown rendered correctly there. Obviously the metadata in the PKG-INFO is correct.

I also (accidentally!) uploaded it to pypi.org, where it also rendered correctly.

Without seeing the actual wheel, I don't have any other guesses about what might have gone wrong. Perhaps you could upload it by changing the extension, or wrapping it in another archive (.tar), or just post a link?

[SimonBaron]

thanks very much for looking at this. I worked out where the ... is coming from - we're appending bitbucket pipeline number to the version number in order to auto-increment versions when we deploy. Obviously running this locally was not achieving the same thing!

this is my wheel
wheel.tar.gz

I'm not sure I fully understand about the readme not getting included - is that normal practice? I thought that setup.py reading it in and rendering it was the key part. (Although I understand why this might be useful to you in trying to help me with my problems lol!)

[jamadden]

I'm not sure I fully understand about the readme not getting included - is that normal practice?

Yes, absolutely, for source distributions (sdists) (a .tar.gz, or for older windows versions, a .zip). This is supposed to contain everything needed to reproduce the binary distributions (bdist) (such as a wheel or an egg). Working source distributions are important to always provide for a number of reasons, both practical (there exist systems that cannot install binary distributions; future binary distribution formats will need to start from a source distribution) and philosophical (is it really open source if there's no source provided?)

[jamadden]

This time I downloaded the wheel, used wheel unpack to extract it, eyeballed that the metadata looked good, tweaked the name, and used wheel pack to recreate it. I uploaded it to test.pypi.org, where everything looks good.

So either this was just a transient error, or the distributions that was produced and uploaded today does not match what bitbucket tried to upload.

[SimonBaron]

frustratingly, i still can't make this work, i've added the readme to the manifest (although it never seemed that that was the problem).

if i change the long description to 'long_description' I can upload it: https://test.pypi.org/project/dativatools/

There must be something I'm doing that is triggering the description validation - do you have any ideas at all?

[SimonBaron]

huh.

As you can see here: https://test.pypi.org/project/dativatools/

I've truncated my readme and used http://rst.ninjs.org/ to help me format it as reStructuredText, now it uploads and then displays as markdown..

I think that confirms my theory that somehow my long_description_content_type="text/markdown", is being ignored and the description is being validated as reStructuredText.

[jamadden]

That suggests using an incompatible version of setuptools, wheel, or twine. The pipeline definitely uploaded using the newest twine, but I can’t tell about the other two. The files I uploaded had been built with current setuptools and wheel, and I uploaded with latest twine. Can you check your versions of all three?

[SimonBaron]

we're specifying in setup (updated this morning to see if it would help)

hmm, i'm not sure what we've been using - i thought we were specifying the versions in setup.py. but that doesn't make sense.

i've added steps to explicitly set the versions in build.sh:

pip install 'setuptools>=41.0.1'
pip install 'wheel=0.33.4'

python setup.py bdist_wheel --universal
python setup.py sdist

pip install 'twine>=1.11.0'

twine upload --skip-existing --repository-url https://test.pypi.org/legacy/ dist/* --verbose

still no luck

[SimonBaron]

aaaaaaaah, I think I've got it

The bitbucket pipeline was not deploying the setuptools correctly. I think I have it now, but (as usual) i have no idea how it ever worked before.

[SimonBaron]

we were specifying 'install_requires', but the environment didn't match that - is it supposed to warn or error in that case?

[jamadden]

install_requires means the version that you want at runtime (i.e., after you're installed). setup_requires is for what you need at build (setup) time. But doing that starts to get complicated and can raise issues of build isolation.

Historically, I just have something like a build_requirements.txt that I pip install before doing either pip wheel . or python setup.py bdist_wheel. The more modern alternative with pip 18+ is to use a PEP 517 pyproject.toml file to specify a PEP 518 isolated build with your build requirements. Here's an example. That can work really well, but there are still some edge cases you can run into just for having a pyproject.toml, notably around editable installs.

[SimonBaron]

i mis-typed. we are setting

      setup_requires=[
          'setuptools>=41.0.1',
          'wheel>=0.33.4']

but clearly i've not understood that. Thanks so much for the links and the help. :)