Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove huge files from the sagemath_objects PyPI repo to fit into 10GB total #39500

Open
dimpase opened this issue Feb 12, 2025 · 8 comments
Open

Comments

@dimpase
Copy link
Member

dimpase commented Feb 12, 2025

The automatic release (dist) GH Action uploads stuff in various sagemath* repos (of them probably only sagemath-standard is of any real use).

It was reported by @tornaria that uploading to sagemath_standard failed for 10.6.beta6, see sage-release:

Note that it seems to be missing in pypi:
https://pypi.org/project/sagemath-standard/#history

upon inspection of Action logs, it turned up that sagemath_objects PyPI repo has reached its 10GB limit.
No wonder, because for the last 15 or so releases, including betas and rc's, these releases of sagemath_objects` reached almost 0.5GB in size each, see below. This madness stopped in 10.6.beta1, but it was too late, as it was already very close to the limit.

So we need to clean most of these huge files up. Or, perhaps, we can just stop putting this thing up completely.
The sizes below are in Mb, and I haven't listed anything smaller than 9.128 Mb.

('9.6rc3.post2', 9.128099)
('9.7', 96.893604)
('9.7rc1', 96.895767)
('9.7rc0', 96.895876)
('9.7rc2', 96.89657)
('9.8b0', 96.927244)
('10.3b5', 103.493559)
('10.3b7', 103.493675)
('10.3b6', 103.493795)
('10.3b8', 103.499072)
('10.3b4', 103.529557)
('10.3b2', 103.529728)
('10.3b3', 103.530044)
('10.3b1', 103.536016)
('10.2b9', 103.562181)
('10.2', 103.60229)
('10.2rc3', 103.604103)
('10.2rc4', 103.61344)
('10.2rc1', 103.613773)
('10.2rc0', 103.613805)
('10.2rc2', 103.613835)
('10.2rc5', 103.619894)
('10.3b0', 103.650675)
('10.3rc1', 103.823199)
('10.3rc2', 103.823285)
('10.3rc0', 103.823307)
('10.3', 103.854625)
('10.4b0', 103.858854)
('10.3rc4', 103.862489)
('10.3rc3', 103.86261)
('10.4b2', 103.944069)
('10.4b1', 103.948653)
('10.4b3', 103.990828)
('10.4b4', 116.615552)
('10.4b7', 126.762104)
('10.4b8', 163.275265)
('10.4', 163.563942)
('10.4b9', 163.570069)
('10.4rc2', 163.577015)
('10.4rc0', 163.577112)
('10.4rc3', 163.577183)
('10.4rc1', 163.577239)
('10.4rc4', 163.577895)
('10.6b1', 356.575292)
('10.5b3', 417.920723)
('10.5b2', 417.9213)
('10.5b1', 417.946706)
('10.5b0', 417.979177)
('10.5b5', 418.042488)
('10.5b6', 418.045587)
('10.5b7', 418.047006)
('10.5b8', 418.052714)
('10.5', 418.31797)
('10.5b9', 418.320515)
('10.5rc0', 418.325001)
('10.6b0', 418.337146)
('10.5rc2', 418.348362)
@dimpase
Copy link
Member Author

dimpase commented Feb 12, 2025

@tornaria @tobiasdiez - can sagemath_objects be just dropped from being put on PyPI ?

@dimpase
Copy link
Member Author

dimpase commented Feb 12, 2025

@kwankyu - do you perhaps know the purpose of this PyPI repo for sagemath_objects ? Is anyone using it ?

@dimpase dimpase changed the title remove huge files sagemath_objects PyPI repo to fit into 10GB total remove huge files from the sagemath_objects PyPI repo to fit into 10GB total Feb 12, 2025
@kwankyu
Copy link
Collaborator

kwankyu commented Feb 12, 2025

Obviously, to distribute the package? I don't know who is using it, but the author of https://lists.debian.org/debian-science/2024/07/msg00023.html may be using it.

@dimpase
Copy link
Member Author

dimpase commented Feb 12, 2025

As the release sizes aren't easily seen at PyPI web interface, here is how I got the numbers. 1st, got the complete project info as a JSON file j.json:

curl -sG -H 'Host: pypi.org' -H 'Accept: application/json' https://pypi.org/pypi/sagemath_objects/json >j.json

Then, opened it in Python and computed these numbers (each release has a bunch of sizes, one per platform.

import json
with open('j.json') as j:
        d=json.load(j)
rels=d['releases']
for y in sorted([(rel, sum(x['size'] for x in rels[rel])/1000000) for rel in rels.keys()], key=lambda t: t[1]):
     print(y)

@dimpase
Copy link
Member Author

dimpase commented Feb 12, 2025

Obviously, to distribute the package? I don't know who is using it, but the author of https://lists.debian.org/debian-science/2024/07/msg00023.html may be using it.

No way Debian people would be using binary packages.
And their hopes for modularisation happening this way were naive in the first place anyway, as this is a dead end as we have seen.

@dimpase
Copy link
Member Author

dimpase commented Feb 12, 2025

I've manually removed most of these almost 0.5Gb files from https://pypi.org/manage/project/sagemath-objects/releases/,
and the project size as shown on https://pypi.org/manage/project/sagemath-objects/settings/ is down to 5Gb, so we should have breathing room now.

@vbraun, @tornaria - let's keep this open until the next beta is out, and we see that sagemath-standard is back to normal.

@tobiasdiez
Copy link
Contributor

tobiasdiez commented Feb 12, 2025

@tornaria @tobiasdiez - can sagemath_objects be just dropped from being put on PyPI ?

I'm not aware of it being used anywhere. That being said, it would be good to have a broader discussion what to do with the half-finished modularization efforts. As this issue shows they do consume developer attention; without giving any value for the project at their current state. But, perhaps, Dima and I are the wrong persons to start this discussion.

@tornaria
Copy link
Contributor

FWIW, I only use sagemath-standard as a sdist from pypi, as a proxy for "sagelib".

For distro purposes, what is needed is a self-contained sdist that builds sagelib with dependencies coming from system. @tobiasdiez is the meson build good for this purpose?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants