Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leak - issue while downloading large files #1706

Open
hexadivine opened this issue Feb 28, 2022 · 2 comments · May be fixed by #2453
Open

Memory leak - issue while downloading large files #1706

hexadivine opened this issue Feb 28, 2022 · 2 comments · May be fixed by #2453
Assignees
@parthea
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@hexadivine
Copy link

hexadivine commented Feb 28, 2022 •
edited
Loading

Issue :
next_chunk method stores downloaded content to 'fd' file object. This downloaded content is stored in ram. As the file size increases, it consumes more RAM to store downloaded data.

Reference -

google-api-python-client/googleapiclient/http.py

Line 755 in 5399bd3

self._fd.write(content)

Related issues:
#512
#198
@hexadivine hexadivine mentioned this issue Feb 28, 2022
Closed
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Feb 28, 2022
@parthea parthea added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. and removed triage me I really want to be triaged. labels Feb 28, 2022
@parthea parthea self-assigned this Feb 28, 2022
@hexadivine hexadivine mentioned this issue Mar 28, 2022
Closed
@hexadivine
Copy link
Author

Hi @parthea,

Please check the pull request. I have added "MediaGenBaseDownload" class which will yield fetched chunks in order to keep RAM consumption linear(equal to chunksize), with the backward compatibility.

@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. labels Jun 26, 2022
@hexadivine
Copy link
Author

hexadivine commented Jul 29, 2024 •
edited
Loading

One of my commit did not had verified tag, and also causing it to fail CLA check. I will have to create new PR.

This was referenced Jul 29, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@parthea parthea

Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
4 participants
@parthea @yoshi-automation @hexadivine and others