Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Measurers use an unreasonable amount of memory #2071

Open
addisoncrump opened this issue Feb 3, 2025 · 2 comments
Open

Measurers use an unreasonable amount of memory #2071

addisoncrump opened this issue Feb 3, 2025 · 2 comments

Comments

@addisoncrump
Copy link
Contributor

See issue title. When I was re-running the measurer for an experiment for which the measurers terminated early, and did not specify the runners, I OOM'd a 750GB RAM machine in a matter of minutes. The measurers stalled and ultimately returned to a stable state.

Screenshot of memory usage. Python processes are using multiple hundreds of GB of RAM each.
@addisoncrump
Copy link
Contributor Author

Actually, I just had this reproduce with the number of runners specified.

@addisoncrump addisoncrump changed the title When the number of runners is unspecified, measurers use an unreasonable amount of memory Measurers use an unreasonable amount of memory Feb 3, 2025
@addisoncrump
Copy link
Contributor Author

This seems to be related to the systemd fuzzer -- certain fuzzers produce inputs which cause it to OOM, but the OOM killer doesn't seem to always activate while the measurement is running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@addisoncrump