New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce memory usage during map creation #185

Open

JoshKarpel opened this issue Jan 24, 2020 · 0 comments

Assignees

Labels

enhancement internals

Contributor

JoshKarpel commented Jan 24, 2020

Right now, we pre-process all of the map data in memory while writing the map out to disk. The primary offender is

htmap/htmap/mapping.py

Line 433 in 5be40dd

def process_args_and_kwargs(args_and_kwargs: Iterator[Tuple[tuple, dict]]):

, but the ease of using the pre-processed data has downstream effects elsewhere.

Hopefully, we can completely remove the need for pre-processing. This would dramatically reduce memory usage during map creation.

JoshKarpel added enhancement internals labels

JoshKarpel self-assigned this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment