Too much memory usage

[aptiko]

In production, Celery workers grow and grow. When they reach a bit more than 1G, the OS occasionally kills a worker.

Gunicorn doesn't look much better either.

[aptiko]

It's natural for Celery memory usage to grow up a little because of fragmentation. This can be mitigated with --max-tasks-per-child. However, in production memory usage grows too fast: after starting celery the worker is at 100M, and after one or two synoptic executions it has grown to 1G and keeps growing. PostgreSQL memory usage also grows to more than 1G during synoptic execution, but this memory is released at the end.

Until this is fixed, I worked around it by adding swap space to the production server. Note that in production the problem is much more severe than in development, presumably because in development only one station has data.

[aptiko]

The bad news is that it's hard to find out why we consume so much memory. enhydris-synoptic only loads the last 24 hours of data of each time series; this should be about 30k per time series. enhydris-autoprocess might also be the problem, but, again, go figure why—the largest time series we have has about a million rows and loading it into a pandas/htimeseries object consumes about 200m of memory. enhydris-autoprocess probably loads a single time series each time, and it probably only loads the part of it that needs to be processed. So we shouldn't be consuming nearly as much as 1.5g.

The good news is that Celery processes stay at 1.5g—they don't grow larger. The server seems to be fine. So we'll keep it as it is and we'll probably not fix this issue, except when we come across it in a different form.

[dkalog]

The bad news is that it's hard to find out why we consume so much memory. enhydris-synoptic only loads the last 24 hours of data of each time series; this should be about 30k per time series. enhydris-autoprocess might also be the problem, but, again, go figure why—the largest time series we have has about a million rows and loading it into a pandas/htimeseries object consumes about 200m of memory. enhydris-autoprocess probably loads a single time series each time, and it probably only loads the part of it that needs to be processed. So we shouldn't be consuming nearly as much as 1.5g.

The good news is that Celery processes stay at 1.5g—they don't grow larger. The server seems to be fine. So we'll keep it as it is and we'll probably not fix this issue, except when we come across it in a different form.

You may consider instead of pandas the Dask dataframes (https://docs.dask.org/en/latest/dataframe.html, https://towardsdatascience.com/trying-out-dask-dataframes-in-python-for-fast-data-analysis-in-parallel-aa960c18a915).