Performance improvement: cache history data and top-up rather than requesting all history each run #2652
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have been investigating the performance of predbat and generated some flame graphs to see where it was spending most of it's time each run. Even on a run with no replan, predbat was taking around 1m10s to run, and a full run was taking ~2m40s.
Delving into this more deeply revealed that about 80% of the time taken during a "no replan" run was spent fetching history (
fetch_sensor_data
) from Home Assistant. I havedays_previous
set to look back 14 days, so fetching this much history each run is time consuming.This PR introduces a (disabled by default) in-memory caching mechanism for Home Assistant history data to improve performance. On first run, the cache will be populated with the history data. Subsequent runs will evict older data and request only new data from Home Assistant. The cache is not persisted; a restart will cause it to be fully re-populated. A config option allows it to be enabled/disabled.
Runtimes reduced to 0m36s (no replan)/1m11s (full run) with the cache enabled. I've been running this since last week with no obvious issues.
Summary of changes:
history_cache_enable
to allow enabling/disabling the in-memory cache for Home Assistant history data (apps/predbat/config.py
).HistoryCache
class for thread-safe, in-memory caching of history data (apps/predbat/history_cache.py
).apps/predbat/ha.py
).