Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use shared Redis cache for jhu data #306

Merged
merged 24 commits into from
Apr 30, 2020
Merged

Use shared Redis cache for jhu data #306

merged 24 commits into from
Apr 30, 2020

Conversation

Kilo59
Copy link
Collaborator

@Kilo59 Kilo59 commented Apr 29, 2020

implements #304

Formerly all gunicorn workers would have to independently make HTTP requests to the various data sources. This is very inefficient and the problem is compounded when an error pulling from one data source occurs, causing a working timeout, a new worker process to be started up and all caches have to be from scratch.

This PR uses redis to be used as a shared cache for all jhu data. It falls back to an in-memory cache if redis is not available at startup.

Other changes

  • move pylint config to `pyproject.tomal
  • use pydantic for config management

Future improvements

  1. use redis for nyt, csbs data
  2. update/simplify/refactor caching logic to work more seamlessly with either redis or in-memory fallback.
  3. update Docker compose to use redis container
  4. build the cache at application startup instead of waiting for workers to spinup.
  5. optimize redis connection handling and pool management

@Kilo59 Kilo59 self-assigned this Apr 29, 2020
@Kilo59 Kilo59 added down Related to API availability source: jhu enhancement New feature or request labels Apr 29, 2020
@Kilo59 Kilo59 linked an issue Apr 29, 2020 that may be closed by this pull request
@Kilo59 Kilo59 changed the title Use shared Redis cache Use shared Redis cache for jhu data Apr 29, 2020
@Kilo59 Kilo59 marked this pull request as ready for review April 29, 2020 03:19
@Kilo59 Kilo59 requested a review from ExpDev07 April 30, 2020 02:15
@Kilo59 Kilo59 merged commit 983fa5c into ExpDev07:master Apr 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
down Related to API availability enhancement New feature or request source: jhu
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use a shared cache for gunicorn workers
1 participant