Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce hits to production servers #342

Open
jechols opened this issue Oct 3, 2024 · 0 comments
Open

Reduce hits to production servers #342

jechols opened this issue Oct 3, 2024 · 0 comments

Comments

@jechols
Copy link
Member

jechols commented Oct 3, 2024

NCA is checking the batches.json endpoint, and then all individual batch JSON endpoints, every five minutes. This was meant to ensure newly-loaded batches are seen "real time", but given #310, this isn't actually that useful a thing to have anyway.

If that were the only problem, this wouldn't be a big deal, but it seems that on some systems, it's blasting a ton of DNS requests, making the process exceptionally slow, and in one case actually causing external DNS servers to block us temporarily. (This doesn't affect production to my knowledge, because the lookup is using local DNS servers, but in dev it can be a nightmare)


A better approach is probably to leave all web issue caching to the once-a-week process that currently does its full refresh of all issue data. Then add something to the automation pipelines that adds/removes items from the cache when an automated job is successful in production. It'll get us a less brittle cache, and should still mirror what's been loaded or purged.

The tricky part is building something into the issue finder / scanner nonsense which allows us to manually modify the web cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant