Skip to content

Small scalability improvements for offchain data sources #6043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 30, 2025

Conversation

lutter
Copy link
Collaborator

@lutter lutter commented May 30, 2025

This changes

  1. Files that are not found are now subject to backoff - the longer a file hasn't been found, the less likely it will ever appear
  2. The maximum backoff for fds can be configured with GRAPH_FDS_MAX_BACKOFF (in seconds) The default is 600 as it was before this PR

@lutter lutter self-assigned this May 30, 2025
@@ -157,6 +162,8 @@ pub struct InnerMappingHandlers {
disable_declared_calls: EnvVarBoolean,
#[envconfig(from = "GRAPH_STORE_ERRORS_ARE_NON_DETERMINISTIC", default = "false")]
store_errors_are_nondeterministic: EnvVarBoolean,
#[envconfig(from = "Graph_FDS_MAX_BACKOFF", default = "600")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use all caps for env var key? GRAPH_FDS_MAX_BACKOFF?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoa .. nice catch. That was CoPilot, crazy

Copy link
Contributor

@fordN fordN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to merge once the formatting comment on the ENV var is addressed. 👍

@fordN fordN added performance offchain Offchain data sources, such as IPFS files labels May 30, 2025
@lutter lutter force-pushed the lutter/fds-backoff branch from 0574ed0 to cbfb912 Compare May 30, 2025 17:30
@lutter lutter merged commit cbfb912 into master May 30, 2025
6 checks passed
@lutter lutter deleted the lutter/fds-backoff branch May 30, 2025 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
offchain Offchain data sources, such as IPFS files performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants