Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: limit bundle by data size #188

Open
wants to merge 44 commits into
base: master
Choose a base branch
from

Conversation

segfault-magnet
Copy link
Contributor

@segfault-magnet segfault-magnet commented Jan 28, 2025

Closes: #157

Bundling can now be configured by two new config parameters:

- **`COMMITTER__APP__BUNDLE__MAX_FRAGMENTS_PER_BUNDLE`**

  - **Description:** Maximum number of fragments allowed per bundle. This limits the size of any individual bundle.
  - **Type:** Positive integer (`NonZeroUsize`)
  - **Example:** `10`
  
  - **`COMMITTER__APP__BUNDLE__BYTES_TO_ACCUMULATE`**

  - **Description:** Byte threshold that, if reached before the accumulation timeout expires, will trigger bundling immediately.
  - **Format:** Human-readable byte string (e.g., `10MB`, `500KB`)
  - **Example:** `1MB`

The idea is that we can configure the committer to give lots of data for the bundler to work with (ie achieve better compression, gas usage) while allowing it to only generate bundles that fit in N eth blobs.

Along with that we also bundle if we accumulate the configured number of blocks, so that we are more lively when we have a large backlog of unbundled blocks that are small, for example.

And there is the traditional timeout that will bundle anyways if we haven't bundled anything for a long time.

Also grouped together blocks before importing them. Mainly to avoid an edgecase mentioned in the bundler code but also to reduce the number of "imported N..=N+1 logs" we had.

@segfault-magnet segfault-magnet self-assigned this Jan 28, 2025
@segfault-magnet segfault-magnet force-pushed the feat/limit_bundle_by_data_size branch from a6715bd to 271115b Compare January 28, 2025 13:57
@segfault-magnet segfault-magnet marked this pull request as ready for review February 5, 2025 22:01
Copy link
Contributor

@hal3e hal3e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done. Left some small comments. In addition to the original task, thanks for the docs update and adding chunked fuel block imports.

hal3e
hal3e previously approved these changes Feb 13, 2025
hal3e
hal3e previously approved these changes Feb 14, 2025
Copy link
Member

@digorithm digorithm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great work on this, looking forward to seeing it live! :shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Limit final bundle size in B rather than num of blocks
3 participants