Skip to content

Conversation

@Jansen-w
Copy link
Contributor

@Jansen-w Jansen-w commented Dec 5, 2025

Summary

This PR fixes some severe performance issues which were discovered when load testing and profiling the disk buffer v2 implementation. UUID's were added to event metadata in #21074, and while regression tests were run on that branch before merging, there weren't any regression tests which test disk buffer performance. While adding UUID's to events is inconsequential in most pipeline configurations, disk buffers were disproportionately affected by this change due to events being serialized to/deserialized from the buffer, which lead to an additional (immediately overwritten) UUID being generated per event upon deserialization (see fix in #24336). This PR enables the fast-rng feature in UUID which speeds up generation significantly, and switches the generator to v4 which further speeds up generation by foregoing v7's timestamp ordering.

All of these changes combined lead to a ~40% increase in bandwidth through a multi-threaded disk buffer load test with identical test settings.

Finally, this PR adds a regression test for disk buffers so we can test for disk buffer performance regressions in future contributions.

Flame graph of a thread before UUID optimizations:
image

After UUID Optimizations:
image

Vector configuration

How did you test this PR?

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Related: #24336

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

v7 are created using a timestamp so they can be ordered, however this comes at a performance cost. We currently don't need to order these UUID's, so for now  we can use v4.
@Jansen-w Jansen-w requested a review from a team as a code owner December 5, 2025 20:16
@github-actions github-actions bot added the domain: core Anything related to core crates i.e. vector-core, core-common, etc label Dec 5, 2025
@github-actions
Copy link

github-actions bot commented Dec 5, 2025


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Jansen-w Jansen-w added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Dec 8, 2025
@bruceg bruceg deleted the branch vectordotdev:refactor-proto-event-metadata December 9, 2025 18:51
@bruceg bruceg closed this Dec 9, 2025
@github-actions github-actions bot locked and limited conversation to collaborators Dec 9, 2025
Copy link
Contributor

@thomasqueirozb thomasqueirozb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

domain: core Anything related to core crates i.e. vector-core, core-common, etc no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants