Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Staging data ingestion and query optimizations #452

Open
3 tasks
nitisht opened this issue Jul 13, 2023 · 0 comments
Open
3 tasks

Staging data ingestion and query optimizations #452

nitisht opened this issue Jul 13, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@nitisht
Copy link
Member

nitisht commented Jul 13, 2023 •
edited
Loading

  • Defer the memtable.push flow outside of the write path. For example make the memtable push async / use a separate thread.
  • The disk.push should be made aysnc, so server doesn't wait for the event to be written always. Also discussed in Mitigating blocking IO in highly batched payloads #441
  • Concat of events (in the memtable) happens at 16384 events in memory. We need to either get rid of concat completely or move it out of ingest / write flow.
@nitisht nitisht added the enhancement New feature or request label Jul 13, 2023
@nitisht nitisht added this to the Release 0.5.2 milestone Jul 15, 2023
@nitisht nitisht removed this from the Release 0.6.2 milestone Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nitisht @trueleo