Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add raw fetch API. #887

Open
twmb opened this issue Jan 15, 2025 · 2 comments
Open

Add raw fetch API. #887

twmb opened this issue Jan 15, 2025 · 2 comments
Labels
enhancement New feature or request

Comments

@twmb
Copy link
Owner

twmb commented Jan 15, 2025

  • Add batch produce API -- canceling, see first comment.
@twmb twmb added the enhancement New feature or request label Jan 15, 2025
@twmb twmb mentioned this issue Jan 15, 2025
13 tasks
@twmb twmb changed the title Add batch produce API; add raw fetch API. Add ~batch produce API~ add raw fetch API. Jan 22, 2025
@twmb twmb changed the title Add ~batch produce API~ add raw fetch API. Add raw fetch API. Jan 22, 2025
@twmb
Copy link
Owner Author

twmb commented Jan 22, 2025

It is not worth it adding a batch produce API. I'm not sure where this would be useful, and there exists an ugly but usable workaround. Batch produce APIs can be useful if you want to somewhat control how much goes into a batch before compression, but that's what lingering is for. The main and only use case I can think of is if you are consuming from one location and producing to another and you want the data on disk at both the source and destination to look the same -- i.e., you want to somewhat control how things are on the broker. A secondary use case is if you want to consume batches and ensure that two records are consumed together -- but, if this is the case, an alternative is to just combine those records into one single typed record that contains both.

Adding batched producing affects nearly all levels of the produce codepaths: the change is extensive, and the benefits seem week at most.

@rockwotj
Copy link

creating raw fetch calls using kmsg also isn't a terrible workaround for batch producing, assuming there was a nice way to construct a batch from some records...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants