-
-
Notifications
You must be signed in to change notification settings - Fork 4k
feat: zstd streaming support #10758
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: zstd streaming support #10758
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 2 Skipped Deployments
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #10758 +/- ##
==========================================
- Coverage 42.60% 38.53% -4.07%
==========================================
Files 262 241 -21
Lines 16129 14673 -1456
Branches 1549 1387 -162
==========================================
- Hits 6871 5654 -1217
+ Misses 9246 9007 -239
Partials 12 12
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This branch is available for testing at |
const inflate = zlib.createZstdDecompress({ | ||
chunkSize: 65_535, | ||
}) as nativeZlib.Inflate; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this will need to set flush: zlib.constants.ZSTD_e_flush
, as the default zstd behaviour is not to flush the output buffer if the received message doesn't close the current frame.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Last I tested this, it worked fine without that 👀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, each ws message is a full payload compressed with zstd, and I've also tested this on a bot in a guild that was getting many payloads/sec with 0 issues. Coulddd you test the hypothesis please and let us know results 🙏
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this applies only to the compression stream, not decompression. Ignore me! 👍
Please describe the changes this PR makes and why it should be merged:
Brings in native zstd streaming support for gateway decompression in environments that have it
That said, I'm leaving this as a draft for now, and seeing how zstd support progresses in node (specifically the backporting)
If someone wants to test it locally with their bots that have bigger payloads, you can use a script like this, after cloning this PR and building the repo
Runnable with
volta run --node 23 node script.mjs
or whichever tool floats your boat.Ideally we'd have some non native zstd module too but there doesn't seem to be any that exist right now 🙃
Status and versioning classification: