-
Notifications
You must be signed in to change notification settings - Fork 602
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
encodeBase64/decodeBase64 seems to be inefficient #5944
Comments
We wouldn't use |
Only |
import { encodeBase64 } from "jsr:@std/encoding/base64";
encodeBase64(Deno.readFileSync(Deno.execPath())); this crashes with out of memory error, if I use node buffer insteada it finish in a couple of milliseconds |
It seems that the memory exhaustion is likely coming from all of the string appending. I imagine that this causes a memory allocation with every new character that is appended. I have no idea if this change makes this as fast as node (probably not,) but at the very least it eliminates unnecessary memory allocations. >>> NOT TESTED <<<
|
What about running the node code if we're on a server context and the current version (or an optimized one) if we detect we're on the browser |
Also is it ok for std to use wasm libraries, in that case even the browser branch can be made efficient |
On large buffers, it seems like Node's Buffer.toString("base64") is about 6-7 times faster than the code I provided. |
@FergoTheGreat Can you check that the above snippet passes with your change? If it passes, it would be far better than the current state. |
Presonally I think you should just use wasm for this , it's the best use case of wasm (pure math work), I imagine it's going to be faster then any optimized js version https://docs.rs/base64/latest/base64/ should work well Also can you confirm it's not possible or a good idea to use node apis conditionally when we detect we're not on the browser? |
With that change, the only limiting factor is that |
I'm not in favor of changing implementation completely based on the environment. If I looked at the Deno implementation, Another point is that encoding and decoding of Base64 are planned as a language feature now (It seems already at Stage 3) https://github.com/tc39/proposal-arraybuffer-base64 |
These function use atob and btoa internally and they oom easily with large strings
Maybe instead they should use node:buffer (Buffer.from(i).toString("base64") Buffer.from(i,"base64"))
The text was updated successfully, but these errors were encountered: