Work out core npm integration story

[aturon]

There are a lot of questions around the precise workflow and metadata expression for integrating Cargo and npm packages.

Design constraints

• Consumers of Rust/wasm-based packages should be completely unaware that Rust
  is involved. In particular, using such a package should not require a local
  Rust toolchain.

  • This means that publication to npm is done in binary form: we upload a
    .wasm file containing the fully-compiled Rust code.

• You should be able to work on the Rust portion of the library using standard
  Cargo workflows.

• There should be a straightforward way to express npm metadata (i.e. the
  contents of package.json) for a Rust/wasm project.

  • That means, in particular, that a Rust project might pull in several crates,
    each of which pulls in their own npm package dependencies.

• There should be an easy way to publish such a project to npm, handling all
  needed transitive dependencies.

• Ultimately, JS bundlers (like WebPack and Parcel) will need to understand
  wasm-based npm packages and generate the appropriate module instantiation.

[aturon]

cc @ashleygwilliams

[steveklabnik]

https://github.com/choojs/bankai is a real-life project that's doing this; we may want to look at what they're doing, and possibly coordinate with yosh.

[linclark]

This diagram reflects the conversation about the toolchain that happened at the Austin All Hands and should give us a good starting place for a more detailed discussion.

[aturon]

Some concrete questions, pulled in part from @linclark's summary diagram:

• How does a Rust crate express JS dependencies?
  • ... on .js files (if we allow this)
  • ... on npm packages
• Can JS included in a Rust crate depend on the functions the Rust code exports?
  • How is that expressed?
• How do we approach general "host bindings" -- functionality we expect the host to provide that we don't intend to get through npm?
  • One possibility is to require explicit .js files that define this functionality.
  • Ultimately this data needs to be fed in to the wasm module instantiation; ideally that would be "automatic"
• If we include .js files in Rust crates, how are these managed in the resulting artifact?
  • As out-of-band .js files?
  • As part of a custom section in the .wasm?
  • How does this work at the .rlib level?
• How does the bundler ultimately know how to instantiate the wasm module?
• How do we consolidate the npm dependencies across a crate graph into a form that is digestible by npm itself?
• Are Rust crates published "directly" to npm (using some tool), or is there a separate, explicit npm package (perhaps as part of the same repo)?

[aturon]

At some point soon, we should probably shard this issue into finer-grained ones tracking the various pieces. However, we need some amount of consensus around what those pieces should be :-)

[aturon]

One point I realized tonight: there may be some trickiness "flattening" a crate dependency graph that uses JS imports within multiple crates. The final .wasm blob has to have a single, flat import list. If we want to allow any of those imports to have meaningful names that are manually provided, we're going to need some kind of mangling scheme to avoid collisions with imports that we want to provide automatically.

It may be that the best strategy is to always provide all imports automatically (i.e., require you to at the very least write little .js files defining the functions you expect to be present), but even there some mangling will be needed.

[alexcrichton]

One thing I was also thinking about yesterday at some point was that I'm not sure how useful this will be without a bindgen-like tool. I'd expect most JS libraries on npm to use more than numbers/floats, so actually calling functionality from Rust will involve holding on to some sort of JS object and calling methods or accessing fields. In that sense @aturon requiring little js files to do this translation initially seems like a good idea, but in the long run I'm not sure what that means...

[fitzgen]

It is not clear to me that npm or whatever bundler should be instantiating modules automatically (or it should at least be configurable).

I am exploring designs where instead of forcing JS programmers to learn manual memory management, I create a wasm module instance encapsulated within its JS interface object and let the GC manage the lifetime of the module instance, since the only GC edge is from the JS interface object.

Essentially the .wasm would have a single global data that all of its exported functions implicitly operate on, and if we want multiple instances of that data type, we create multiple wasm module instances.

Now JS programmers don't need to learn manual memory management (which I can tell you from my experience writing heap profiling tools for JS, most of them will have a hard time with).

[lukewagner]

@aturon Yes, sharding makes sense, probably along the lines of @linclark 's diagram above.

@fitzgen

It is not clear to me that npm or whatever bundler should be instantiating modules automatically (or it should at least be configurable).

If we take the perspective that the bundler (not npm, but webpack or parcel or rollup or browserify) is just generating JS API calls (to instantiate() et al) as a polyfill for the eventual wasm ES module integration, then the wasm instantiation is directly implied by the semantics of ES modules; it's what the browser itself when it implement wasm ES module integration and, at that point, no explicit JS API calls will be needed.

I am exploring designs where instead of forcing JS programmers to learn manual memory management, I create a wasm module instance encapsulated within its JS interface object and let the GC manage the lifetime of the module instance, since the only GC edge is from the JS interface object.

Unless I'm missing your meaning, that's what we have with the above proposal; with the wasm instance being kept alive like any other ES module. The underlying assumption here is each instance is creating its own Memory and so, with wasm-bindgen, most clients of a wrapped wasm module shouldn't be doing any manual memory management.