Suggestion for installing third-party libraries

[treyhunner]

I'd like to provide a way to download arbitrary ZIP archives from PyPI and install them in the WASM environment.

One way would be to just hit the PyPI API to find and download the ZIP file for a package. The issue there is that running pip to actually install the package doesn't work (since pip is disabled).

It looks like pyodide addressed this problem by creating a micropip package.

Any recommendations for how to approach this without inventing or copying micropip? It could be that this isn't something core Python should support directly and this should be entirely left up to distributions like pyodide. But if there is (or could be) a simple way to install a Python package given a ZIP file, I'd love to know!

[brettcannon]

You can try using https://github.com/pypa/installer to install a wheel which is what you want for any py3-none-any wheel. The trick is going to be where those files end up in WebAssembly since there isn't necessarily a file system to write to (i.e. this quickly gets very emscripten-specific).

And actually micropip is a solution for Pyodide, but not WASM in general as micropip doesn't work for WASI. 😅

[pmp-p]

The trick is going to be where those files end up in WebAssembly since there isn't necessarily a file system to write to (i.e. this quickly gets very emscripten-specific).

indeed.

maybe zipimport could be slightly modified to use a MemoryView instead of file api - or maybe it supports that already ?

[treyhunner]

You can try using https://github.com/pypa/installer to install a wheel which is what you want for any py3-none-any wheel. The trick is going to be where those files end up in WebAssembly since there isn't necessarily a file system to write to (i.e. this quickly gets very emscripten-specific).

Thanks @brettcannon! I'll give installer a try at some point and see if I can figure out where the files end up.

[tiran]

For wasm32-emscripten there are actually multiple file systems. You should be able to create your FS data file and mount it on /usr/local/lib/python3.11/site-packages.

[pradyunsg]

If something doesn't work with installer, holler at me -- file an issue over there or just @-mention me on GitHub or Twitter -- I'll take a look. :)

[brettcannon]

maybe zipimport could be slightly modified to use a MemoryView instead of file api - or maybe it supports that already ?

zipimport doesn't do much that's really that magical, so there isn't a key requirement to build on top of it. Making an importer work off of an in-memory wheel file shouldn't be too bad.

[ethanhs]

A custom finder/loader would definitely be a good option. It might be nice to have a tool that runs installer into a temporary directory, zips that up, then you can either put than in an Emscripten filesystem and use zipimport or use the custom finder/loader to read from memory (the wasi engines all seem to support custom memory).

[pmp-p]

Making an importer work off of an in-memory wheel file shouldn't be too bad.

i already have code for a custom importer for getting files directly from one apk archive on android and/or multiple zip contiguous memory for wasi ( via miniz + https://github.com/kuba--/zip/issues/126 ).

But here there would be need for a memory map of the different wheel/zip and their base address + size instead of just zip names in sys.path and a function to fill the memory from file or socket with wheel data and complete the index after success.

Maybe use the same format as emscripten file_packager.py - or fully re-use it - for wasi ? instead of single files it will be wheel of files and it will be used directly in addition to sys.path ?

[schell]

Did anyone make any headway on this? I'm interested in running python.wasm on wasi with numpy.

[pmp-p]

@schell i would be very glad to help once you've built numpy for emscripten python 3.11 first (and keep the whole build tree with object files ). This could be very time consumming so i did not dive in, i'm cowardly waiting for pyodide to upgrade its build system to 3.11 ( pyodide/pyodide#289 )

The original question is probably more browser oriented because wasi does not support dynamically loading binary modules like numpy.

As discussed on discord around https://discord.com/channels/453584038356058112/915046161126137856/1004297329169408031 , installing third parties at runtime could use pypi json format or short html format (mousebender) + a map of import names <=> pypi packages names for most commonly used modules but there are traps ( or bottomless pits ?) in doing that ( this is what pyodide is actually doing with its package/repodata .json here in prettyfied form https://github.com/pygame-web/archives/blob/main/0.3.0/repodata.json )

Usefull ressources : https://github.com/pypa/installer https://github.com/brettcannon/mousebender

[schell]

Thank you @pmp-p! I may be thinking a bit too far ahead as my first hurdle is to figure out how to do some kind of simple FFI between WASI python.wasm and my Rust host application. Once I figure that out I can attempt building python.wasm on my machine, and then work on numpy. Do you have any tips on how to get some kind of FFI working?

[rth]

If it helps, we could unvendor micropip from the pyodide monorepo as a standalone package, so one could use it to install wheels in this Python build. Though it would need to be adapted to use a different FFI.

[pmp-p]

Though it would need to be adapted to use a different FFI.

would not it be time to upstream libffi changes ? Brion+Hood (maybe me too i don't remember if i added stuff to brion's : it's buried in my CI for years)

[pmp-p]

@schell ffi is experimental and only working on emscripten with javascript support ( browser or node), not wasi or standalone

you may be able to load numpy on wasi but as i said earlier you need to fully build it for emscripten first and keep the object files.

[rth]

would not it be time to upstream libffi changes

Do you mean https://github.com/hoodmane/libffi-emscripten ? Sure that would be good.

[schell]

ffi is experimental and only working on emscripten with javascript support

@pmp-p I see! I'll have to put my project on ice until then. What is the best way to help out getting experimental-grade ffi on wasi?

[pmp-p]

What is the best way to help out getting experimental-grade ffi on wasi?

the way i see it , there are two ways:

• python vm runs wasi submodules in its wasm segment, and ffi glue is written in python/c/wasm
  @vshymanskyy's pywasm3 or @mohanson's https://github.com/mohanson/pywasm could be helpfull.

• you change wasi a interpreter so it can run multiple submodules and it provides the libc (dlfcn.h) glue in its imports like emscripten's JS glue does on browser.
  wasm3 is usefull for protyping that but not for the long run where you would need a JIT.

the first would be still a bit in python scope of discussion and this thread

the second is probably more for Bytecode Alliance wasi working group and wasi vm vendoring
WebAssembly/WASI#84