std::random_device fails on windows with NODERAWFS

[mgreter]

Given the following emscripten-bug.cpp

#include <random>
#include <iostream>

#ifdef __cplusplus
extern "C" {
#endif

void emscripten_test() {
  std::random_device rd;
  // Fails on windows if NODERAWFS is enabled
  std::cerr << "Random " << rd() << "\n";
  std::cerr << "Random " << rd() << "\n";
  std::cerr << "Random " << rd() << "\n";
}

#ifdef __cplusplus
}
#endif

Compiled via

em++ -Wall -O3 -pipe -DNDEBUG -fomit-frame-pointer -std=c++11 -c \
  -o build/emscripten-bug.o emscripten-bug.cpp
emcc build/emscripten-bug.o -o build/emscripten-bug.js \
	--js-opts 0 -O3 \
	-s WASM=0 \
	-s DETERMINISTIC=1 \
	-s EXPORTED_FUNCTIONS="['_emscripten_test']" \
	-s USE_CLOSURE_COMPILER=0 \
	-s ALLOW_MEMORY_GROWTH=1 \
	-s ENVIRONMENT=node \
	-s NODERAWFS=1 \
	--memory-init-file 0

And invoked by

const sass = require('./build/emscripten-bug');
sass._emscripten_test();

This fails to work on windows. Either removing the random_device or using NODERAWFS=0 fixes the problem, but I really want to have both :/

The reason why it fails is because NODERAWFS overwrites the virtual FS with the following code:

if (ENVIRONMENT_IS_NODE) {
  var _wrapNodeError = function(func) {
    return function() {
      try {
        return func.apply(this, arguments)
      } catch (e) {
        if (!e.code) throw e;
        throw new FS.ErrnoError(ERRNO_CODES[e.code]);
      }
    }
  };
  var VFS = Object.assign({}, FS);
  for (var _key in NODERAWFS) {
    FS[_key] = _wrapNodeError(NODERAWFS[_key]);
  }
} else {
  throw new Error("NODERAWFS is currently only supported on Node.js environment.");
}

The error happens when it tries to read /dev/urandom. Logically this file will not be available on windows and it seems that's why the original virtual FS defines those directly via:

        FS.createDevice('/dev', 'random', random_device);
        FS.createDevice('/dev', 'urandom', random_device);

So instead of using the "shimmed" device it tries to use the real ones.

The fix seems somewhat simple, but I'm not sure what the best way is.
I think it's reasonable to use the VFS for dev and proc even if NODERAWFS is enabled, as those are probably never really portable. Maybe another flag is needed (or use NODERAWFS=2) so users can decide whether they want to use "emulated" dev and proc instead of real ones.

Following is a quick hack/hotfix to get it working:

if (ENVIRONMENT_IS_NODE) {
  var VFS = Object.assign({}, FS);
  var _wrapNodeError = function(key, func) {
    return function() {
      if (arguments.length && arguments['0'].match && arguments['0'].match(/^\/+(?:dev|proc)\//)) {
        return VFS[key].apply(this, arguments);
      }
      try {
        return func.apply(this, arguments)
      } catch (e) {
        if (!e.code) throw e;
        throw new FS.ErrnoError(ERRNO_CODES[e.code]);
      }
    }
  };
  for (var _key in NODERAWFS) {
    FS[_key] = _wrapNodeError(_key, NODERAWFS[_key]);
  }
} else {
  throw new Error("NODERAWFS is currently only supported on Node.js environment.");
}

I might even be able to come up with a PR once it's decided how you want to move forward with this. Thank you and have a nice day!

[kripken]

@mgreter interesting, thanks for the writeup!

Overall the idea with NODERAWFS is to always just access the local filesystem. So it makes sense that /dev/random would not be there when on windows. But our libc does assume a POSIX-like environment, so std::random_device will look there, yeah...

I'm not sure what the best solution to that is. But as you said, always providing /dev/ etc. even in NODERAWFS may be a reasonable approach. It should probably be as an option, though, as it violates the "just access the local filesystem" rule.

Btw, what is your use case for this, I'm curious?

[mgreter]

The use case is libsass respectively https://github.com/TomiBelan/node-sass-wasm

I'm not sure what the best solution to that is.

Same here and as I wrote I agree that it could/should be decidable by users. I basically see a few different approaches, e.g. the following options seem reasonable:

• Always "shim" dev and proc fully on windows
• Only "shim" whatever is explicitly defined for VFS.
• Only shim dev/u?random on windows
• Specific options for stdx, tty, random, shm, self, etc.

As I said, I would be willing to try to come up with a PR if you can tell me what kind of options and granularity you would accept in such a PR, and preferably how the option(s) should be named. Also if you want to use different option names, or use something like NODERAWFS=2,3,....

I think the option to always "shim" dev and proc fully on windows seems reasonable for a start.

The next question would be how to implement this in the JS code. I don't think we can get around the if clause before deciding which implementation to call (VFS or NODERAWFS). Here I see two ways, one is as posted above by injecting directly into _wrapNodeError. An alternative implementation to the approach above could be:

  var _wrapNodeError = function(key, func) {
    return function() {
      if (NODEFS.isWindows) {
        switch (key) {
          case 'open':
          case 'stats':
          case 'lookupPath':
            if (arguments[0].match(/^\/+(?:dev|proc)\//)) {
              return VFS[key].apply(this, arguments);
            }
            break;
          default:
            break;
        }
      }
      try {
        return func.apply(this, arguments)
      } catch (e) {
        if (!e.code) throw e;
        throw new FS.ErrnoError(ERRNO_CODES[e.code]);
      }
    }
  };

Note: The if (NODEFS.isWindows) should be moved up to return a function on non windows envs without the path check. This should make it zero caller overhead on posix envs.

Another approach could be to wrap each NODERAWFS function:

var NODERAWFS={
  ...
  mkdir:function() { (isShimmedPath(arguments[0]) ? vfs.mkdirSync : fs.mkdirSync).apply(void 0, arguments); },
  ...
};

Personally I would prefer the first solution, as it seems less intrusive and easier to be done. Of course the if clause should only be present if the user has opted in.

Another point would be how tolerant the path check should be. E.g. it will probably not correctly handle /./dev/random. I don't think we get canonical paths at those APIs. But I believe such quirks/short-comings can be expected when compiling C++ to JS. But it should be noted when documenting such a new compile time option.

[kripken]

Thinking more on this, I'm not sure how easy it is to shim these directories. I think NODERAWFS just replaces all the filesystem support with direct calls to node - is it possible to capture some of them and handle those in MEMFS? Maybe you've already prototyped this and know the answer? :)

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.

[amit-bst]

I am also getting the same error.
Uncaught Error: NODERAWFS is currently only supported on Node.js environment.

Just trying to load json from a char* using json_loads function of jansson library in c++

[stevenwdv]

@kripken Hi, this ticket was closed automatically, but a similar issue with the same cause is still present. While std::random_device now works with -sNODERAWFS, because it has multiple supported sources (maybe added upstream?), other libraries like OpenSSL still have problems.

Specifically, OpenSSL fails to open a random source to initialize its PRNG. It supports multiple random sources, and the default, "os", means getrandom¹/getentropy with fallback to /dev/urandom//dev/random. This would lead you to think that it should work, as getentropy works with Emscripten via MUSL, but it doesn't, probably because it doesn't have the right Emscripten ifdefs, although they do have a __wasi__ ifdef, but that doesn't seem to be defined.

So, it would still be nice if the /dev/random issue could be fixed.

As an aside, do you think it's worth reporting this to OpenSSL as well? Which ifdef should they use, __EMSCRIPTEN__?

Footnotes

1. Link error when using getrandom #27257 ↩

[kleisauke]

@stevenwdv Opening /dev/urandom or /dev/random with -sNODERAWFS -sWASMFS no longer works after PR #26607 because these are absolute paths, which are no longer virtualized. PR #24733 will fix this, at least on Unix-like OSes. For non-Unix platforms, I recommend using getentropy() instead (see e.g. commit rust-lang/rust@e303661).

[kleisauke]

Sorry, I read that too quickly, I thought this was about WASMFS.

OpenSSL should probably use defined(__wasi__) || defined(__EMSCRIPTEN__) instead, or possibly just defined(__wasm__).

[stevenwdv]

@kleisauke Thanks for your reaction! I was typing a reaction but indeed this is not about the experimental FS implementation. Do you maybe have rights to reopen this issue? Or should I make a new one since the title is outdated?

I'll see if I can make a PR for OpenSSL.

[kleisauke]

It looks like this was fixed in Emscripten with commit 048f028 (which was later upstreamed in commit llvm/llvm-project@b86771a). That is, libc++ uses getentropy() to implement std::random_device.

emscripten/system/lib/libcxx/include/__config

Lines 337 to 338 in 3e979b6

	# elif defined(__wasi__) || defined(__EMSCRIPTEN__)
	# define _LIBCPP_USING_GETENTROPY

I think OpenSSL should take a similar approach, as was done in Rust in commit rust-lang/rust@e303661. Beyond that, I'm not sure there is much Emscripten can do to address this. We could try always virtualizing /dev/urandom and /dev/random, but I don't think that would work well in practice.

/tmp has a similar quirk, although it can usually be overridden by setting the TMPDIR env variable (as -sNODERAWFS implies -sNODE_HOST_ENV, which should make that possible).