Does not work in node environment (upstream v3.42)

[chuanqisun]

Hi, I'm trying to use sqlite-wasm in both browser and node environment. The node environment is used for unit test. I don't expect OPFS to work so I switched to in-memory DB instead. When I run the sample code, it looks like the library is still expecting browser environment and throws an error:

Loading and initializing SQLite3 module...
Process exited with code 1
Exception loading sqlite3 module: TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11) {cause: Error: not implemented... yet...
    at makeNe…or (node:internal/deps/undici/undici:6802:35)…, stack: 'TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11)', message: 'fetch failed'}
Uncaught TypeError TypeError: fetch failed
    at fetch (internal/deps/undici/undici:11457:11)

Here is the screenshot of the setup. My node version is 20.0.0

I noticed that 3.42 ships with a sqlite-node.mjs file. So I hard coded the import path to it:

And got a self error seemingly related to #3

sqlite3 bootstrap initializer threw: ReferenceError: self is not defined
    at StructBinderFactory (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:7859:20)
    at /workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:9321:43
    at /workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:6686:13
    at Array.forEach (<anonymous>)
    at sqlite3ApiBootstrap (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:6685:44)
    at /workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:13243:32
    at callRuntimeCallbacks (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:681:26)
    at postRun (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:459:7)
    at doRun (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:5642:9)
    at run (file:///workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:5654:9) {stack: 'ReferenceError: self is not defined
    at St…m/sqlite-wasm/jswasm/sqlite3-node.mjs:5654:9)', message: 'self is not defined'}

I understand if there is no official support for node.js runtime due to lack of dev resource. I'd still like to propose such a feature as it allows test driven development. In the js ecosystem, most unit test setups require node.js based runtime. Also, wasm is supposed to be the universal binary format that is agnostic of browser vs. node.js (or even deno), so it would be amazing for sqlite to work anywhere wasm runs.

[sgbeal]

i'll fix the dangling "self" references tomorrow (it's late here). We don't officially/directly support node for the very simple reason that nobody in the sqlite project uses node in any capacity, and attempting to support a platform you don't actually use is a lost cause.

[sgbeal]

(Oooops - just deleted the closure comment because github showed it as duplicately posted when it wasn't. Rewriting...)

The "self" references have been replaced with "globalThis" upstream. That change will appear in the next 3.42.x and 3.43 releases.

As to fetch(), that comes from Emscripten-generated code. The sqlite3-node.mjs build variant blocks out that call when run from node:

  if (!binary &&
      typeof WebAssembly.instantiateStreaming == 'function' &&
      !isDataURI(binaryFile) &&
      // Avoid instantiateStreaming() on Node.js environment for now, as while
      // Node.js v18.1.0 implements it, it does not have a full fetch()
      // implementation yet.
      //
      // Reference:
      //   https://github.com/emscripten-core/emscripten/pull/16917
      !ENVIRONMENT_IS_NODE &&
      typeof fetch == 'function') {
    return fetch(binaryFile, { credentials: 'same-origin' }).then((response) => {

[irisjae]

I'm still seeing the use of self and the associated error on 3.45.3-build1; what's the plan for merging this?

[sgbeal]

what's the plan for merging this?

Merging what, specifically? The code snippet shown above is from Emscripten-generated code which we have on influence over. There is precisely one "self" use in the current code, but it also comes from Emscripten and there's a relevant comment with it:

// Note that this includes Node.js workers when relevant (pthreads is enabled).
// Node.js workers are detected as a combination of ENVIRONMENT_IS_WORKER and
// ENVIRONMENT_IS_NODE.
if (ENVIRONMENT_IS_WEB || ENVIRONMENT_IS_WORKER) {
  if (ENVIRONMENT_IS_WORKER) { // Check worker, not web, since window could be polyfilled
    scriptDirectory = self.location.href;
  }
...
}

If your issue is that "self" use, please file a ticket upstream with Emscripten.

[irisjae]

(Oooops - just deleted the closure comment because github showed it as duplicately posted when it wasn't. Rewriting...)

The "self" references have been replaced with "globalThis" upstream. That change will appear in the next 3.42.x and 3.43 releases.

As to fetch(), that comes from Emscripten-generated code. The sqlite3-node.mjs build variant blocks out that call when run from node:

  if (!binary &&
      typeof WebAssembly.instantiateStreaming == 'function' &&
      !isDataURI(binaryFile) &&
      // Avoid instantiateStreaming() on Node.js environment for now, as while
      // Node.js v18.1.0 implements it, it does not have a full fetch()
      // implementation yet.
      //
      // Reference:
      //   https://github.com/emscripten-core/emscripten/pull/16917
      !ENVIRONMENT_IS_NODE &&
      typeof fetch == 'function') {
    return fetch(binaryFile, { credentials: 'same-origin' }).then((response) => {

Ah, thanks, I see. I interpreted your comment as saying that self had already been replaced in 3.42.x/3.43.x.

If this doesn't yet work in node, perhaps we can leave this issue open?

[sgbeal]

I interpreted your comment as saying that self had already been replaced in 3.42.x/3.43.x.

The ones in our code have all bee changed - we're "self-free" since (IIRC) that change. We have no influence over the Emscripten-generated bits, though.

perhaps we can leave this issue open?

We don't specifically target node, and having this ticket open won't change that ;).

Nobody in the sqlite project uses node in any capacity except behind-the-scenes in the Emscripten build process, and claiming/attempting to support a platform we don't use would go poorly for us.

[kumavis]

heres what i needed to do to modify the node environment to get it to run

async function fetchFile(uri: string, mimeType: string = 'text/plain;charset=UTF-8'): Promise<Response> {
  if (uri.startsWith('file://')) {
    try {
      const path = new URL(uri).pathname;
      const data = await fs.promises.readFile(path);
      return new Response(data, { status: 200, statusText: 'OK', headers: { 'Content-Type': mimeType } });
    } catch (error) {
      console.error('Error reading file:', error);
      return new Response(null, { status: 404, statusText: 'File Not Found' });
    }
  }
  return new Response(null, { status: 500, statusText: 'Only file:// supported' });
}

(globalThis as any).self = globalThis;
const globalFetch = globalThis.fetch;
const wrappedFetchForSqliteWasm = async (url: string, options: RequestInit = {}): Promise<Response> => {
  if (url.endsWith('@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3.wasm')) {
    return fetchFile(url, 'application/wasm');
  }
  console.log(`fetching ${url}`);
  return globalFetch(url, options);
};
(globalThis as any).fetch = wrappedFetchForSqliteWasm;

[tomayac]

Did you see #48? Does your solution allow persisting the database?