Add support for Node.js

[tomayac]

No description provided.

[tomayac]

@sgbeal I managed to get a transient database to work on Node. I had to patch one self to be globalThis and it ran, see the changes (in sqlite3-node.mjs specifically) for details.

$ node demo/node.js
# Loading and initializing SQLite3 module...
# Done initializing. Running demo...
# Running SQLite3 version 3.43.2
# Creating a table...
# Insert some data using exec()...
# Query data with exec()...
# [ 20 ]
# [ 21 ]
# [ 22 ]

How can I create a persisted database, though? I tried:

// Works, but not persisted:
const db = new sqlite3.oo1.DB('./local', 'cw');

// `sqlite3.oo1.OpfsDb` doesn't exist.
const db = new sqlite3.oo1.OpfsDb('./local', 'cw');

// No such `vfs: kvvs`, same for `file:session?vfs=kvvs`. 
const db = new sqlite3.oo1.DB('file:local?vfs=kvvs', 'cw');

// Returns `0`.
sqlite3.capi.sqlite3_vfs_find("opfs")

Is this supposed to work somehow?

[sgbeal]

Bummer, i thought we had eliminated the last of the "self" references. i'll fix that when i'm back on the computer. OPFS can't work on node. i would expect dbs to be filesystem local, but what you're seeing above is the Emscripten i/o proxy, which is transient. Perhaps there's a way to tell Emscripten to use node's native i/o support?

Please excuse brevity - mobile device.

[sgbeal]

It just now occurs to me that node's native i/o is probably async, not posix-style, in which case it could not work with sqlite without an async-to-sync proxy vfs like the first OPFS vfs.

[sgbeal]

It turns out that that particular "self" reference is dead code and can be removed entirely:

Index: ext/wasm/api/sqlite3-api-worker1.js
==================================================================
--- ext/wasm/api/sqlite3-api-worker1.js
+++ ext/wasm/api/sqlite3-api-worker1.js
@@ -331,11 +331,10 @@
   'use strict';
   const toss = (...args)=>{throw new Error(args.join(' '))};
   if(!(globalThis.WorkerGlobalScope instanceof Function)){
     toss("initWorker1API() must be run from a Worker thread.");
   }
-  const self = this.self;
   const sqlite3 = this.sqlite3 || toss("Missing this.sqlite3 object.");
   const DB = sqlite3.oo1.DB;
 
   /**
      Returns the app-wide unique ID for the given db, creating one if
@@ -655,7 +654,7 @@
       //},
       result: result
     }, wState.xfer);
   };
   globalThis.postMessage({type:'sqlite3-api',result:'worker1-ready'});
-}.bind({self, sqlite3});
+}.bind({sqlite3});
 });

Upstream is being patched to remove that part and fix one more self ref which was just uncovered.

[tomayac]

Perhaps there's a way to tell Emscripten to use node's native i/o support?

@sgbeal There's the NODEFS that should work. Have you looked at this?

It uses node’s synchronous FS API to immediately persist any data written to the Emscripten file system to your local disk.

—Source

[sgbeal]

There's the NODEFS that should work. Have you looked at this?

i have not - i don't use node and don't currently have the bandwidth to go down that rabbit hole :/. Through mid-January my computing time is far more limited than usual because of familial responsibilities.

[sgbeal]

But yes, i believe doing a separate build with that enabled would resolve the persistent storage in node, and have added that to my TODOs to try out.

[tomayac]

According to the docs, it can be activated via the -lnodefs.js flag.

[zoren]

The addition of sqlite3InitModuleNode to index.mjs in itself would be helpful for creating transient databases in node.

[tomayac]

The addition of sqlite3InitModuleNode to index.mjs in itself would be helpful for creating transient databases in node.

I agree, but my fear is that people won't read the small print of "only transient databases supported" and expect full Node.js support. Hopefully we can actually make this happen soon.

[zoren]

If I use this branch I can use sqlite3InitModuleNode fine from node, but I cannot use the package from vite as it complains about the node dependencies being loaded.

[DallasHoff]

Yeah, sqlite3InitModuleNode should probably be exported from a different entry point like @sqlite.org/sqlite-wasm/node.

[tomayac]

I have just created a separate entry point for Node: node.mjs. This should allow interested parties to play with transient databases in Node, and at the same time work in the browser.

[zoren]

node needs to be before import for it to work for me:

    ".": {
      "types": "./index.d.ts",
      "node": "./node.mjs",
      "import": "./index.mjs",
      "main": "./index.mjs",
      "browser": "./index.mjs"
    },

But then it does work in both node and browser!

[sgbeal]

i feel compelled to point out that relying on object property order is not a great idea, as whether or not the order is retained is dependent on the how the object is traversed. See:

https://stackoverflow.com/a/30919039/1458521

In this case, the order of traversal is an implementation detail of whatever reads that import block, and that block may or may not use an order-honoring traversal. Since it's read as JSON, and JSON (rightfully) leaves the ordering implementation-defined, that block can get re-ordered if it passes through arbitrary JSON implementations.

[zoren]

i feel compelled to point out that relying on object property order is not a great idea, as whether or not the order is retained is dependent on the how the object is traversed. See:

I completely agree. However node does rely on the ordering of exports. For instance:

"default" This condition should always come last.

https://nodejs.org/api/packages.html#conditional-exports

[zoren]

I made a small demo repo in case anyone is interested. It's using a fork of this branch where "node": "./node.mjs", is before "import": "./index.mjs", as mentioned above.

It runs demo-123 on main thread and in a worker side by side in the browser. In node they run one after the other.

https://github.com/zoren/sqlite-wasm-demo

[tomayac]

I made a small demo repo in case anyone is interested. It's using a fork of this branch where "node": "./node.mjs", is before "import": "./index.mjs", as mentioned above.

It runs demo-123 on main thread and in a worker side by side in the browser. In node they run one after the other.

https://github.com/zoren/sqlite-wasm-demo

Thanks, merged your PR. This should allow you to use the current branch. We're slowly getting there. Transient seems solved now, pending is persistent databases.

[zoren]

Adding:

export function sqlite3InitModuleNode(opts?: InitOptions): Promise<Sqlite3Static>;

to index.d.ts would reflect the interface as it is.

But maybe it would be better if node.mjs default exported a function like index.mjs does to keep the same interface?

[tomayac]

But maybe it would be better if node.mjs default exported a function like index.mjs does to keep the same interface?

Changed to this.

[matthewp]

This looks ready to go, anything that's being waited on?

[tomayac]

This looks ready to go, anything that's being waited on?

It's ready to go if you don't need to persist your database, which makes me believe it's not ready for the majority of use cases.

[PabbleDabble]

Is there value to call out that this won't work in node in a persistent context in the readme?

Our app uses WebSql (only for local development) and I'm scrambling to find a replacement after reading https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system

I've spent a couple weeks now trying to get this or another option working, and just now got to this issue about node which seems to say that that this plugin doesn't support NODE + PERSTANCE

[sgbeal]

Is there value to call out that this won't work in node in a persistent context in the readme?

Given that the only supported persistence options are browser-specific, calling out that they can't work in node seems superfluous.

I've spent a couple weeks now trying to get this or another option working, and just now got to this issue about node which seems to say that that this plugin doesn't support NODE + PERSTANCE

The documentation specifically says that browsers are the only current target: https://sqlite.org/wasm/doc/trunk/about.md.

[sgbeal]

There's the NODEFS that should work. Have you looked at this?

i have not - i don't use node and don't currently have the bandwidth to go down that rabbit hole :/. Through mid-January my computing time is far more limited than usual because of familial responsibilities.

The main source tree now has a branch which modifies the node-specific build to use -lnodefs.js, which should enable NODEFS:

https://sqlite.org/src/timeline?r=wasm-nodefs&c=0bcbde7c

What is not clear is whether it will use that filesystem API by default, or whether it will still use the internal posix I/O wrappers for node. i can find no mention in the Emscripten docs about how to ensure that nodefs is made the default. It's possible that this only expose nodefs to JS, which wouldn't be helpful in our case because our module does not expose any Emscripten pieces to client-level code.

It is 100% untested as i am still, and hope to forever remain, a node-zero.

There are two options for trying this out:

First, from a checked-out copy of the main sqlite source tree:

$ fossil update wasm-nodefs
$ ./configure
$ cd ext/wasm
$ make
# go get coffee

the results will include jswasm/sqlite3-node.mjs, which uses the same jswasm/sqlite3.wasm as the rest of the builds (adding -lnodefs.js does not influence the wasm side of the build). Though i can tell emcc to build that, i've no clue how to test it.

Secondly, there is currently/temporarily a prebuilt copy of that at:

https://wasm-testing.sqlite.org/tmp/

(The filename will either be obvious or it will have been removed by the time you read this.)

[PabbleDabble]

Is there value to call out that this won't work in node in a persistent context in the readme?

Our app uses WebSql (only for local development) and I'm scrambling to find a replacement after reading https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system

I've spent a couple weeks now trying to get this or another option working, and just now got to this issue about node which seems to say that that this plugin doesn't support NODE + PERSTANCE

Thank you for the response! I don't know if there's a better 'discussion' type place to ask about this, so apologies if this is off-topic. I have a ionic/cordova/angular app that I'm trying to implement a websql replacement for my local ng serve that I run in a local browswer using localhost. Should this sqlite-wasm work in a persistent manner in an angular app?

I think I got it somewhat plumbed in to a service, but I get this message Ignoring inability to install OPFS sqlite3_vfs: The OPFS sqlite3_vfs cannot run in the main thread because it requires Atomics.wait(). or this line log('OPFS is not available, created transient database', db.filename); for the other method.

[sgbeal]

Thank you for the response! I don't know if there's a better 'discussion' type place to ask about this, so apologies if this is off-topic. I have a ionic/cordova/angular app that I'm trying to implement a websql replacement for my local ng serve that I run in a local browswer using localhost. Should this sqlite-wasm work in a persistent manner in an angular app?

I think I got it somewhat plumbed in to a service, but I get this message Ignoring inability to install OPFS sqlite3_vfs: The OPFS sqlite3_vfs cannot run in the main thread because it requires Atomics.wait(). or this line log('OPFS is not available, created transient database', db.filename); for the other method.

i can't say anything useful about node-based dev processes or any given JS framework (within the sqlite project we use, write, and publish only vanilla JS), but OPFS is not available in the main thread. The only main-thread persistence option is the so-called "kvvfs," but that one has severe size limitations because it uses localStorage or sessionStorage as its back-end:

https://sqlite.org/wasm/doc/trunk/persistence.md

There is unfortunately no direct path from WebSQL to OPFS because the former is main-thread-only and OPFS is worker-only. It is possible to "remote-control" an sqlite instance from the main thread via a worker-style interface, for example:

https://sqlite.org/wasm/doc/trunk/api-worker1.md

but that particular API has severe limitations (see that link) and is not recommended for anything more than very basic/toy applications.

The recommended way to operate it from the main thread, assuming that persistent storage is needed, is to move all of your db code into a worker thread and load sqlite into that same worker thread. That gives your worker full access to the API and eliminates all of the limitations of remote-controlling it over a proxy worker. The main thread would then communicate with that worker using postMessage() and high-level app-specific messages (e.g. {op:"getCustomerData",args:[1234]} instead of {sql:"select x,y,z from customerData where id=1234"). The latter is certainly possible as well, but is arguably more difficult to maintain long-term.

[thorsten-wolf-neptune]

Thank you for the response! I don't know if there's a better 'discussion' type place to ask about this, so apologies if this is off-topic. I have a ionic/cordova/angular app that I'm trying to implement a websql replacement for my local ng serve that I run in a local browswer using localhost. Should this sqlite-wasm work in a persistent manner in an angular app?

I think I got it somewhat plumbed in to a service, but I get this message Ignoring inability to install OPFS sqlite3_vfs: The OPFS sqlite3_vfs cannot run in the main thread because it requires Atomics.wait(). or this line log('OPFS is not available, created transient database', db.filename); for the other method.

@PabbleDabble We have exactely the same requirement. We offer a Low Code development platform where customers can create cordova apps and use WebSQL to store data and went now with the SQLite WASM approach as a replacement. The existing code of our customers is all in the main thread and we were challenged to get something running also for backwards compatibility. So we are using this library together with https://github.com/magieno/sqlite-client as we (and all our low code customers) cannot easily move the logic to the service worker level. Besides some performance considerations (#61) it currently works really well.

[nestarz]

I published on npm this forked version because of a critical issue with Deno (not accepting anymore the global property "self" since it's last version 1.45.3, see denoland/deno#24726):

https://www.npmjs.com/package/@eliaspourquoi/sqlite-node-wasm

It's a quick fix waiting for the owners to approve the merge of the current package so it can be used in Deno as well.

[samuelstroschein]

NodeJS is working after this #94 PR has been merged, replacing self with globalThis. I think this PR can be closed.

[tomayac]

NodeJS is working after this #94 PR has been merged, replacing self with globalThis. I think this PR can be closed.

I think most people would still want support for database persistence. I don't think this is currently possible with the official package, but I'm happy to be proven wrong. We'd need a way to implement the OPFS persistence option on the server.

[sgbeal]

I think most people would still want support for database persistence. I don't think this is currently possible with the official package,

Correct.

Not only is there a distinct lack of interest in node within the upstream project, but node already has popular SQLite wrappers, so there's no acute motivation for us to go down the rabbit hole of introducing one.

To the best of my knowledge, our experimental branch mentioned in #48 (comment) was never tested by any interested parties, so was allowed to stagnate.

We'd need a way to implement the OPFS persistence option on the server.

OPFS isn't strictly necessary - "just" a VFS which can use node's I/O API to handle the I/O on SQLite's behalf. However, like OPFS, node's I/O is very unfortunately async, which makes it exceedingly painful to integrate into an SQLite VFS.

[jlarmstrongiv]

Not only is there a distinct lack of interest in node within the upstream project, but node already has popular SQLite wrappers, so there's no acute motivation for us to go down the rabbit hole of introducing one.

While I’m glad projects like better-sqlite3 and the experimental sqlite3 module exist in NodeJS, I have found it difficult to share code between them. Each one has different implementations of everything from the db and statements, to how to exec, run, and return queries, and even whether to prefix variable names or not. I’d rather use wasm without automatic persistence and risk data loss, than deal with all of the subtle differences between libraries.

To the best of my knowledge, our experimental branch mentioned in #48 (comment) was never tested by any interested parties, so was allowed to stagnate.

I patch in that PR every time I install this package.

OPFS isn't strictly necessary - "just" a VFS which can use node's I/O API to handle the I/O on SQLite's behalf. However, like OPFS, node's I/O is very unfortunately async, which makes it exceedingly painful to integrate into an SQLite VFS.

I’d be curious how https://github.com/tndrle/node-sqlite3-wasm implements it

I’m about to go traveling, but would be glad to reply when I return. Thanks!

[joeltg]

I'm also really interested in getting this PR merged since it would let us do unit testing without launching puppeteer!

I don't think persistent databases in NodeJS are in scope for this library at all, but just loading and running transient databases in WASM is a completely reasonable expectation especially since it's so simple to support.

[tomayac]

Let's maybe do a highly scientific emoji voting:

• If you want the PR merged, knowing that it works in Node.js, but databases aren't persisted, vote with 👍.
• If you don't want the PR merged because you require persisted databases in Node.js, vote with 👎.

(As the volunteer maintainer of this package, I am slightly worried about people not reading the limitaion note that we would put up in the README and endless Issues wondering about persistence with Node.js.)

[sgbeal]

(As the volunteer maintainer of this package, I am slightly worried about people not reading the limitaion note that we would put up in the README and endless Issues wondering about persistence with Node.js.)

As the upstream maintainer i'll refrain from voting but am currently making a bag of popcorn to eat while watching this thread :).

PS: thank you, Thomas, for taking on this subproject!

[tomayac]

I can't believe the emoji reactions don't include the popcorn reaction. I had to go into DevTools to fix it. Here's my secret master plan: get hired at GitHub, add the emoji, and quit.