Update meteor-spk to use the meteor-installed binaries

[gischer]

It no longer seems possible to build niscu, since the flags used have been removed, for one thing.

In this fork, what I have done is grab the binaries used to run a working meteor app in development, and put them into the spk.

This is a radical change, and I'm not sure you will want to take this, but it's available, as meteor-spk otherwise doesn't work with current versions of Meteor (because of incompatibility with the latest versions of mongo used by Meteor).

This has radically reduced the size of this tool. I did a bit more work to get it down to a single shell script. Namely, I encapsulated start.js and package.json as base64 encodings into meteor-spk to simplify installation and use of the tool.

[ocdtrekkie]

Hey @gischer

We definitely want to get meteor-spk updated (and probably migrated to the new community-run sandstorm.org) at some point here. So if you were able to help us with a good way to make meteor-spk supportable that would be huge for the project.

niscu isn't actually used by the current meteor-spk for new grains. niscu is only included so that old grains created using niscu can upgrade to Mongo 3.0, which is what production meteor-spk was stuck at for new grains. And then we actually have a lot of people who are stuck on needing meteor-spk to build support for upgrading from Mongo 3.0 grains to the current Mongo.

So my guess here is that production meteor-spk is great for existing grains (but cannot use modern Meteor) and then your PR works for current Meteor, but I assume can not upgrade from older grains at all, which would be needed to upgrade Wekan and similar older meteor-spk apps, right?

[gischer]

Whether it can upgrade existing grains or not is an interesting question. To test this, I took a meteor app that I had previously built an spk and deployed to sandstorm and ran meteor-spk dev on it successfully.

So that was good. What I haven't tried is updating a running grain in the field with an spk built by it. I frankly do not know much about how that works.

[gischer]

Also, I personally need a working meteor-spk, so I'm happy to share what I do in that area.

[gischer]

Thinking about it for a while, it seems to me that the primary issue that older meteor-based apps might have is mongo compatibility issues: Does the newer version of mongo read and handle the database file(s) created by the older one. I don't know the answer to that.

I do not think my method would create compatibility issues between the meteor app and mongo, since it takes whatever is in the existing installation - which one would presume is known to work.

But I could easily be overlooking something. I'm trying to figure out how to test this in a non-destructive way.

[ocdtrekkie]

I have a cache of basically every SPK ever, so we could presumably create test grains using extremely old LibreBoard/Wekan SPKs to then try updating, but I would also tag @xet7 into the conversation.

Because we are moving to a community model with sandstorm.org, it should be much easier for us to merge and ship a meteor-spk update... if someone can get it working well.

I also don't know if Mongo like... can't upgrade itself or if the existing design was based on a concern like what if a grain was killed mid-migration or something.

[xet7]

@gischer

Does the newer version of mongo read and handle the database file(s) created by the older one. I don't know the answer to that.

No. It requires using MongoDB 3 compatible mongodump command, and then MongoDB 6 compatible mongorestore command. Also, LD_LiBRARY paths etc.

Some of those are here, still incomplete, I try to migrate also Snap from MongoDB 3 to 6:

https://github.com/wekan/migratemongo

https://github.com/wekan/wekan/blob/main/snap-src/bin/mongodb-control#L21

@ocdtrekkie

if the existing design was based on a concern like what if a grain was killed mid-migration or something.

When starting migration, write mongomigration3to6started.txt, and when finished, write mongomigration3to6finished.txt. If there is started, but not finished, delete temporary files and start migration again. This way it does not matter if grain was killed mid-migration.

[xet7]

WeKan has fork of accounts-sandstorm that works with Meteor 2.x:

https://github.com/wekan/wekan/tree/main/packages/wekan-accounts-sandstorm

https://atmospherejs.com/kenton/accounts-sandstorm

[ocdtrekkie]

Yeah, I'm used to touching files to indicate where a migration is at. And one thing we are going to be able to do as the organization is publish working solutions officially so hopefully we will not need to rely as much on forks.

@xet7 Did you look at what Ian did to dump Sandstorm's database for the Tempest import? I think he used like the last version of the Go Mongo driver which supported MongoDB 3.

[xet7]

@ocdtrekkie

I did not yet look at Ian's code, I try to look at it.

I do also have some test Javascript code for Node.js/Bun/Deno, where it uses drivers for both MongoDB 3 and 6:

https://github.com/wekan/wekan-node20

At least for Bun and Deno, it's possible to create one executeable that has all the code included in same executeable. There was also some way for Node.js, but I have not tried it yet.

[gischer]

Hmmm, so I know very little about how sandstorm updates are done. The elements I see needed are:

1. A way to tell that this is an old mongo db file.
2. A way to tell that "I should port this now". Some sort of hook from Sandstorm to do the migration.
3. A way to convert (which it appears that @xet7 has been working on) old db files to new ones.).
4. A way to package this stuff into a meteor-spk tool. Sigh. I was so happy to get it down to one file. Looks like that isn't going to work.

Anything else? What am I wrong about?

[ocdtrekkie]

Fundamentally Sandstorm doesn't actually do any updates to app data. Updating an app just replaces the read-only app files which get started up to interact with the grain data. That's why all the update logic ends up inside the app package, and allows Sandstorm to be very stack independent.

So I think the current start.js in meteor-spk looks for some folders in the grain data which indicate whether the grain contains niscu data, mongo3 data, or an incomplete migration, and then migrates/cleans things up as necessary.

I love touching empty files to indicate grain state. So ideally, you could look for a file like /var/.mongo6migrated and if it doesn't exist... run the existing logic for niscu-to-mongo3 if needed, export the database as BSON from mongo3 using the existing binary, import it into mongo6, and then touch /var/.mongo6migrated so all future launches of a grain can skip all of that logic.

[gischer]

I clearly have more reading to do. Does anybody here know how Meteor addresses Mongo compatibility issues?

[xet7]

@gischer

Meteor is always adding support for newest Node.js and MongoDB versions.

Current Meteor 2 uses Node.js 14 and MongoDB 6. WeKan uses Meteor 2 currently, but not the newest Meteor 2, because upgrading to newest Meteor 2 breaks adding new board, by not redirecting to new board. I'm trying to figure out how to upgrade to Meteor 3.

Meteor 3 beta uses Node.js 20 and MongoDB 6. There is in progress to make many Meteor packages compatible with Meteor 3.

Also sometime it will be possible to replace MongoDB with FerretDB proxy to SQLite or FerretDB proxy to PostgreSQL. For WeKan, most likely trying to use SQLite.

[ocdtrekkie]

What happens in Meteor land when an old MongoDB goes out of support? If someone had a service running on MongoDB 3, did they have to upgrade and migrate the database manually when moving to newer Meteor versions which require MongoDB 5 or 6?

[xet7]

@ocdtrekkie

Old MongoDB 3 server raw database files do not work with MongoDB 6 server, they are incompatible. There is errors about incompatible MongoDB wire protocol, etc. It requires MongoDB 3 compatible mongodump, and MongoDB 6 compatible mongorestore.

Sure, it is possible to use in Node.js with both new and old Node.js MongoDB 3 and Node.js MongoDB 6 drivers at the same time, like at example at https://github.com/wekan/wekan-node20 . But newest Meteor code is using newest MongoDB syntax, like for creating indexes, etc.

[xet7]

@ocdtrekkie

Upgrading to newest Meteor 3 has many steps, like changing code to use async/await based functions, etc.

https://guide.meteor.com/3.0-migration

[ocdtrekkie]

Right, but like if you have an older bare metal Wekan install, and need to upgrade to the latest, does Meteor upgrade the Mongo database for you or does the admin have to do something manually?

[xet7]

@ocdtrekkie

For non-Sandstorm, admin does mongodump of MongoDB 3, and mongorestore to MongoDB 6, like at https://github.com/wekan/wekan/wiki/Backup

For Sandstorm, it would be possible to save WeKan grain .zip, use raw database files with MongoDB 3, do mongodump, mongorestore to MongoDB 6, copy raw database files to WeKan grain .zip, upload .zip, have newest WeKan .spk using those files.

[xet7]

But with old MongoDB 3 raw database files, newest WeKan would complain about incompatible raw database files etc.

[xet7]

That Sandstorm .zip file related stuff is described at:

https://github.com/wekan/wekan/wiki/Export-from-Wekan-Sandstorm-grain-.zip-file

[ocdtrekkie]

Okay, so yeah, it would be a manual process outside of Sandstorm too. Just wanted to make sure we weren't reinventing the wheel.

[xet7]

For the migration "wheel", I try to make it working with Snap here, but not finished yet:
https://github.com/wekan/wekan/blob/main/snap-src/bin/mongodb-control#L21

For some versions of WeKan, if someone has used WeKan feature to move attachment from MongoDB to filesystem, those files would need to be copied from filesystem too. Usually WeKan still uses attachments from MongoDB.

[xet7]

Although, I'm not sure, is it a good idea to try to support MongoDB 6, just for Meteor WeKan.

Because I'm also developing WeKan Studio, that uses SQLite. It could be possible to WeKan Studio also support MongoDB 3, not requiring any migrations.

[xet7]

https://github.com/wekan/wekanstudio

[ocdtrekkie]

Fixing the Meteor stack on Sandstorm is something we want to do in general, gischer has apps that need it, other individuals I've talked to have apps that need it, etc. Being able to get Wekan updated again would be another big win, but we're definitely looking at this as a benefit to a lot of uses.

[xet7]

@ocdtrekkie

Current Meteor WeKan is full-featured, and is getting fixes, updates, etc. Some still wait that I add more features and fixes to Meteor WeKan ASAP. While some complain Meteor WeKan is too slow with a lot of cards, they still very much like modern looking UI with many drag-drop features etc.

WeKan Studio is still minimal prototype, no ability to login yet, only All Boards page visible.

[xet7]

I would prefer changing from Meteor WeKan to WeKan Studio soon, because it would help to fix many issues that I don't know how to fix in Meteor WeKan.

[gischer]

Well, there's a lot of things about Meteor that I liked when I started with it, and I still like. I've used Ruby on Rails with SQLite before and liked that too. However, being able to reuse library code on both server and client side is really valuable, and having Meteor sync (and predict) database changes automatically is really good. Though I think some other platforms do that kind of thing now.

But wow, I read your thing about porting and that's kind of breathtaking.

One really, really dumb thing with me is have the hardest time finding mongo binaries (including mongodump and mongorestore). The source is available (but building it is, uh, complicated). Is there a good place to find it?

I just checked but meteor doesn't include them in the dev_bundle. Sigh. That would have been convenient.