Create Firestore backups files manually

[IchordeDionysos]

When doing large-scale data exports and imports, it's often faster to do them via the import/export feature provided by Firestore.
However, when you need to do data transformations, this is no longer possible.

It would be nice if Firestore would provide a way to parse and create backup files using scripts.
With that, we could export data, transform it using a script, and then re-import that data (e.g., into a different database).

Given that the Firestore export data structure is mostly undocumented, it's hard to work with that file format to do such data transformation.

From the available documentation, it seems like it's GRPC compiled:
https://cloud.google.com/datastore/docs/export-import-entities#metadata_files

And while it works to decode the .export_metadata file, decoding fails for the .overall_export_metadata and the individual output-0 metadata files.

I've tried the following command:

protoc --decode_raw < all_namespaces/kind_flashcardSets/output-0

Another idea would be to download and work with the local Firestore emulator, but even there, it might run into issues when dealing with millions of documents...

Do you have any other tips or tricks on how to best do that?

[IchordeDionysos]

@samtstern @rwhogg do you have any insights into this?
This feels like a topic where not much documentation is available for 😌

[IchordeDionysos]

My current findings:
The data is stored as leveldb log format and the data of each leveldb record is a protocol buffer.
However. this protocol buffer seems to be an old format of Datastore and not documented anywhere properly.

My current state:
https://gist.github.com/IchordeDionysos/435b60c5ab72defb810b4753ac611253

[IchordeDionysos]

Would appreciate some hint on where to continue as this is quite a complex problem to solve 🤔