Build a tool leveraging the existing Cardano-node DB to produce snapshots

[abailly]

Why?

• Amaru still depends on specialised snapshots of chain data to bootstrap and will probably need those for a few months, until we reach a self-sustaining point in the ledger code.
• The initial snapshot(s) in data have been manually built by @KtorZ using a mixture of Ogmios queries and consensus code tweaking.
• While we could live with it, assuming we can refresh this data every so often and sync when needed, it seems quite cumbersome and not very scalable: what if we want to support more networks, including private ones? what if we need to iterate faster and snapshots are too old?
• The ouroboros-consensus codebase already contains some tools that directly leverage the on-disk data, whether for blocks data (immutable and volatile) or ledger snapshots.
• We could adapt or build some other tool to be able to produce Amaru-compatible snapshots in a simple and automated way out of a Cardano-node's on-disk data
• Automating the snapshot production process could pave the way for multi signature publication and sharing of those snapshots, including based on Mithril stake-based threshold multi signature

What?

• build a command-line tool that can produce Amaru-compatible snapshots at arbitrary point in time (e.g slot.blockhash)
• we need some kind of specification for these snapshots, and a way to track when they change so that we know how to update the tool

[abailly]

Starting this as a discussion following @KtorZ's preferences

[jeluard]

The idea would be to leverage cardano-node persistency to feed amaru? Sounds like a neat idea!
Am I right that it would allow to bootstrap amaru from a mithril published and signed snapshot? If so, we might also consider designing a lib that could be consumed by amaru, on top of an external tool.

Note that it looks like dolos does something similar albeit for blocks only. Might be worth looking at.

[abailly]

The idea of such a tool is to extract information from Cardano-node DB so you are definitely right: We could do this with a mithril snapshot. So basically yes, this is somewhat similar to what is done in Dolos, which I was unaware of, which is great because this means we have an opportunity for reusing code and share work.
The main difference is that we want to use the ledger db part, or reconstruct a useful snapshot from this db and possibly other information. The ledger DB format is currently private and undocumented, using the Haskell code and possibly wrapping it in a command-line utility could go a long way towards supporting a more publicly accessible format.

[abailly]

Started working on such a tool here: https://github.com/pragma-org/db-server

[abailly]

I am considering providing a GET /123123/abcdef/ledger endpoint to serve a standardised ledger snapshot, possibly with access to sub-structure, from an "arbitrary" point.