db: state snapshots

[canepat]

One of the main features of Erigon3 is moving most of the state content (i.e. account/storage/code) and its validity proof (i.e. commitment) to snapshot flat files. This is accomplished by introducing new abstractions for modelling the common requirements these entities have in terms storage, starting from the kind of queries they must support.

Obviously, some refactoring of the existing snapshot support is required in order to accommodate the new abstractions in the cleanest, most coherent and ultimately most useful way.

This is the main tracking issue. Individual issues are listed below:

• b+tree index implementation db: add BTree index for kv snapshots #2367
• bloom filter and existence index db: Bloom filter and existence index for kv snapshots #2386
• layering: separation between dal and datastore in db module db: datastore layer #2416 snapshots: new folder structure #2424 snapshots: KV segments #2427
• decouple SnapshotType from SnapshotPath and generalisation of SnapshotPath snapshots: SnapshotPath.step_range #2419
• definition of timed-series unit (e.g. Timestamp) and move to dal package db: datastore layer #2416 snapshots: SnapshotPath.step_range #2419
• refactoring of Snapshot, SnapshotReader as Segment, SegmentReader snapshots: new folder structure #2424
• introduce KVSegment: 3 types of iterators - pairs, keys, values snapshots: more state schema configurations #2509 snapshots: AccessorIndex rename, BTreeIndex refactoring #2518 snapshots: RAII open of snapshot files #2526
• schema definition
  • entity identifier type datastore: DataStore and related refactorings #2467 datastore: EntityName replaces SnapshotType #2491
  • decouple schema (i.e. SnapshotBundleData) from bundle and move to dal package datastore: SnapshotBundle schema #2484
  • use fluent config (a.k.a. builder) pattern for DDL-like definition of entities datastore: SnapshotBundle schema #2484
• index support evolution
  • use specific type directly as index (e.g. RecSplitIndex) and remove Index class
  • refactor IndexBuilder: multiple methods for each type or multiple builders
• segment/index decoders
  • segment decoders snapshots: state snapshot files decoders #2544
  • inverted index decoders snapshots: state snapshot files decoders #2544
  • index reader: new abstraction like Segment which decouples K/I serialization
• snapshot repository
  • refactor to work with any timestamp (BlockNum, TxStep) snapshots: timestamp-based repository #2437
  • refactor to support multiple snapshot repository instances db: provide SnapshotRepository to DataModel #2440
  • 2 instances as starting point: block snapshots and state snapshots datastore: DataStore and related refactorings #2467 datastore: EntityName replaces SnapshotType #2491
• state snapshot repository datastore: state repository schema #2494 snapshots: more state schema configurations #2509 snapshots: add state_repository to DataStoreRef #2538
• state snapshot path snapshots: e3 path format #2507
• new MDBX data model
  • refactor datastore in preparation to add D/H/II db: fix datastore namespaces #2559
  • implement inverted indexes in MDBX
  • implement domains in MDBX
• new DHII queries
  • analysis to figure out which queries are needed for D/H/II ("query objects" pattern)
  • implementation of queries
  • helper to create queries (i.e. factory)
• Data API a.k.a. Temporal KV
  • implement Inverted Index API on top of queries
  • implement History API on top of queries
  • implement Domain API on top of queries
  • gRPC DII Data API server
• Support state snapshots in C API and in silkworm-go