minimal snapshot #2904
minimal snapshot #2904
Conversation
This special case is not needed as `big.NewInt(0).Bytes()` already returns empty []byte start key. This reverts commit 898d5c3.
magicxyyz
requested review from
eljobe,
tsahee and
ganeshvanahalli
and removed request for
eljobe and
tsahee
snapshotter/api.go
Outdated
| @@ -0,0 +1,96 @@ | |||
| package snapshotter | |||
There was a problem hiding this comment.
let's put snapshotter as a subdir of execution for good order
snapshotter/api.go
Outdated
| case rpc.FinalizedBlockNumber: | ||
| header = a.bc.CurrentFinalBlock() | ||
| case rpc.SafeBlockNumber: | ||
| header = a.bc.CurrentSafeBlock() |
There was a problem hiding this comment.
not for this PR: we should also have an option for latest accepted assertion
snapshotter/api.go
Outdated
| if a.promise.Ready() && rewind { | ||
| a.promise = nil | ||
| } | ||
| return promise.Current() |
There was a problem hiding this comment.
there could be a weird race condition between promise.Ready above and this promise.Current() which will be confusing.
Need to only read promise.Current() once, then if err == nil & rewind set a.promise=nil ( a ready promise cannot become errored, but a non-ready could become ready)
There was a problem hiding this comment.
fixed, thanks!
added (err == nil || errors.Is(err, containers.NotReady), to not "rewind" and loose the promise in case the job hasn't finished yet
snapshotter/blockchain_exporter.go
Outdated
| ExportTD(number uint64, hash common.Hash, tdRlp []byte) error | ||
| ExportBlockHeader(number uint64, hash common.Hash, headerRlp []byte) error | ||
| ExportBlockBody(number uint64, hash common.Hash, bodyRlp []byte) error | ||
| ExportBlockReceipts(number uint64, hash common.Hash, receiptsRlp []byte) error |
There was a problem hiding this comment.
I think all from ExportCanonical to ExportBlockReceipts should be a single function that writes a single block which is also assumed to be canonical.
Not 100% certain but I think I would make this function accept a types.Block. Conversion to/from RLP is probably negligable compared to i/o reads/writes in this function.
As for Td - one option is to read it with the block and pass it to this write function.
Specifically for nitro, though, there is another interesting option - difficulty for classic was always zero and for nitro always 1, so Td should always be block_number - genesis (I think, check me) and can be written without ever reading it.
There was a problem hiding this comment.
yeah, it felt a little like pre-mature optimization to avoid RLP decoding-encoding. I collapsed block/header/receipt/difficulty/canonical hash export methods into one method and it is much simpler and cleaner now
snapshotter/api.go
Outdated
| } | ||
| } | ||
|
|
||
| func (a *DatabaseSnapshotterAPI) Snapshot(ctx context.Context, number rpc.BlockNumber) error { |
There was a problem hiding this comment.
possibly for a different PR:
I see a few main options:
- opening a separate dir per activation of "Snapshot"
- having a separate "prepare" command. Prepare stores (about in that order) genesis, blocks, and head - but not state in head. Prepare might be called multiple times. If prepare finds an existing database that had prepared called a few times - it updates it by adding blocks and moving forward head. "Snapshot" does prepare + state of head + chain config (as a marker, or some other thing). It fails if it's called for an existing database with a chain config.
Should probably also talk with SRE about their preferance.
Resolves NIT-3057
This PR adds database snapshotter that allows exporting a minimal execution database (here called
snapshot) that consists of:In connection with manual copying of consensus database (
arbitrumdata) this can be used to create a complete databases snapshot for a full node.The snapshotter can be enabled with flag:
--execution.database-snaphotter.enable(disabled by default).Currently there are two ways to trigger the snapshot export:
SIGUSR2to nitro process - starts the export with snapshot head set to latest block which state is found in disk database (e.g.docker kill -s "SIGUSR2" nitro-node)snapshotter_snapshotrpc method that takes block number or block tag parameter - starts the export with snapshot head set to latest block that is not newer then specified block and has state available in disk database;snapshotter_resultmight be called to check the export status (the method takes "rewind" boolean parameter if set true and previous snapshotter run finished it discards cached result and new snapshot can be started)Initial version supports exporting only to geth format database. The output database is specified by following options:
--execution.database-snapshotter.geth-exporter.output.datadirectory of stored chain state (default "snapshot")
--execution.database-snapshotter.geth-exporter.output.ancientdirectory of ancient where the chain freezer can be opened (default "ancient")
--execution.database-snapshotter.geth-exporter.output.cachethe capacity(in megabytes) of the data caching (default 2048)
--execution.database-snapshotter.geth-exporter.output.db-enginebacking database implementation to use ('leveldb' or 'pebble')
--execution.database-snapshotter.geth-exporter.output.handlesnumber of files to be open simultaneously (default 512)
--execution.database-snapshotter.geth-exporter.output.namespacemetrics namespace (default "l2chaindata_export")
--execution.database-snapshotter.geth-exporter.output.pebble.*extra options for pebble engine, see --help for details
--execution.database-snapshotter.geth-exporter.ideal-batch-sizeideal write batch size in bytes (default 104857600)
Number of threads used by the snapshotter is specified by
--execution.database-snapshoter.threadsoption.Pulls: OffchainLabs/go-ethereum#402