Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Erigon 3.0 Crashes During Block Insertion on Windows 11 #13912

Open
BjarkeCK opened this issue Feb 22, 2025 · 17 comments
Open

Erigon 3.0 Crashes During Block Insertion on Windows 11 #13912

BjarkeCK opened this issue Feb 22, 2025 · 17 comments
Assignees
@JkLondon

Comments

@BjarkeCK
Copy link

BjarkeCK commented Feb 22, 2025 •
edited
Loading

System information

Ram: 64gb
Cpu: Ryzen 9 5950X 16 cores
Disk: NVME

Erigon version: ./erigon --3.0

OS & Version: Windows 11, Version 24H2

Commit hash: 5832165

Erigon Command (with flags/config):

erigon --datadir="E:\mk\holesky_data" --chain=holesky --verbosity=5 --torrent.verbosity=5 --http.port=8547 --http --http.api=eth,debug,net,trace,web3,erigon,txpool --maxpeers=10000 --torrent.download.rate=999999mb --torrent.upload.rate=999999mb --nat=none

Chain/Network:

Happens on all test networks I've tried so far:
holesky, sepolia, and one more I can't remember the name of.

Steps to reproduce the behaviour

  1. Clone 3.0 according to your readme
  2. Run wmake.ps1
  3. erigon --datadir="E:\mk\holesky_data" --chain=holesky --verbosity=5 --torrent.verbosity=5 --http.port=8547 --http --http.api=eth,debug,net,trace,web3,erigon,txpool --maxpeers=10000 --torrent.download.rate=999999mb --torrent.upload.rate=999999mb --nat=none

Note that I've tried many different versions of that command, from scratch most of the time the past couple of days.

Error:

INFO[02-22|19:45:02.735] [Caplin] Inserting blocks                from=3356000 to=3356999
Exception 0xc0000005 0x0 0x10 0x7ff7cb8868aa
PC=0x7ff7cb8868aa
signal arrived during external code execution

runtime.cgocall(0x7ff7cb882cc0, 0xc0dc203030)
        runtime/cgocall.go:167 +0x3e fp=0xc0dc203008 sp=0xc0dc202fa0 pc=0x7ff7c9988e5e
github.com/erigontech/mdbx-go/mdbx._Cfunc_mdbx_txn_commit_ex(0x1b5875de230, 0xc0efb111f0)
        _cgo_gotypes.go:994 +0x50 fp=0xc0dc203030 sp=0xc0dc203008 pc=0x7ff7c9ed5810
github.com/erigontech/mdbx-go/mdbx.(*Txn).commit.func1(...)

Backtrace

Note that I've not been able to catch the bug in a from scratch setup, (althoug it happens there too, it's just that error pushes all the top info out of context so very quickly), so this is me re-launching the crashed instance, and getting to the error sooner, than having to wait for 400gb download.

E:\mk\erigon\build\bin>erigon --datadir="E:\mk\holesky_data" --chain=holesky --verbosity=5 --torrent.verbosity=5 --http.port=8547 --http --http.api=eth,debug,net,trace,web3,erigon,txpool --maxpeers=10000 --torrent.download.rate=999999mb --torrent.upload.rate=999999mb --nat=none
INFO[02-22|19:17:55.837] logging to file system                   log dir=E:\\mk\\holesky_data\\logs file prefix=erigon log level=info json=false
INFO[02-22|19:17:55.839] Build info                               git_branch=release/3.0 git_tag=v3.0.0-beta2-9-g5832165b71 git_commit=5832165b7139b8c4917fc0f5e2c945f2a0f96bdf
INFO[02-22|19:17:55.840]
        ########b          oo                               d####b.
        ##                                                      '##
        ##aaaa    ##d###b. dP .d####b. .d####b. ##d###b.     aaad#'
        ##        ##'  '## ## ##'  '## ##'  '## ##'  '##        '##
        ##        ##       ## ##.  .## ##.  .## ##    ##        .##
        ########P dP       dP '####P## '#####P' dP    dP    d#####P
                                   .##
                               d####P

INFO[02-22|19:17:55.840] Starting Erigon on Holesky testnet...
INFO[02-22|19:17:55.851] Maximum peer count                       total=10000
INFO[02-22|19:17:55.852] starting HTTP APIs                       port=8547 APIs=eth,debug,net,trace,web3,erigon,txpool
INFO[02-22|19:17:55.855] Set global gas cap                       cap=50000000
INFO[02-22|19:17:55.856] torrent verbosity                        level=DBG
INFO[02-22|19:17:55.864] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=999999mb upload.rate=999999mb
INFO[02-22|19:17:55.881] Opening Database                         label=chaindata path=E:\\mk\\holesky_data\\chaindata
INFO[02-22|19:18:20.538] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-22|19:18:20.541] Initialised chain configuration          config="{ChainID: 17000, Terminal Total Difficulty: 0, Shapella: 2023-09-29 15:18:24 +0000 UTC, Dencun: 2024-02-07 11:34:24 +0000 UTC, Pectra: 2025-02-24 21:55:12 +0000 UTC, Fusaka: <nil>, Engine: unknown}" genesis=0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4
INFO[02-22|19:18:23.634] UPnP device at 192.168.1.51: mapping internal UDP port 42069: success: external port 42069
INFO[02-22|19:18:23.690] UPnP device at 192.168.1.51: mapping internal TCP port 42069: success: external port 42069
INFO[02-22|19:19:01.862] Initialising Ethereum protocol           network=17000
INFO[02-22|19:19:01.863] Disk storage enabled for ethash DAGs     dir=E:\\mk\\holesky_data\\ethash-dags count=2
INFO[02-22|19:19:01.884] new subscription to logs established
INFO[02-22|19:19:01.884] rpc filters: subscribing to Erigon events
INFO[02-22|19:19:01.884] Starting private RPC server              on=127.0.0.1:9090
INFO[02-22|19:19:01.884] New txns subscriber joined
INFO[02-22|19:19:01.884] new subscription to newHeaders established
INFO[02-22|19:19:01.886] Reading JWT secret                       path=E:\\mk\\holesky_data/jwt.hex
INFO[02-22|19:19:01.887] JsonRpc endpoint opened                  ws=false ws.compression=true grpc=false http.url=127.0.0.1:8547
INFO[02-22|19:19:01.887] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
INFO[02-22|19:19:02.399] [Checkpoint Sync] Requesting beacon state uri=https://beaconstate-holesky.chainsafe.io/eth/v2/debug/beacon/states/finalized
INFO[02-22|19:19:06.930] Starting caplin
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
WARN[02-22|19:19:07.302] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.
INFO[02-22|19:19:07.313] [txpool] Started
INFO[02-22|19:19:07.314] [OtterSync] Starting Ottersync
INFO[02-22|19:19:07.314]
   _____ _             _   _                ____  _   _
  / ____| |           | | (_)              / __ \| | | |
 | (___ | |_ __ _ _ __| |_ _ _ __   __ _  | |  | | |_| |_ ___ _ __ ___ _   _ _ __   ___
  \___ \| __/ _ | '__| __| | '_ \ / _ | | |  | | __| __/ _ \ '__/ __| | | | '_ \ / __|
  ____) | || (_| | |  | |_| | | | | (_| | | |__| | |_| ||  __/ |  \__ \ |_| | | | | (__ _ _ _
 |_____/ \__\__,_|_|   \__|_|_| |_|\__, |  \____/ \__|\__\___|_|  |___/\__, |_| |_|\___(_|_|_)
                                    __/ |                               __/ |
                                   |___/                               |___/


                                        .:-===++**++===-:
                                   :=##%@@@@@@@@@@@@@@@@@@%#*=.
                               .=#@@@@@@%##+====--====+##@@@@@@@#=.     ...
                   .=**###*=:+#@@@@%*=:.                  .:=#%@@@@#==#@@@@@%#-
                 -#@@@@%%@@@@@@%+-.                            .=*%@@@@#*+*#@@@%=
                =@@@*:    -%%+:                                    -#@+.     =@@@-
                %@@#     +@#.                                        :%%-     %@@*
                @@@+    +%=.     -+=                        :=-       .#@-    %@@#
                *@@%:  #@-      =@@@*                      +@@@%.       =@= -*@@@:
                 #@@@##@+       #@@@@.                     %@@@@=        #@%@@@#-
                  :#@@@@:       +@@@#       :=++++==-.     *@@@@:        =@@@@-
                  =%@@%=         +#*.    =#%#+==-==+#%%=:  .+#*:         .#@@@#.
                 +@@%+.               .+%+-.          :=##-                :#@@@-
                -@@@=                -%#:     ..::.      +@*                 +@@%.
    .::-========*@@@..              -@#      +%@@@@%.     -@#               .-@@@+=======-
.:-====----:::::#@@%:--=::::..      #@:      *@@@@@%:      *@=      ..:-:-=--:@@@+::::----
                =@@@:.......        @@        :+@#=.       -@+        .......-@@@:
       .:=++####*%@@%=--::::..      @@   %#     %*    :@*  -@+      ...::---+@@@#*#*##+=-:
  ..--==::..     :%@@@-   ..:::..   @@   +@*:.-#@@+-.-#@-  -@+   ..:::..  .+@@@#.     ..:-
                  .#@@@##-:.        @@    :+#@%=.:+@@#=.   -@+        .-=#@@@@+
             -=+++=--+%@@%+=.       @@       +%*=+#%-      -@+       :=#@@@%+--++++=:
         .=**=:.      .=*@@@@@#=:.  @@         :--.        -@+  .-+#@@@@%+:       .:=*+-.
        ::.              .=*@@@@@@%#@@+=-:..         ..::=+#@%#@@@@@@%+-.             ..-.
                            ..=*#@@@@@@@@@@@@@@@%%@@@@@@@@@@@@@@%#+-.
                                  .:-==++*#######%######**+==-:



INFO[02-22|19:19:07.315] [1/6 OtterSync] Requesting downloads
INFO[02-22|19:19:07.351] Started P2P networking                   version=68 self=enode://f23caea9261aaf9bb7c96a0ffa3933fb0423bcfa9b9dcc33ad51061cd980fea4c335182d459547b4970a7e8a7e3ec9a56c019d89120e3223d9f23d2a99b3a256@127.0.0.1:30303 name=erigon/v3.00.0-beta2-5832165b/windows-amd64/go1.24.0
INFO[02-22|19:19:07.355] Started P2P networking                   version=67 self=enode://f23caea9261aaf9bb7c96a0ffa3933fb0423bcfa9b9dcc33ad51061cd980fea4c335182d459547b4970a7e8a7e3ec9a56c019d89120e3223d9f23d2a99b3a256@127.0.0.1:30304 name=erigon/v3.00.0-beta2-5832165b/windows-amd64/go1.24.0
INFO[02-22|19:19:08.207] Static peers                             len=0
INFO[02-22|19:19:08.208] [Sentinel] Sentinel started              app=caplin enr=enr:-LS4QG96v-quBSnrYJkkPkh_rUqZhfsiJ8hCWKxUidUzPK_QZ_NxW0WA0Yl6oWIok3Ypv95TGXLDs6qZV2TV_bCNl10Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpBprg6ZBgFwAAAAAAAAAcUAgmlkgnY0iXNlY3AyNTZrMaEDPq1hR1E0FNL1uksSX7nRDxLbUuGjeRtWnmTzxdwPddCIc3luY25ldHMAg3RjcIIPoYN1ZHCCD6A
INFO[02-22|19:19:08.209] Started Ethereum 2.0 Gossip Service      app=caplin
INFO[02-22|19:19:08.210] [Caplin] starting clstages loop          app=caplin
INFO[02-22|19:19:08.210] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=3695424
INFO[02-22|19:19:09.207] [Sentinel] Update ENR on subscription    subnet=51 subscribe=true type=attestation
INFO[02-22|19:19:09.207] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/beacon_attestation_51/ssz_snappy
INFO[02-22|19:19:09.208] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_7/ssz_snappy
INFO[02-22|19:19:09.209] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/proposer_slashing/ssz_snappy
INFO[02-22|19:19:09.209] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_4/ssz_snappy
INFO[02-22|19:19:09.209] [Sentinel] Update ENR on subscription    subnet=11 subscribe=true type=attestation
INFO[02-22|19:19:09.219] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/beacon_attestation_11/ssz_snappy
INFO[02-22|19:19:09.219] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_1/ssz_snappy
INFO[02-22|19:19:09.219] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/sync_committee_contribution_and_proof/ssz_snappy
INFO[02-22|19:19:09.219] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_3/ssz_snappy
INFO[02-22|19:19:09.221] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/beacon_block/ssz_snappy
INFO[02-22|19:19:09.221] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/bls_to_execution_change/ssz_snappy
INFO[02-22|19:19:09.222] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_0/ssz_snappy
INFO[02-22|19:19:09.222] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_8/ssz_snappy
INFO[02-22|19:19:09.222] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/attester_slashing/ssz_snappy
INFO[02-22|19:19:09.222] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_5/ssz_snappy
INFO[02-22|19:19:09.222] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_6/ssz_snappy
INFO[02-22|19:19:09.223] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/blob_sidecar_2/ssz_snappy
INFO[02-22|19:19:09.224] [Gossip] Subscribed to topic             topic=/eth2/69ae0e99/beacon_aggregate_and_proof/ssz_snappy
INFO[02-22|19:19:20.693] [1/6 OtterSync] Requesting downloads
INFO[02-22|19:19:20.709] [snapshots:download] Stat                blocks=3.35M indices=3.35M alloc=3.1GB sys=4.7GB
INFO[02-22|19:19:20.711] [snapshots:history] Stat                 blocks=3.26M txs=118.75M txNum2blockNum="64=2557K,72=2966K,76=3266K" first_history_idx_in_db=3266681 last_comitment_block=3266681 last_comitment_tx_num=118750000 alloc=3.1GB sys=4.7GB
INFO[02-22|19:20:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:21:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:22:01.862] [mem] memory stats                       alloc=2.3GB sys=4.7GB
WARN[02-22|19:22:01.862] [p2p] No GoodPeers
INFO[02-22|19:22:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.3GB sys=4.7GB
INFO[02-22|19:22:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:23:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:23:08.213] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3694879 blockNumber=3405216 blk/sec=18.2 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:23:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3693700 blockNumber=3404149 blk/sec=39.3 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:24:08.209] P2P                                      app=caplin peers=2
INFO[02-22|19:24:08.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3691783 blockNumber=3402414 blk/sec=63.9 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:24:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3689928 blockNumber=3400727 blk/sec=61.8 snapshots=0 frozenBlocks=3355999
WARN[02-22|19:25:01.862] [p2p] No GoodPeers
INFO[02-22|19:25:01.862] [mem] memory stats                       alloc=2.7GB sys=5.4GB
INFO[02-22|19:25:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.6GB sys=5.4GB
INFO[02-22|19:25:08.209] P2P                                      app=caplin peers=3
INFO[02-22|19:25:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3688136 blockNumber=3399098 blk/sec=59.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:25:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3686283 blockNumber=3397405 blk/sec=61.8 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:26:08.209] P2P                                      app=caplin peers=3
INFO[02-22|19:26:08.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3684110 blockNumber=3395429 blk/sec=72.4 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:26:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3682258 blockNumber=3393711 blk/sec=61.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:26:41.003] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-4151319383
INFO[02-22|19:27:08.209] P2P                                      app=caplin peers=2
INFO[02-22|19:27:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3680532 blockNumber=3392108 blk/sec=57.5 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:27:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3678550 blockNumber=3390270 blk/sec=66.1 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:28:01.862] [mem] memory stats                       alloc=3.8GB sys=5.9GB
WARN[02-22|19:28:01.862] [p2p] No GoodPeers
INFO[02-22|19:28:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=4.5GB sys=5.9GB
INFO[02-22|19:28:08.209] P2P                                      app=caplin peers=2
INFO[02-22|19:28:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3676694 blockNumber=3388510 blk/sec=61.9 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:28:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3674710 blockNumber=3386638 blk/sec=66.1 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:29:08.209] P2P                                      app=caplin peers=5
INFO[02-22|19:29:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3672791 blockNumber=3384864 blk/sec=64.0 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:29:33.786] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-2064624766
INFO[02-22|19:29:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3670744 blockNumber=3382925 blk/sec=68.2 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:30:08.209] P2P                                      app=caplin peers=5
INFO[02-22|19:30:08.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3668568 blockNumber=3380860 blk/sec=72.5 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:30:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3666328 blockNumber=3378793 blk/sec=74.7 snapshots=0 frozenBlocks=3355999
WARN[02-22|19:31:01.862] [p2p] No GoodPeers
INFO[02-22|19:31:01.862] [mem] memory stats                       alloc=3.6GB sys=5.9GB
INFO[02-22|19:31:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=4.3GB sys=5.9GB
INFO[02-22|19:31:08.209] P2P                                      app=caplin peers=4
INFO[02-22|19:31:08.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3664666 blockNumber=3377245 blk/sec=55.4 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:31:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3662876 blockNumber=3375602 blk/sec=59.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:32:08.209] P2P                                      app=caplin peers=6
INFO[02-22|19:32:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3660575 blockNumber=3373481 blk/sec=76.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:32:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3659233 blockNumber=3372255 blk/sec=44.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:33:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:33:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3658337 blockNumber=3371425 blk/sec=29.9 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:33:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3658019 blockNumber=3371125 blk/sec=10.6 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:34:01.862] [mem] memory stats                       alloc=5.2GB sys=5.9GB
WARN[02-22|19:34:01.862] [p2p] No GoodPeers
INFO[02-22|19:34:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.0GB sys=5.9GB
INFO[02-22|19:34:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:34:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3657699 blockNumber=3370836 blk/sec=10.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:34:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3657315 blockNumber=3370482 blk/sec=12.8 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:35:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:36:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:36:08.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3657123 blockNumber=3370308 blk/sec=6.4 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:36:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3656227 blockNumber=3369477 blk/sec=29.9 snapshots=0 frozenBlocks=3355999
WARN[02-22|19:37:01.862] [p2p] No GoodPeers
INFO[02-22|19:37:01.862] [mem] memory stats                       alloc=2.7GB sys=5.9GB
INFO[02-22|19:37:07.313] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.0GB sys=5.9GB
INFO[02-22|19:37:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:37:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3655335 blockNumber=3368665 blk/sec=29.7 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:37:12.449] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-3915214301
INFO[02-22|19:37:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3653353 blockNumber=3366840 blk/sec=66.1 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:38:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:38:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3651243 blockNumber=3364899 blk/sec=70.3 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:38:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3649196 blockNumber=3363007 blk/sec=68.2 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:39:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:39:08.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3647085 blockNumber=3361077 blk/sec=70.4 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:39:38.211] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3646191 blockNumber=3360247 blk/sec=29.8 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:40:01.862] [p2p] GoodPeers                          eth67=1
INFO[02-22|19:40:01.862] [mem] memory stats                       alloc=3.1GB sys=5.9GB
INFO[02-22|19:40:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.1GB sys=5.9GB
INFO[02-22|19:40:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:40:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3646127 blockNumber=3360189 blk/sec=2.1 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:41:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:42:08.209] P2P                                      app=caplin peers=0
INFO[02-22|19:42:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3646063 blockNumber=3360128 blk/sec=2.1 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:42:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3645041 blockNumber=3359173 blk/sec=34.1 snapshots=0 frozenBlocks=3355999
WARN[02-22|19:43:01.862] [p2p] No GoodPeers
INFO[02-22|19:43:01.862] [mem] memory stats                       alloc=3.9GB sys=5.9GB
INFO[02-22|19:43:07.312] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=4.5GB sys=5.9GB
INFO[02-22|19:43:08.209] P2P                                      app=caplin peers=2
INFO[02-22|19:43:08.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3642929 blockNumber=3357220 blk/sec=70.4 snapshots=0 frozenBlocks=3355999
INFO[02-22|19:43:38.212] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=3641650 blockNumber=3356023 blk/sec=42.6 snapshots=0 frozenBlocks=3355999
false
INFO[02-22|19:43:38.378] Missing blocks download finished (note: this does not mean that the history is complete, only that the missing blocks need for sync have been downloaded) app=caplin stage=DownloadHistoricalBlocks
INFO[02-22|19:43:43.296] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
INFO[02-22|19:43:43.297] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=3695416 to=3695618
INFO[02-22|19:44:08.209] P2P                                      app=caplin peers=1
INFO[02-22|19:44:29.413] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=3695455 distance-from-chain-tip=32m36s estimated-time-remaining=2m5s
INFO[02-22|19:44:43.666] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=3695507 distance-from-chain-tip=22m12s estimated-time-remaining=1m4s
INFO[02-22|19:45:02.637] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-2814751486
INFO[02-22|19:45:02.735] [Caplin] Inserting blocks                from=3356000 to=3356999
Exception 0xc0000005 0x0 0x10 0x7ff7cb8868aa
PC=0x7ff7cb8868aa
signal arrived during external code execution

runtime.cgocall(0x7ff7cb882cc0, 0xc0dc203030)
        runtime/cgocall.go:167 +0x3e fp=0xc0dc203008 sp=0xc0dc202fa0 pc=0x7ff7c9988e5e
github.com/erigontech/mdbx-go/mdbx._Cfunc_mdbx_txn_commit_ex(0x1b5875de230, 0xc0efb111f0)
        _cgo_gotypes.go:994 +0x50 fp=0xc0dc203030 sp=0xc0dc203008 pc=0x7ff7c9ed5810
github.com/erigontech/mdbx-go/mdbx.(*Txn).commit.func1(...)
        github.com/erigontech/[email protected]/mdbx/txn.go:246
github.com/erigontech/mdbx-go/mdbx.(*Txn).commit(0xc11c0fd9a0)
        github.com/erigontech/[email protected]/mdbx/txn.go:246 +0xba fp=0xc0dc203120 sp=0xc0dc203030 pc=0x7ff7c9eda2ba
github.com/erigontech/mdbx-go/mdbx.(*Txn).Commit(...)
        github.com/erigontech/[email protected]/mdbx/txn.go:185
github.com/erigontech/erigon-lib/kv/mdbx.(*MdbxTx).Commit(0xc11c0fd9f0)
        github.com/erigontech/[email protected]/kv/mdbx/kv_mdbx.go:896 +0x157 fp=0xc0dc2033c0 sp=0xc0dc203120 pc=0x7ff7c9f34317
github.com/erigontech/erigon-lib/kv/temporal.(*Tx).Commit(0xc0f13a7140)
        github.com/erigontech/[email protected]/kv/temporal/kv_temporal.go:201 +0x45 fp=0xc0dc2033d8 sp=0xc0dc2033c0 pc=0x7ff7cad95cc5
github.com/erigontech/erigon/turbo/execution/eth1.(*EthereumExecutionModule).InsertBlocks(0xc03a5263c0, {0x7ff7cdb46230, 0x7ff7cf345c60}, 0xc0f13a7100)
        github.com/erigontech/erigon/turbo/execution/eth1/inserters.go:88 +0x7a2 fp=0xc0dc2035c8 sp=0xc0dc2033d8 pc=0x7ff7cb7d0c22
github.com/erigontech/erigon-lib/direct.(*ExecutionClientDirect).InsertBlocks(0xc03379fc80?, {0x7ff7cdb46230?, 0x7ff7cf345c60?}, 0x61?, {0x7ff7cd06d600?, 0x7ff7cf0d1950?, 0x7ff7cb9184a0?})
        github.com/erigontech/[email protected]/direct/execution_client.go:58 +0x25 fp=0xc0dc2035f8 sp=0xc0dc2035c8 pc=0x7ff7ca218545
github.com/erigontech/erigon/turbo/execution/eth1/eth1_chain_reader.ChainReaderWriterEth1.InsertBlocksAndWait({0xc000ac4dc0?, {0x7ff7cdb6b6e0?, 0xc03439fb00?}, 0x7ff7cd2e76c3?}, {0x7ff7cdb46230, 0x7ff7cf345c60}, {0xc0e8d06008, 0x3e8, 0x4?})
        github.com/erigontech/erigon/turbo/execution/eth1/eth1_chain_reader/chain_reader.go:296 +0x1ae fp=0xc0dc2036c8 sp=0xc0dc2035f8 pc=0x7ff7cb13a52e
github.com/erigontech/erigon/cl/phase1/execution_client.(*ExecutionClientDirect).InsertBlocks(0x333947?, {0x7ff7cdb46230?, 0x7ff7cf345c60?}, {0xc0e8d06008?, 0xc0edc48380?, 0x10?}, 0x10?)
        github.com/erigontech/erigon/cl/phase1/execution_client/execution_client_direct.go:136 +0x55 fp=0xc0dc203720 sp=0xc0dc2036c8 pc=0x7ff7cb13ec75
github.com/erigontech/erigon/cl/phase1/execution_client/block_collector.(*blockCollector).Flush.func1({0xc0cc23f050, 0x28, 0x28}, {0xc0e6ba0000?, 0xc0dc203968?, 0x7ff7c9fdfef3?}, {0xc0dc203a70?, 0x0?}, 0x7ff7cd7d3490)
        github.com/erigontech/erigon/cl/phase1/execution_client/block_collector/block_collector.go:137 +0x782 fp=0xc0dc203920 sp=0xc0dc203720 pc=0x7ff7cb2ac1a2
github.com/erigontech/erigon-lib/etl.(*Collector).Load.func3({0xc0cc23f050?, 0xc0cc23f050?, 0x0?}, {0xc0e6ba0000?, 0xc0e6ba0000?, 0x0?})
        github.com/erigontech/[email protected]/etl/collector.go:267 +0x31 fp=0xc0dc203978 sp=0xc0dc203920 pc=0x7ff7c9fdba31
github.com/erigontech/erigon-lib/etl.mergeSortFiles({0x7ff7cd2c2e2a, 0xd}, {0xc131df5540, 0x4, 0x7ff7cb7c52c5?}, 0xc0dc203c30, {0x0, 0x0, 0x0, {0x0, ...}, ...}, ...)
        github.com/erigontech/[email protected]/etl/collector.go:353 +0x74a fp=0xc0dc203b18 sp=0xc0dc203978 pc=0x7ff7c9fdce6a
github.com/erigontech/erigon-lib/etl.(*Collector).Load(0xc033e1c480, {0x0, 0x0}, {0x0, 0x0}, 0xc0e2e08580, {0x0, 0x0, 0x0, {0x0, ...}, ...})
        github.com/erigontech/[email protected]/etl/collector.go:269 +0x6eb fp=0xc0dc203ce0 sp=0xc0dc203b18 pc=0x7ff7c9fdb74b
github.com/erigontech/erigon/cl/phase1/execution_client/block_collector.(*blockCollector).Flush(0xc034d230e0, {0x7ff7cdb46230, 0x7ff7cf345c60})
        github.com/erigontech/erigon/cl/phase1/execution_client/block_collector/block_collector.go:97 +0x1e7 fp=0xc0dc203df0 sp=0xc0dc203ce0 pc=0x7ff7cb2ab687
github.com/erigontech/erigon/cl/phase1/stages.chainTipSync({0x7ff7cdb46770, 0xc0397f4820}, {0x7ff7cdb5d708, 0xc04dd0af20}, 0xc00013dc08, {0x1, 0x1c31f, 0x1c320, 0x386409, 0x386405, ...})
        github.com/erigontech/erigon/cl/phase1/stages/chain_tip_sync.go:238 +0x190 fp=0xc0dc203f18 sp=0xc0dc203df0 pc=0x7ff7cb2ae370
github.com/erigontech/erigon/cl/clstages.(*StageGraph[...]).StartWithStage.func1()
        github.com/erigontech/erigon/cl/clstages/clstages.go:54 +0x6e fp=0xc0dc203fe0 sp=0xc0dc203f18 pc=0x7ff7cb2b964e
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0dc203fe8 sp=0xc0dc203fe0 pc=0x7ff7c9994581
created by github.com/erigontech/erigon/cl/clstages.(*StageGraph[...]).StartWithStage in goroutine 5765
        github.com/erigontech/erigon/cl/clstages/clstages.go:50 +0x39d

goroutine 1 gp=0xc0000021c0 m=nil [chan receive, 25 minutes]:
runtime.gopark(0xc004087740?, 0x7ff7c996209f?, 0x20?, 0x77?, 0x7ff7c99620e0?)
        runtime/proc.go:435 +0xce fp=0xc033aad708 sp=0xc033aad6e8 pc=0x7ff7c998c8ce
runtime.chanrecv(0xc000058620, 0x0, 0x1)
        runtime/chan.go:664 +0x445 fp=0xc033aad780 sp=0xc033aad708 pc=0x7ff7c99231a5
runtime.chanrecv1(0xc00178dc70?, 0xc000ac4dc0?)
        runtime/chan.go:506 +0x12 fp=0xc033aad7a8 sp=0xc033aad780 pc=0x7ff7c9922d32
github.com/erigontech/erigon/node.(*Node).Wait(...)
        github.com/erigontech/erigon/node/node.go:270
github.com/erigontech/erigon/turbo/node.(*ErigonNode).Serve(0xc03439fba0)
        github.com/erigontech/erigon/turbo/node/node.go:51 +0x65 fp=0xc033aad7f8 sp=0xc033aad7a8 pc=0x7ff7cb7e81c5
main.runErigon(0xc001907440)
        github.com/erigontech/erigon/cmd/erigon/main.go:106 +0x425 fp=0xc033aadad0 sp=0xc033aad7f8 pc=0x7ff7cb813945
github.com/erigontech/erigon/turbo/app.MakeApp.func1(0xc001907440)
        github.com/erigontech/erigon/turbo/app/make_app.go:71 +0x17f fp=0xc033aadbf8 sp=0xc033aadad0 pc=0x7ff7cb7fcc7f
github.com/urfave/cli/v2.(*Command).Run(0xc000c34420, 0xc001907440, {0xc00006c100, 0xc, 0x10})
        github.com/urfave/cli/[email protected]/command.go:276 +0x7be fp=0xc033aade68 sp=0xc033aadbf8 pc=0x7ff7c9cf113e
github.com/urfave/cli/v2.(*App).RunContext(0xc001952000, {0x7ff7cdb46230, 0x7ff7cf345c60}, {0xc00006c100, 0xc, 0x10})
        github.com/urfave/cli/[email protected]/app.go:333 +0x5a5 fp=0xc033aadec8 sp=0xc033aade68 pc=0x7ff7c9cedb85
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/[email protected]/app.go:307
main.main()
        github.com/erigontech/erigon/cmd/erigon/main.go:51 +0x85 fp=0xc033aadf50 sp=0xc033aadec8 pc=0x7ff7cb813445
runtime.main()
        runtime/proc.go:283 +0x27d fp=0xc033aadfe0 sp=0xc033aadf50 pc=0x7ff7c99583fd
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc033aadfe8 sp=0xc033aadfe0 pc=0x7ff7c9994581

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle), 5 minutes]:
runtime.gopark(0xd66258bcd2c?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000c3fa8 sp=0xc0000c3f88 pc=0x7ff7c998c8ce
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.forcegchelper()
        runtime/proc.go:348 +0xb8 fp=0xc0000c3fe0 sp=0xc0000c3fa8 pc=0x7ff7c9958718
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x7ff7c9994581
created by runtime.init.7 in goroutine 1
        runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000c5f80 sp=0xc0000c5f60 pc=0x7ff7c998c8ce
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.bgsweep(0xc0000d2000)
        runtime/mgcsweep.go:316 +0xdf fp=0xc0000c5fc8 sp=0xc0000c5f80 pc=0x7ff7c993e37f
runtime.gcenable.gowrap1()
        runtime/mgc.go:204 +0x25 fp=0xc0000c5fe0 sp=0xc0000c5fc8 pc=0x7ff7c9932685
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c5fe8 sp=0xc0000c5fe0 pc=0x7ff7c9994581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1f3e0c?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000d9f78 sp=0xc0000d9f58 pc=0x7ff7c998c8ce
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.(*scavengerState).park(0x7ff7cf2f2140)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc0000d9fa8 sp=0xc0000d9f78 pc=0x7ff7c993bdc9
runtime.bgscavenge(0xc0000d2000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc0000d9fc8 sp=0xc0000d9fa8 pc=0x7ff7c993c359
runtime.gcenable.gowrap2()
        runtime/mgc.go:205 +0x25 fp=0xc0000d9fe0 sp=0xc0000d9fc8 pc=0x7ff7c9932625
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x7ff7c9994581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000003340 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0xc0934c8900?, 0x10?, 0xe0?, 0x1000000010?)
        runtime/proc.go:435 +0xce fp=0xc0000dbe30 sp=0xc0000dbe10 pc=0x7ff7c998c8ce
runtime.runfinq()
        runtime/mfinal.go:196 +0x107 fp=0xc0000dbfe0 sp=0xc0000dbe30 pc=0x7ff7c9931687
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x7ff7c9994581
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:166 +0x3d

goroutine 6 gp=0xc000003500 m=nil [GC worker (idle)]:
runtime.gopark(0xd98586531e8?, 0x3?, 0x0?, 0x0?, 0x7ff7cb906780?)
        runtime/proc.go:435 +0xce fp=0xc0000c7f38 sp=0xc0000c7f18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000c7fc8 sp=0xc0000c7f38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000c7fe0 sp=0xc0000c7fc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c7fe8 sp=0xc0000c7fe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 18 gp=0xc0001061c0 m=nil [GC worker (idle)]:
runtime.gopark(0xd98586eab4c?, 0x3?, 0x54?, 0xab?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000d5f38 sp=0xc0000d5f18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000d5fc8 sp=0xc0000d5f38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000d5fe0 sp=0xc0000d5fc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d5fe8 sp=0xc0000d5fe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 34 gp=0xc000528000 m=nil [GC worker (idle)]:
runtime.gopark(0xd98584c1d34?, 0x1?, 0xe8?, 0x8d?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00052ff38 sp=0xc00052ff18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc00052ffc8 sp=0xc00052ff38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00052ffe0 sp=0xc00052ffc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 7 gp=0xc0000036c0 m=nil [GC worker (idle)]:
runtime.gopark(0xd98586eab4c?, 0x3?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00052bf38 sp=0xc00052bf18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc00052bfc8 sp=0xc00052bf38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00052bfe0 sp=0xc00052bfc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00052bfe8 sp=0xc00052bfe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 19 gp=0xc000106380 m=nil [GC worker (idle)]:
runtime.gopark(0xd98586531e8?, 0x3?, 0xc4?, 0xcb?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000d7f38 sp=0xc0000d7f18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000d7fc8 sp=0xc0000d7f38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000d7fe0 sp=0xc0000d7fc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d7fe8 sp=0xc0000d7fe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 35 gp=0xc0005281c0 m=nil [GC worker (idle)]:
runtime.gopark(0xd98586eab4c?, 0x3?, 0xf8?, 0x8f?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000531f38 sp=0xc000531f18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc000531fc8 sp=0xc000531f38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc000531fe0 sp=0xc000531fc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 8 gp=0xc000003880 m=nil [GC worker (idle)]:
runtime.gopark(0xd98586eab4c?, 0x3?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00052df38 sp=0xc00052df18 pc=0x7ff7c998c8ce
runtime.gcBgMarkWorker(0xc000058f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc00052dfc8 sp=0xc00052df38 pc=0x7ff7c9934ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00052dfe0 sp=0xc00052dfc8 pc=0x7ff7c9934a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x7ff7c9994581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

.... and on and on it goes!
@erthink
Copy link
Contributor

erthink commented Feb 22, 2025

@AskAlexSharov, похоже моё. Сейчас отпишу в телегу.

@erthink
Copy link
Contributor

erthink commented Feb 22, 2025

@BjarkeCK, seems this is libmdbx' bug, which only relevant under Windows.
I'll try to reproduce it.

@BjarkeCK
Copy link
Author

@BjarkeCK, seems this is libmdbx' bug, which only relevant under Windows. I'll try to reproduce it.

Did it reproduce? And do you know if there is anything I can do to work around the problem besides switching to Linux?

@erthink
Copy link
Contributor

erthink commented Feb 25, 2025 •
edited
Loading

Did it reproduce?

No, it hasn't been yet.

And do you know if there is anything I can do to work around the problem besides switching to Linux?

No.

Unfortunately, I've been a trip (away from my main work location, where the equipment is located) for a week now, and I won't be back until a month later.

Nevertheless, I hope to reproduce your situation during a week, find the cause and fix the error (if that's really the case).

It takes a lot of bustle just to get started. Right now I only have a laptop in which I will replace the NVMe and then install Windows aside Linux, etc...

In addition to the above, it is very likely that I will need your assistance.
For example, I can ask you to build the Erigon with certain options, run it under WinDbg or Visual Studio debugger and collect the necessary information.

To do this, it would be more convenient for me to contact you in a telegram (personally or in the libmdbx group).

@BjarkeCK
Copy link
Author

I appreciate it, enjoy your trip! In the meantime i can try to see if it reproduces on my other windows devices, and report back.

I will be happy to help out testing builds.

@erthink
Copy link
Contributor

erthink commented Feb 27, 2025

For now I didn't reproduce the problem. However, I got some oddities from MinGW:

In file included from Z:/xyz/libmdbx/src/alloy.c:38:
In function 'osal_bootid',
    inlined from 'osal_ctor' at Z:/xyz/libmdbx/src/osal.c:3463:20,
    inlined from 'mdbx_init' at Z:/xyz/libmdbx/src/global.c:183:3,
    inlined from 'DllMain' at Z:/xyz/libmdbx/src/global.c:38:5:
Z:/xyz/libmdbx/src/osal.c:3031:9: warning: 'UnbiasedBootTime' may be used uninitialized [-Wmaybe-uninitialized]
 3031 |         bootid_collect(&uuid, &UnbiasedBootTime, sizeof(UnbiasedBootTime));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c: In function 'DllMain':
Z:/xyz/libmdbx/src/osal.c:2733:20: note: by argument 2 of type 'const void *' to 'bootid_collect' declared here
 2733 | __cold static void bootid_collect(bin128_t *p, const void *s, size_t n) {
      |                    ^~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c:3028:22: note: 'UnbiasedBootTime' declared here
 3028 |       const uint64_t UnbiasedBootTime =
      |                      ^~~~~~~~~~~~~~~~
In function 'osal_bootid',
    inlined from 'osal_ctor' at Z:/xyz/libmdbx/src/osal.c:3463:20,
    inlined from 'mdbx_init' at Z:/xyz/libmdbx/src/global.c:183:3,
    inlined from 'DllMain' at Z:/xyz/libmdbx/src/global.c:38:5:
Z:/xyz/libmdbx/src/osal.c:3039:9: warning: 'boottime' may be used uninitialized [-Wmaybe-uninitialized]
 3039 |         bootid_collect(&uuid, &boottime, sizeof(boottime));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c: In function 'DllMain':
Z:/xyz/libmdbx/src/osal.c:2733:20: note: by argument 2 of type 'const void *' to 'bootid_collect' declared here
 2733 | __cold static void bootid_collect(bin128_t *p, const void *s, size_t n) {
      |                    ^~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c:3037:16: note: 'boottime' declared here
 3037 |       uint64_t boottime = windows_bootime();
      |                ^~~~~~~~
[46/66] Building C object CMakeFiles/mdbx-static.dir/src/alloy.c.obj
In file included from Z:/xyz/libmdbx/src/alloy.c:38:
In function 'osal_bootid',
    inlined from 'osal_ctor' at Z:/xyz/libmdbx/src/osal.c:3463:20,
    inlined from 'mdbx_init' at Z:/xyz/libmdbx/src/global.c:183:3,
    inlined from 'mdbx_module_handler' at Z:/xyz/libmdbx/src/global.c:38:5,
    inlined from 'mdbx_module_handler' at Z:/xyz/libmdbx/src/global.c:31:5:
Z:/xyz/libmdbx/src/osal.c:3031:9: warning: 'UnbiasedBootTime' may be used uninitialized [-Wmaybe-uninitialized]
 3031 |         bootid_collect(&uuid, &UnbiasedBootTime, sizeof(UnbiasedBootTime));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c: In function 'mdbx_module_handler':
Z:/xyz/libmdbx/src/osal.c:2733:20: note: by argument 2 of type 'const void *' to 'bootid_collect' declared here
 2733 | __cold static void bootid_collect(bin128_t *p, const void *s, size_t n) {
      |                    ^~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c:3028:22: note: 'UnbiasedBootTime' declared here
 3028 |       const uint64_t UnbiasedBootTime =
      |                      ^~~~~~~~~~~~~~~~
In function 'osal_bootid',
    inlined from 'osal_ctor' at Z:/xyz/libmdbx/src/osal.c:3463:20,
    inlined from 'mdbx_init' at Z:/xyz/libmdbx/src/global.c:183:3,
    inlined from 'mdbx_module_handler' at Z:/xyz/libmdbx/src/global.c:38:5,
    inlined from 'mdbx_module_handler' at Z:/xyz/libmdbx/src/global.c:31:5:
Z:/xyz/libmdbx/src/osal.c:3039:9: warning: 'boottime' may be used uninitialized [-Wmaybe-uninitialized]
 3039 |         bootid_collect(&uuid, &boottime, sizeof(boottime));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c: In function 'mdbx_module_handler':
Z:/xyz/libmdbx/src/osal.c:2733:20: note: by argument 2 of type 'const void *' to 'bootid_collect' declared here
 2733 | __cold static void bootid_collect(bin128_t *p, const void *s, size_t n) {
      |                    ^~~~~~~~~~~~~~
Z:/xyz/libmdbx/src/osal.c:3037:16: note: 'boottime' declared here
 3037 |       uint64_t boottime = windows_bootime();
      |                ^~~~~~~~

These warnings are completely insane, as the compiler reports the use of an uninitialized variable in the assignment/initialization operation of this variable.

This can be either a consequence of a harmless flaw, or a manifestation of a catastrophic error in the compiler, leading to the generation of incorrect code.

I will continue to analyze the code and try to reproduce the issue, but hypothetically, a akin bug in MinGW is enough to explain the issue and the fact that I did not notice any problems when testing using MSVC.

@erthink
Copy link
Contributor

erthink commented Feb 27, 2025

@BjarkeCK, please show the output of mdbx_chk -V from your Erigon build.
ASAP.

@erthink
Copy link
Contributor

erthink commented Feb 27, 2025 •
edited
Loading

@BjarkeCK, It turned out that Erigon uses an outdated version 0.12.10 of libmdbx, where the corresponding error has not fixed.

The Erigon team has already been notified of this issue and it will be fixed in a reasonable time.

@JkLondon
Copy link
Member

@BjarkeCK hi, u can try out mdbx-mega-stable branch with 0.12.13 version
(32b4c61 revision number if needed)

@BjarkeCK
Copy link
Author

Thank you! @JkLondon

Gave it a try - cloned / build / downloaded from scartch, I'm sadly still getting the same error as far as I can tell:

Exception 0xc0000005 0x0 0x10 0x7ff6294c7aaa
PC=0x7ff6294c7aaa
signal arrived during external code execution
E:\eth\erigon>git status
On branch mdbx-mega-stable
Your branch is up to date with 'origin/mdbx-mega-stable'.

nothing to commit, working tree clean
E:\eth\erigon\build\bin>erigon --datadir="E:\eth\holesky_data" --chain=holesky --verbosity=5 --torrent.verbosity=5 --http.port=8546 --http --http.api=eth,debug,net,trace,web3,erigon --maxpeers=10000 --torrent.download.rate=999999mb --torrent.upload.rate=999999mb --nat=none
INFO[02-28|18:34:09.158] logging to file system                   log dir=E:\\eth\\holesky_data\\logs file prefix=erigon log level=info json=false
INFO[02-28|18:34:09.159] Build info                               git_branch=mdbx-mega-stable git_tag=v3.0.0-beta1-235-g32b4c61ce8 git_commit=32b4c61ce801fc700b88ae6decba8756987753e5
INFO[02-28|18:34:09.160]
        ########b          oo                               d####b.
        ##                                                      '##
        ##aaaa    ##d###b. dP .d####b. .d####b. ##d###b.     aaad#'
        ##        ##'  '## ## ##'  '## ##'  '## ##'  '##        '##
        ##        ##       ## ##.  .## ##.  .## ##    ##        .##
        ########P dP       dP '####P## '#####P' dP    dP    d#####P
                                   .##
                               d####P

INFO[02-28|18:34:09.161] Starting Erigon on Holesky testnet...
INFO[02-28|18:34:10.797] Maximum peer count                       total=10000
INFO[02-28|18:34:10.798] starting HTTP APIs                       port=8546 APIs=eth,debug,net,trace,web3,erigon
INFO[02-28|18:34:10.799] Set global gas cap                       cap=50000000
INFO[02-28|18:34:10.801] torrent verbosity                        level=DBG
INFO[02-28|18:34:10.815] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=999999mb upload.rate=999999mb
INFO[02-28|18:34:10.816] Opening Database                         label=chaindata path=E:\\eth\\holesky_data\\chaindata
INFO[02-28|18:34:10.820] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-28|18:34:10.821] Re-Opening DB in exclusive mode to apply migrations
INFO[02-28|18:34:10.830] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-28|18:34:10.831] Apply migration                          name=db_schema_version5
INFO[02-28|18:34:10.833] Applied migration                        name=db_schema_version5
INFO[02-28|18:34:10.833] Apply migration                          name=prohibit_new_downloads_lock
INFO[02-28|18:34:10.834] Applied migration                        name=prohibit_new_downloads_lock
INFO[02-28|18:34:10.834] Apply migration                          name=prohibit_new_downloads_lock2
INFO[02-28|18:34:10.836] Applied migration                        name=prohibit_new_downloads_lock2
INFO[02-28|18:34:10.836] Apply migration                          name=clear_bor_tables
INFO[02-28|18:34:10.837] Applied migration                        name=clear_bor_tables
INFO[02-28|18:34:10.837] Apply migration                          name=reset_stage_txn_lookup
INFO[02-28|18:34:10.839] Applied migration                        name=reset_stage_txn_lookup
INFO[02-28|18:34:10.840] Updated DB schema to                     version=7.0.0
INFO[02-28|18:34:10.848] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-28|18:34:10.860] Writing custom genesis block             hash=0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4
INFO[02-28|18:34:10.861] Initialised chain configuration          config="{ChainID: 17000, Terminal Total Difficulty: 0, Shapella: 2023-09-29 15:18:24 +0000 UTC, Dencun: 2024-02-07 11:34:24 +0000 UTC, Pectra: 2025-02-24 21:55:12 +0000 UTC, Fusaka: <nil>, Engine: unknown}" genesis=0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4
INFO[02-28|18:34:10.880] Initialising Ethereum protocol           network=17000
INFO[02-28|18:34:10.880] Disk storage enabled for ethash DAGs     dir=E:\\eth\\holesky_data\\ethash-dags count=2
INFO[02-28|18:34:10.887] rpc filters: subscribing to Erigon events
INFO[02-28|18:34:10.887] Starting private RPC server              on=127.0.0.1:9090
INFO[02-28|18:34:10.887] New txns subscriber joined
INFO[02-28|18:34:10.889] Reading JWT secret                       path=E:\\eth\\holesky_data/jwt.hex
INFO[02-28|18:34:10.891] JsonRpc endpoint opened                  ws=false ws.compression=true grpc=false http.url=127.0.0.1:8546
INFO[02-28|18:34:10.891] Generated JWT secret                     path=E:\\eth\\holesky_data/jwt.hex
INFO[02-28|18:34:10.891] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
INFO[02-28|18:34:13.149] UPnP device at 192.168.1.51: mapping internal TCP port 42069: success: external port 42069
INFO[02-28|18:34:13.205] UPnP device at 192.168.1.51: mapping internal UDP port 42069: success: external port 42069
INFO[02-28|18:34:15.445] Started P2P networking                   version=67 self=enode://14bea92b12455ae7592b17019ab8cd36d936836330d2aa2f121a29676edd2d5ed791dcd9f10a7e1fbb99e5a9943901ed13f8cbf51ecd3ab079c5538f009047f2@127.0.0.1:30304 name=erigon/v3.01.0-dev-32b4c61c/windows-amd64/go1.24.0
INFO[02-28|18:34:15.447] Started P2P networking                   version=68 self=enode://14bea92b12455ae7592b17019ab8cd36d936836330d2aa2f121a29676edd2d5ed791dcd9f10a7e1fbb99e5a9943901ed13f8cbf51ecd3ab079c5538f009047f2@127.0.0.1:30303 name=erigon/v3.01.0-dev-32b4c61c/windows-amd64/go1.24.0
INFO[02-28|18:34:15.449] [txpool] Started
INFO[02-28|18:34:15.450] [OtterSync] Starting Ottersync
INFO[02-28|18:34:15.451]
   _____ _             _   _                ____  _   _
  / ____| |           | | (_)              / __ \| | | |
 | (___ | |_ __ _ _ __| |_ _ _ __   __ _  | |  | | |_| |_ ___ _ __ ___ _   _ _ __   ___
  \___ \| __/ _ | '__| __| | '_ \ / _ | | |  | | __| __/ _ \ '__/ __| | | | '_ \ / __|
  ____) | || (_| | |  | |_| | | | | (_| | | |__| | |_| ||  __/ |  \__ \ |_| | | | | (__ _ _ _
 |_____/ \__\__,_|_|   \__|_|_| |_|\__, |  \____/ \__|\__\___|_|  |___/\__, |_| |_|\___(_|_|_)
                                    __/ |                               __/ |
                                   |___/                               |___/


                                        .:-===++**++===-:
                                   :=##%@@@@@@@@@@@@@@@@@@%#*=.
                               .=#@@@@@@%##+====--====+##@@@@@@@#=.     ...
                   .=**###*=:+#@@@@%*=:.                  .:=#%@@@@#==#@@@@@%#-
                 -#@@@@%%@@@@@@%+-.                            .=*%@@@@#*+*#@@@%=
                =@@@*:    -%%+:                                    -#@+.     =@@@-
                %@@#     +@#.                                        :%%-     %@@*
                @@@+    +%=.     -+=                        :=-       .#@-    %@@#
                *@@%:  #@-      =@@@*                      +@@@%.       =@= -*@@@:
                 #@@@##@+       #@@@@.                     %@@@@=        #@%@@@#-
                  :#@@@@:       +@@@#       :=++++==-.     *@@@@:        =@@@@-
                  =%@@%=         +#*.    =#%#+==-==+#%%=:  .+#*:         .#@@@#.
                 +@@%+.               .+%+-.          :=##-                :#@@@-
                -@@@=                -%#:     ..::.      +@*                 +@@%.
    .::-========*@@@..              -@#      +%@@@@%.     -@#               .-@@@+=======-
.:-====----:::::#@@%:--=::::..      #@:      *@@@@@%:      *@=      ..:-:-=--:@@@+::::----
                =@@@:.......        @@        :+@#=.       -@+        .......-@@@:
       .:=++####*%@@%=--::::..      @@   %#     %*    :@*  -@+      ...::---+@@@#*#*##+=-:
  ..--==::..     :%@@@-   ..:::..   @@   +@*:.-#@@+-.-#@-  -@+   ..:::..  .+@@@#.     ..:-
                  .#@@@##-:.        @@    :+#@%=.:+@@#=.   -@+        .-=#@@@@+
             -=+++=--+%@@%+=.       @@       +%*=+#%-      -@+       :=#@@@%+--++++=:
         .=**=:.      .=*@@@@@#=:.  @@         :--.        -@+  .-+#@@@@%+:       .:=*+-.
        ::.              .=*@@@@@@%#@@+=-:..         ..::=+#@%#@@@@@@%+-.             ..-.
                            ..=*#@@@@@@@@@@@@@@@%%@@@@@@@@@@@@@@%#+-.
                                  .:-==++*#######%######**+==-:



INFO[02-28|18:34:15.451] [1/6 OtterSync] Requesting downloads
INFO[02-28|18:34:15.610] [snapshots] no metadata yet              files=169 list=v1-000000-000100-bodies.seg,v1-000100-000200-headers.seg,v1-000400-000500-headers.seg,v1-000900-001000-bodies.idx,v1-003340-003350-headers.seg,...
INFO[02-28|18:34:28.908] [Checkpoint Sync] Requesting beacon state uri=https://checkpoint-sync.holesky.ethpandaops.io/eth/v2/debug/beacon/states/finalized
INFO[02-28|18:34:30.879] [1/6 OtterSync] Downloading              progress="(0/169 files) 0.00% - 0B/0B" time-left=999hrs:99m total-time=15s download-rate=0B/s completion-rate=0B/s alloc=866.1MB sys=1.3GB
INFO[02-28|18:34:30.881] [snapshots] no metadata yet              files=169 list=v1-001200-001300-headers.idx,v1-001200-001300-headers.seg,v1-001600-001700-bodies.seg,v1-002800-002900-bodies.idx,v1-003360-003370-headers.seg,...
INFO[02-28|18:34:38.052] Starting caplin
INFO[02-28|18:34:39.238] Static peers                             len=0
INFO[02-28|18:34:39.239] [Sentinel] Sentinel started              app=caplin enr=enr:-LS4QFuZTHP4pLJeZjHhAAutkSrs4nu8es3VPQlNmFggJZ5ILuDEY7IKQfrxaQ_GlAIWwOOAMunnAEIio5kmJGF2NGMBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpABniGtBgFwAAAAAAAAAAAAgmlkgnY0iXNlY3AyNTZrMaEDO2VHDoanuyPRBCWY5JR9QsjBrQXGl2AwqrujU6CbbNCIc3luY25ldHMAg3RjcIIPoYN1ZHCCD6A
INFO[02-28|18:34:39.240] Started Ethereum 2.0 Gossip Service      app=caplin
INFO[02-28|18:34:39.242] [Caplin] starting clstages loop          app=caplin
INFO[02-28|18:34:39.242] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=3710944
INFO[02-28|18:34:40.238] [Sentinel] Update ENR on subscription    subnet=5 subscribe=true type=attestation
INFO[02-28|18:34:40.239] [Sentinel] Update ENR on subscription    subnet=24 subscribe=true type=attestation
INFO[02-28|18:34:50.883] [snapshots] no progress yet              files=47 list=v1-002600-002700-bodies.seg,v1-003330-003340-bodies.seg,v1-003000-003100-bodies.idx,v1-003300-003310-headers.seg,v1-003310-003320-headers.idx,...
INFO[02-28|18:34:50.884] [1/6 OtterSync] Downloading              progress="(169/169 files) 6.51% - 148.1MB/2.2GB" time-left=999hrs:99m total-time=35s download-rate=0B/s completion-rate=0B/s alloc=3.2GB sys=3.6GB
INFO[02-28|18:35:10.882] [1/6 OtterSync] Downloading              progress="(169/169 files) 85.82% - 1.9GB/2.2GB" time-left=0hrs:0m total-time=55s download-rate=90.2MB/s completion-rate=90.0MB/s alloc=3.5GB sys=3.7GB
INFO[02-28|18:35:30.880] [1/6 OtterSync] Downloading complete     time=1m15.4292262s
INFO[02-28|18:35:35.972] [1/6 OtterSync] Requesting downloads
INFO[02-28|18:35:36.177] [snapshots] no metadata yet              files=171 list=v1-001600-001700-transactions.idx,v1-002400-002500-transactions-to-block.idx,v1-002800-002900-transactions.seg,domain/v1-code.64-72.bt,v1-003320-003330-transactions-to-block.idx,...
INFO[02-28|18:35:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:35:50.879] [1/6 OtterSync] Downloading              progress="(169/340 files) 100.00% - 2.2GB/2.2GB" time-left=0hrs:0m total-time=1m35s download-rate=8.1MB/s completion-rate=8.2MB/s alloc=3.7GB sys=3.8GB
INFO[02-28|18:35:50.882] [snapshots] no metadata yet              files=171 list=v1-003320-003330-transactions-to-block.idx,domain/v1-code.76-77.kvei,v1-000100-000200-transactions.idx,v1-000300-000400-transactions.seg,v1-001200-001300-transactions-to-block.idx,...
INFO[02-28|18:36:10.879] [1/6 OtterSync] Downloading              progress="(169/340 files) 100.00% - 2.2GB/2.2GB" time-left=0hrs:0m total-time=1m55s download-rate=4.0MB/s completion-rate=4.1MB/s alloc=3.8GB sys=3.9GB
INFO[02-28|18:36:10.888] [snapshots] no progress yet              files=171 list=v1-000300-000400-transactions.idx,v1-001300-001400-transactions-to-block.idx,v1-002500-002600-transactions-to-block.idx,v1-003200-003300-transactions.idx,v1-001800-001900-transactions.idx,...
INFO[02-28|18:36:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 1.82% - 2.2GB/121.8GB" time-left=16hrs:52m total-time=2m15s download-rate=2.0MB/s completion-rate=2.0MB/s alloc=3.8GB sys=3.9GB
INFO[02-28|18:36:30.888] [snapshots] no progress yet              files=171 list=domain/v1-storage.0-64.kvei,v1-000700-000800-transactions.seg,v1-002700-002800-transactions.idx,v1-001600-001700-transactions-to-block.idx,v1-001700-001800-transactions.idx,...
INFO[02-28|18:36:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:36:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 1.82% - 2.2GB/121.8GB" time-left=33hrs:44m total-time=2m35s download-rate=1.0MB/s completion-rate=1.0MB/s alloc=3.4GB sys=4.2GB
INFO[02-28|18:36:50.888] [snapshots] no progress yet              files=171 list=v1-001100-001200-transactions-to-block.idx,v1-003000-003100-transactions-to-block.idx,v1-003350-003360-transactions.idx,domain/v1-commitment.0-64.kv,v1-000400-000500-transactions.seg,...
INFO[02-28|18:37:10.880] [p2p] GoodPeers                          eth68=1
INFO[02-28|18:37:10.880] [mem] memory stats                       alloc=3.4GB sys=4.2GB
INFO[02-28|18:37:10.887] [snapshots] no progress yet              files=171 list=v1-000200-000300-transactions-to-block.idx,v1-001400-001500-transactions.idx,v1-003320-003330-transactions.seg,domain/v1-commitment.64-72.kvi,v1-000200-000300-transactions.idx,...
INFO[02-28|18:37:10.888] [1/6 OtterSync] Downloading              progress="(340/340 files) 1.82% - 2.2GB/121.8GB" time-left=134hrs:59m total-time=2m55s download-rate=258.1KB/s completion-rate=261.3KB/s alloc=3.4GB sys=4.2GB
INFO[02-28|18:37:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:37:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 1.82% - 2.2GB/121.8GB" time-left=134hrs:59m total-time=3m15s download-rate=258.1KB/s completion-rate=261.3KB/s alloc=6.7GB sys=7.1GB
INFO[02-28|18:37:30.890] [snapshots] no progress yet              files=113 list=v1-002800-002900-transactions.idx,domain/v1-accounts.0-64.bt,v1-001800-001900-transactions-to-block.idx,domain/v1-accounts.76-77.bt,domain/v1-accounts.72-76.bt,...
INFO[02-28|18:37:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:37:50.896] [snapshots] no progress yet              files=46 list=v1-003360-003370-transactions.seg,domain/v1-accounts.72-76.kv,v1-001900-002000-transactions.idx,v1-000900-001000-transactions.idx,domain/v1-accounts.0-64.kvei,...
INFO[02-28|18:37:50.897] [1/6 OtterSync] Downloading              progress="(340/340 files) 3.73% - 4.5GB/121.8GB" time-left=0hrs:22m total-time=3m35s download-rate=90.7MB/s completion-rate=64.0MB/s alloc=6.9GB sys=7.2GB
INFO[02-28|18:38:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 3.73% - 4.5GB/121.8GB" time-left=0hrs:22m total-time=3m55s download-rate=90.7MB/s completion-rate=64.0MB/s alloc=7.2GB sys=7.5GB
INFO[02-28|18:38:10.895] [snapshots] no progress yet              files=14 list=v1-001700-001800-transactions.seg,v1-001500-001600-transactions.idx,domain/v1-accounts.0-64.bt,domain/v1-accounts.72-76.bt,v1-002500-002600-transactions-to-block.idx,...
INFO[02-28|18:38:30.891] [snapshots] no progress yet              files=11 list=v1-001000-001100-transactions.idx,v1-001100-001200-transactions-to-block.idx,v1-003350-003360-transactions.idx,v1-003100-003200-transactions.idx,v1-000700-000800-transactions-to-block.idx,...
INFO[02-28|18:38:30.892] [1/6 OtterSync] Downloading              progress="(340/340 files) 6.50% - 7.9GB/121.8GB" time-left=0hrs:22m total-time=4m15s download-rate=84.9MB/s completion-rate=95.0MB/s alloc=7.6GB sys=7.9GB
INFO[02-28|18:38:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:38:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 6.50% - 7.9GB/121.8GB" time-left=0hrs:22m total-time=4m35s download-rate=84.9MB/s completion-rate=95.0MB/s alloc=7.8GB sys=8.2GB
INFO[02-28|18:38:50.892] [snapshots] no progress yet              files=1 list=v1-001700-001800-transactions.seg
INFO[02-28|18:39:10.891] [1/6 OtterSync] Downloading              progress="(340/340 files) 9.23% - 11.2GB/121.8GB" time-left=0hrs:22m total-time=4m55s download-rate=82.9MB/s completion-rate=82.7MB/s alloc=8.1GB sys=8.4GB
INFO[02-28|18:39:30.882] [1/6 OtterSync] Downloading              progress="(340/340 files) 9.23% - 11.2GB/121.8GB" time-left=0hrs:22m total-time=5m15s download-rate=82.9MB/s completion-rate=82.7MB/s alloc=6.7GB sys=8.7GB
INFO[02-28|18:39:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:39:50.893] [1/6 OtterSync] Downloading              progress="(340/340 files) 11.93% - 14.5GB/121.8GB" time-left=0hrs:22m total-time=5m35s download-rate=82.0MB/s completion-rate=82.1MB/s alloc=7.0GB sys=8.7GB
INFO[02-28|18:40:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 11.93% - 14.5GB/121.8GB" time-left=0hrs:22m total-time=5m55s download-rate=82.0MB/s completion-rate=82.1MB/s alloc=7.2GB sys=8.7GB
INFO[02-28|18:40:10.880] [p2p] GoodPeers                          eth67=1 eth68=3
INFO[02-28|18:40:10.880] [mem] memory stats                       alloc=7.2GB sys=8.7GB
INFO[02-28|18:40:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:40:30.890] [1/6 OtterSync] Downloading              progress="(340/340 files) 14.58% - 17.8GB/121.8GB" time-left=0hrs:22m total-time=6m15s download-rate=79.2MB/s completion-rate=79.5MB/s alloc=7.5GB sys=8.7GB
INFO[02-28|18:40:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:40:50.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 14.58% - 17.8GB/121.8GB" time-left=0hrs:22m total-time=6m35s download-rate=79.2MB/s completion-rate=79.5MB/s alloc=7.7GB sys=8.7GB
INFO[02-28|18:41:10.888] [1/6 OtterSync] Downloading              progress="(340/340 files) 17.12% - 20.9GB/121.8GB" time-left=0hrs:22m total-time=6m55s download-rate=78.3MB/s completion-rate=78.5MB/s alloc=7.9GB sys=8.7GB
INFO[02-28|18:41:30.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 17.12% - 20.9GB/121.8GB" time-left=0hrs:22m total-time=7m15s download-rate=78.3MB/s completion-rate=78.5MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:41:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:41:50.890] [1/6 OtterSync] Downloading              progress="(340/340 files) 19.61% - 23.9GB/121.8GB" time-left=0hrs:21m total-time=7m35s download-rate=77.9MB/s completion-rate=77.7MB/s alloc=6.9GB sys=8.7GB
INFO[02-28|18:42:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 19.61% - 23.9GB/121.8GB" time-left=0hrs:21m total-time=7m55s download-rate=77.9MB/s completion-rate=77.7MB/s alloc=7.2GB sys=8.7GB
INFO[02-28|18:42:30.888] [1/6 OtterSync] Downloading              progress="(340/340 files) 22.21% - 27.1GB/121.8GB" time-left=0hrs:20m total-time=8m15s download-rate=80.7MB/s completion-rate=80.7MB/s alloc=7.4GB sys=8.7GB
INFO[02-28|18:42:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:42:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 22.21% - 27.1GB/121.8GB" time-left=0hrs:20m total-time=8m35s download-rate=80.7MB/s completion-rate=80.7MB/s alloc=7.7GB sys=8.7GB
INFO[02-28|18:43:10.880] [p2p] GoodPeers                          eth67=1 eth68=3
INFO[02-28|18:43:10.880] [mem] memory stats                       alloc=7.9GB sys=8.7GB
INFO[02-28|18:43:10.889] [1/6 OtterSync] Downloading              progress="(340/340 files) 24.73% - 30.1GB/121.8GB" time-left=0hrs:20m total-time=8m55s download-rate=77.9MB/s completion-rate=77.9MB/s alloc=7.9GB sys=8.7GB
INFO[02-28|18:43:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:43:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 24.73% - 30.1GB/121.8GB" time-left=0hrs:20m total-time=9m15s download-rate=77.9MB/s completion-rate=77.9MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:43:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:43:50.887] [1/6 OtterSync] Downloading              progress="(340/340 files) 27.22% - 33.2GB/121.8GB" time-left=0hrs:19m total-time=9m35s download-rate=76.3MB/s completion-rate=76.3MB/s alloc=6.8GB sys=8.7GB
INFO[02-28|18:44:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 27.22% - 33.2GB/121.8GB" time-left=0hrs:19m total-time=9m55s download-rate=76.3MB/s completion-rate=76.3MB/s alloc=7.1GB sys=8.7GB
INFO[02-28|18:44:30.886] [1/6 OtterSync] Downloading              progress="(340/340 files) 29.59% - 36.0GB/121.8GB" time-left=0hrs:20m total-time=10m15s download-rate=72.9MB/s completion-rate=72.8MB/s alloc=7.3GB sys=8.7GB
INFO[02-28|18:44:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:44:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 29.59% - 36.0GB/121.8GB" time-left=0hrs:20m total-time=10m35s download-rate=72.9MB/s completion-rate=72.8MB/s alloc=7.5GB sys=8.7GB
INFO[02-28|18:45:10.887] [1/6 OtterSync] Downloading              progress="(340/340 files) 31.89% - 38.8GB/121.8GB" time-left=0hrs:19m total-time=10m55s download-rate=70.9MB/s completion-rate=70.9MB/s alloc=7.8GB sys=8.7GB
INFO[02-28|18:45:30.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 31.89% - 38.8GB/121.8GB" time-left=0hrs:19m total-time=11m15s download-rate=70.9MB/s completion-rate=70.9MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:45:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:45:50.887] [1/6 OtterSync] Downloading              progress="(340/340 files) 34.15% - 41.6GB/121.8GB" time-left=0hrs:19m total-time=11m35s download-rate=71.5MB/s completion-rate=71.5MB/s alloc=6.8GB sys=8.7GB
INFO[02-28|18:46:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 34.15% - 41.6GB/121.8GB" time-left=0hrs:19m total-time=11m55s download-rate=71.5MB/s completion-rate=71.5MB/s alloc=7.1GB sys=8.7GB
INFO[02-28|18:46:10.880] [mem] memory stats                       alloc=7.1GB sys=8.7GB
INFO[02-28|18:46:10.880] [p2p] GoodPeers                          eth68=2 eth67=2
INFO[02-28|18:46:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:46:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 35.31% - 43.0GB/121.8GB" time-left=0hrs:18m total-time=12m15s download-rate=72.4MB/s completion-rate=72.3MB/s alloc=7.3GB sys=8.7GB
INFO[02-28|18:46:39.241] P2P                                      app=caplin peers=2
INFO[02-28|18:46:50.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 36.48% - 44.4GB/121.8GB" time-left=0hrs:18m total-time=12m35s download-rate=72.5MB/s completion-rate=72.6MB/s alloc=7.6GB sys=8.7GB
INFO[02-28|18:47:10.886] [1/6 OtterSync] Downloading              progress="(340/340 files) 38.79% - 47.3GB/121.8GB" time-left=0hrs:17m total-time=12m55s download-rate=74.1MB/s completion-rate=74.1MB/s alloc=7.8GB sys=8.7GB
INFO[02-28|18:47:30.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 38.79% - 47.3GB/121.8GB" time-left=0hrs:17m total-time=13m15s download-rate=74.1MB/s completion-rate=74.1MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:47:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:47:50.886] [1/6 OtterSync] Downloading              progress="(340/340 files) 41.14% - 50.1GB/121.8GB" time-left=0hrs:16m total-time=13m35s download-rate=74.9MB/s completion-rate=75.0MB/s alloc=6.8GB sys=8.7GB
INFO[02-28|18:48:10.888] [1/6 OtterSync] Downloading              progress="(340/340 files) 42.31% - 51.5GB/121.8GB" time-left=0hrs:16m total-time=13m55s download-rate=72.7MB/s completion-rate=72.6MB/s alloc=7.1GB sys=8.7GB
INFO[02-28|18:48:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 42.31% - 51.5GB/121.8GB" time-left=0hrs:16m total-time=14m15s download-rate=72.7MB/s completion-rate=72.6MB/s alloc=7.3GB sys=8.7GB
INFO[02-28|18:48:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:48:50.887] [1/6 OtterSync] Downloading              progress="(340/340 files) 44.66% - 54.4GB/121.8GB" time-left=0hrs:15m total-time=14m35s download-rate=72.9MB/s completion-rate=73.1MB/s alloc=7.6GB sys=8.7GB
INFO[02-28|18:49:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 44.66% - 54.4GB/121.8GB" time-left=0hrs:15m total-time=14m55s download-rate=72.9MB/s completion-rate=73.1MB/s alloc=8.0GB sys=8.7GB
INFO[02-28|18:49:10.880] [p2p] GoodPeers                          eth68=2 eth67=2
INFO[02-28|18:49:10.880] [mem] memory stats                       alloc=8.0GB sys=8.7GB
INFO[02-28|18:49:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:49:30.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 47.03% - 57.3GB/121.8GB" time-left=0hrs:14m total-time=15m15s download-rate=75.4MB/s completion-rate=75.3MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:49:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:49:50.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 47.03% - 57.3GB/121.8GB" time-left=0hrs:14m total-time=15m35s download-rate=75.4MB/s completion-rate=75.3MB/s alloc=6.9GB sys=8.7GB
INFO[02-28|18:50:10.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 49.38% - 60.2GB/121.8GB" time-left=0hrs:14m total-time=15m55s download-rate=73.2MB/s completion-rate=73.2MB/s alloc=7.2GB sys=8.7GB
INFO[02-28|18:50:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 49.38% - 60.2GB/121.8GB" time-left=0hrs:14m total-time=16m15s download-rate=73.2MB/s completion-rate=73.2MB/s alloc=7.5GB sys=8.7GB
INFO[02-28|18:50:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:50:50.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 51.70% - 63.0GB/121.8GB" time-left=0hrs:14m total-time=16m35s download-rate=71.5MB/s completion-rate=71.5MB/s alloc=7.8GB sys=8.7GB
INFO[02-28|18:51:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 51.70% - 63.0GB/121.8GB" time-left=0hrs:14m total-time=16m55s download-rate=71.5MB/s completion-rate=71.5MB/s alloc=8.1GB sys=8.7GB
INFO[02-28|18:51:30.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 53.98% - 65.8GB/121.8GB" time-left=0hrs:13m total-time=17m15s download-rate=71.8MB/s completion-rate=71.7MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:51:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:51:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 53.98% - 65.8GB/121.8GB" time-left=0hrs:13m total-time=17m35s download-rate=71.8MB/s completion-rate=71.7MB/s alloc=6.9GB sys=8.7GB
INFO[02-28|18:52:10.880] [mem] memory stats                       alloc=7.2GB sys=8.7GB
INFO[02-28|18:52:10.880] [p2p] GoodPeers                          eth68=1 eth67=2
INFO[02-28|18:52:10.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 56.33% - 68.6GB/121.8GB" time-left=0hrs:12m total-time=17m55s download-rate=74.9MB/s completion-rate=74.8MB/s alloc=7.2GB sys=8.7GB
INFO[02-28|18:52:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:52:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 56.33% - 68.6GB/121.8GB" time-left=0hrs:12m total-time=18m15s download-rate=74.9MB/s completion-rate=74.8MB/s alloc=7.5GB sys=8.7GB
INFO[02-28|18:52:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:52:50.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 58.58% - 71.4GB/121.8GB" time-left=0hrs:12m total-time=18m35s download-rate=71.2MB/s completion-rate=71.2MB/s alloc=7.8GB sys=8.7GB
INFO[02-28|18:53:10.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 58.58% - 71.4GB/121.8GB" time-left=0hrs:12m total-time=18m55s download-rate=71.2MB/s completion-rate=71.2MB/s alloc=8.1GB sys=8.7GB
INFO[02-28|18:53:30.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 60.88% - 74.2GB/121.8GB" time-left=0hrs:11m total-time=19m15s download-rate=71.3MB/s completion-rate=71.1MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:53:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:53:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 60.88% - 74.2GB/121.8GB" time-left=0hrs:11m total-time=19m35s download-rate=71.3MB/s completion-rate=71.1MB/s alloc=6.9GB sys=8.7GB
INFO[02-28|18:54:10.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 63.22% - 77.0GB/121.8GB" time-left=0hrs:10m total-time=19m55s download-rate=71.0MB/s completion-rate=71.1MB/s alloc=7.2GB sys=8.7GB
INFO[02-28|18:54:30.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 63.22% - 77.0GB/121.8GB" time-left=0hrs:10m total-time=20m15s download-rate=71.0MB/s completion-rate=71.1MB/s alloc=7.5GB sys=8.7GB
INFO[02-28|18:54:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:54:50.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 65.54% - 79.8GB/121.8GB" time-left=0hrs:10m total-time=20m35s download-rate=70.4MB/s completion-rate=70.3MB/s alloc=7.8GB sys=8.7GB
INFO[02-28|18:55:10.881] [p2p] GoodPeers                          eth68=1 eth67=2
INFO[02-28|18:55:10.881] [mem] memory stats                       alloc=8.1GB sys=8.7GB
INFO[02-28|18:55:10.887] [1/6 OtterSync] Downloading              progress="(340/340 files) 66.72% - 81.3GB/121.8GB" time-left=0hrs:9m total-time=20m55s download-rate=73.4MB/s completion-rate=73.4MB/s alloc=8.1GB sys=8.7GB
INFO[02-28|18:55:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:55:30.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 66.72% - 81.3GB/121.8GB" time-left=0hrs:9m total-time=21m15s download-rate=73.4MB/s completion-rate=73.4MB/s alloc=6.3GB sys=8.7GB
INFO[02-28|18:55:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:55:50.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 69.12% - 84.2GB/121.8GB" time-left=0hrs:8m total-time=21m35s download-rate=77.3MB/s completion-rate=77.3MB/s alloc=6.6GB sys=8.7GB
INFO[02-28|18:56:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 69.12% - 84.2GB/121.8GB" time-left=0hrs:8m total-time=21m55s download-rate=77.3MB/s completion-rate=77.3MB/s alloc=7.0GB sys=8.7GB
INFO[02-28|18:56:30.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 71.59% - 87.2GB/121.8GB" time-left=0hrs:7m total-time=22m15s download-rate=76.9MB/s completion-rate=76.8MB/s alloc=7.3GB sys=8.7GB
INFO[02-28|18:56:39.241] P2P                                      app=caplin peers=1
INFO[02-28|18:56:50.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 71.59% - 87.2GB/121.8GB" time-left=0hrs:7m total-time=22m35s download-rate=76.9MB/s completion-rate=76.8MB/s alloc=7.6GB sys=8.7GB
INFO[02-28|18:57:10.884] [1/6 OtterSync] Downloading              progress="(340/340 files) 74.07% - 90.2GB/121.8GB" time-left=0hrs:7m total-time=22m55s download-rate=76.8MB/s completion-rate=76.9MB/s alloc=8.0GB sys=8.7GB
INFO[02-28|18:57:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 74.07% - 90.2GB/121.8GB" time-left=0hrs:7m total-time=23m15s download-rate=76.8MB/s completion-rate=76.9MB/s alloc=5.9GB sys=8.7GB
INFO[02-28|18:57:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:57:50.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 76.46% - 93.1GB/121.8GB" time-left=0hrs:6m total-time=23m35s download-rate=74.2MB/s completion-rate=74.0MB/s alloc=6.3GB sys=8.7GB
INFO[02-28|18:58:10.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 76.46% - 93.1GB/121.8GB" time-left=0hrs:6m total-time=23m55s download-rate=74.2MB/s completion-rate=74.0MB/s alloc=6.7GB sys=8.7GB
INFO[02-28|18:58:10.880] [p2p] GoodPeers                          eth68=1 eth67=3
INFO[02-28|18:58:10.880] [mem] memory stats                       alloc=6.7GB sys=8.7GB
INFO[02-28|18:58:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|18:58:30.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 77.74% - 94.7GB/121.8GB" time-left=0hrs:5m total-time=24m15s download-rate=79.7MB/s completion-rate=79.9MB/s alloc=7.0GB sys=8.7GB
INFO[02-28|18:58:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:58:50.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 80.41% - 98.0GB/121.8GB" time-left=0hrs:4m total-time=24m35s download-rate=85.1MB/s completion-rate=85.1MB/s alloc=7.4GB sys=8.7GB
INFO[02-28|18:59:10.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 80.41% - 98.0GB/121.8GB" time-left=0hrs:4m total-time=24m55s download-rate=85.1MB/s completion-rate=85.1MB/s alloc=7.9GB sys=8.7GB
INFO[02-28|18:59:30.885] [1/6 OtterSync] Downloading              progress="(340/340 files) 83.18% - 101.3GB/121.8GB" time-left=0hrs:4m total-time=25m15s download-rate=87.3MB/s completion-rate=87.4MB/s alloc=4.7GB sys=8.7GB
INFO[02-28|18:59:39.241] P2P                                      app=caplin peers=0
INFO[02-28|18:59:50.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 83.18% - 101.3GB/121.8GB" time-left=0hrs:4m total-time=25m35s download-rate=87.3MB/s completion-rate=87.4MB/s alloc=5.1GB sys=8.7GB
INFO[02-28|19:00:10.883] [1/6 OtterSync] Downloading              progress="(340/340 files) 85.94% - 104.7GB/121.8GB" time-left=0hrs:3m total-time=25m55s download-rate=86.6MB/s completion-rate=86.4MB/s alloc=5.5GB sys=8.7GB
INFO[02-28|19:00:30.879] [1/6 OtterSync] Downloading              progress="(340/340 files) 85.94% - 104.7GB/121.8GB" time-left=0hrs:3m total-time=26m15s download-rate=86.6MB/s completion-rate=86.4MB/s alloc=6.0GB sys=8.7GB
INFO[02-28|19:00:39.241] P2P                                      app=caplin peers=0
INFO[02-28|19:00:50.882] [1/6 OtterSync] Downloading              progress="(340/340 files) 88.68% - 108.0GB/121.8GB" time-left=0hrs:2m total-time=26m35s download-rate=87.2MB/s completion-rate=87.0MB/s alloc=6.4GB sys=8.7GB
INFO[02-28|19:01:10.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 88.68% - 108.0GB/121.8GB" time-left=0hrs:2m total-time=26m55s download-rate=87.2MB/s completion-rate=87.0MB/s alloc=6.9GB sys=8.7GB
INFO[02-28|19:01:10.881] [mem] memory stats                       alloc=6.9GB sys=8.7GB
INFO[02-28|19:01:10.881] [p2p] GoodPeers                          eth68=1 eth67=3
INFO[02-28|19:01:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|19:01:30.882] [1/6 OtterSync] Downloading              progress="(340/340 files) 91.41% - 111.4GB/121.8GB" time-left=0hrs:2m total-time=27m15s download-rate=86.6MB/s completion-rate=86.8MB/s alloc=3.6GB sys=8.8GB
INFO[02-28|19:01:39.241] P2P                                      app=caplin peers=0
INFO[02-28|19:01:50.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 91.41% - 111.4GB/121.8GB" time-left=0hrs:2m total-time=27m35s download-rate=86.6MB/s completion-rate=86.8MB/s alloc=4.0GB sys=8.8GB
INFO[02-28|19:02:10.885] [1/6 OtterSync] Downloading              progress="(340/340 files) 94.19% - 114.7GB/121.8GB" time-left=0hrs:1m total-time=27m55s download-rate=87.0MB/s completion-rate=86.8MB/s alloc=4.5GB sys=8.8GB
INFO[02-28|19:02:30.882] [1/6 OtterSync] Downloading              progress="(340/340 files) 95.58% - 116.4GB/121.8GB" time-left=0hrs:1m total-time=28m15s download-rate=87.1MB/s completion-rate=87.3MB/s alloc=5.0GB sys=8.8GB
INFO[02-28|19:02:39.241] P2P                                      app=caplin peers=0
INFO[02-28|19:02:50.881] [1/6 OtterSync] Downloading              progress="(340/340 files) 97.02% - 118.2GB/121.8GB" time-left=0hrs:0m total-time=28m35s download-rate=89.5MB/s completion-rate=89.5MB/s alloc=5.7GB sys=8.8GB
INFO[02-28|19:03:10.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 97.02% - 118.2GB/121.8GB" time-left=0hrs:0m total-time=28m55s download-rate=89.5MB/s completion-rate=89.5MB/s alloc=6.0GB sys=8.8GB
INFO[02-28|19:03:30.882] [1/6 OtterSync] Downloading              progress="(340/340 files) 99.65% - 121.4GB/121.8GB" time-left=0hrs:0m total-time=29m15s download-rate=76.8MB/s completion-rate=55.7MB/s alloc=3.1GB sys=8.8GB
INFO[02-28|19:03:39.241] P2P                                      app=caplin peers=0
INFO[02-28|19:03:50.880] [1/6 OtterSync] Downloading              progress="(340/340 files) 99.65% - 121.4GB/121.8GB" time-left=0hrs:0m total-time=29m35s download-rate=76.8MB/s completion-rate=55.7MB/s alloc=3.2GB sys=8.8GB
INFO[02-28|19:04:10.880] [p2p] GoodPeers                          eth67=3 eth68=2
INFO[02-28|19:04:10.880] [mem] memory stats                       alloc=5.9GB sys=9.1GB
INFO[02-28|19:04:15.449] [txpool] stat                            pending=0 baseFee=0 queued=0
INFO[02-28|19:04:27.802] [1/6 OtterSync] MaxTxNums index: 3.10M/3.38M
INFO[02-28|19:04:29.114] [snapshots:download] Stat                blocks=3.38M indices=3.38M alloc=3.2GB sys=9.2GB
INFO[02-28|19:04:29.121] [snapshots:history] Stat                 blocks=3.36M txs=120.31M txNum2blockNum="64=2557K,72=2966K,76=3266K,77=3363K" first_history_idx_in_db=0 last_comitment_block=3363531 last_comitment_tx_num=120312500 alloc=3.2GB sys=9.2GB
INFO[02-28|19:04:29.169] [1/6 OtterSync] DONE                     in=30m13.720594s block=3380999
INFO[02-28|19:04:29.175] [4/6 Execution] starting                 from=3363531 to=3380999 fromTxNum=120312499 offsetFromBlockBeginning=0 initialCycle=true useExternalTx=false inMem=false
INFO[02-28|19:04:29.179] [4/6 Execution] starting                 from=3363531 to=3380999 fromTxNum=120312492 offsetFromBlockBeginning=7 initialCycle=true useExternalTx=false
INFO[02-28|19:04:39.241] P2P                                      app=caplin peers=1
INFO[02-28|19:04:49.200] [4/6 Execution]                          blk=3364339 blks=809 blk/s=40.4 txs=16654 tx/s=831 gas/s=652.07M buf=161.2MB/512.0MB stepsInDB=0.00 step=77.0 inMem=false alloc=3.3GB sys=9.3GB
INFO[02-28|19:05:09.204] [4/6 Execution][agg] computing trie      progress=34.28k/134.30k alloc=3.2GB sys=9.3GB
INFO[02-28|19:05:29.204] [4/6 Execution][agg] computing trie      progress=72.79k/134.30k alloc=5.5GB sys=9.3GB
INFO[02-28|19:05:39.241] P2P                                      app=caplin peers=2
INFO[02-28|19:05:49.204] [4/6 Execution][agg] computing trie      progress=128.27k/134.30k alloc=5.6GB sys=9.5GB
INFO[02-28|19:05:53.512] Committed                                time=1m4.3084206s block=3364339 txNum=120329154 step=77.0 flush+commitment=1m3.7556567s tx.commit=465.4208ms prune=86.3223ms
INFO[02-28|19:05:53.610] [4/6 Execution]                          blk=3364340 blks=2 blk/s=0.0 txs=15 tx/s=0 gas/s=394.23k buf=389.0KB/512.0MB stepsInDB=0.01 step=77.0 inMem=false alloc=3.2GB sys=9.5GB
INFO[02-28|19:06:09.191] [4/6 Execution]                          blk=3365035 blks=696 blk/s=44.7 txs=14064 tx/s=902 gas/s=719.13M buf=140.0MB/512.0MB stepsInDB=0.01 step=77.0 inMem=false alloc=4.7GB sys=9.5GB
INFO[02-28|19:06:29.209] [4/6 Execution]                          blk=3365962 blks=928 blk/s=46.4 txs=17674 tx/s=882 gas/s=730.93M buf=319.7MB/512.0MB stepsInDB=0.01 step=77.0 inMem=false alloc=6.5GB sys=9.5GB
INFO[02-28|19:06:39.241] P2P                                      app=caplin peers=1
INFO[02-28|19:06:49.179] [4/6 Execution]                          blk=3366893 blks=932 blk/s=46.7 txs=17367 tx/s=869 gas/s=740.43M buf=501.4MB/512.0MB stepsInDB=0.01 step=77.0 inMem=false alloc=5.9GB sys=9.7GB
INFO[02-28|19:07:09.205] [4/6 Execution]                          blk=3367837 blks=945 blk/s=47.2 txs=17172 tx/s=857 gas/s=846.76M buf=713.7MB/512.0MB stepsInDB=0.01 step=77.1 inMem=false alloc=7.2GB sys=9.7GB
INFO[02-28|19:07:10.881] [mem] memory stats                       alloc=7.3GB sys=9.7GB
INFO[02-28|19:07:10.881] [p2p] GoodPeers                          eth68=1 eth67=3
INFO[02-28|19:07:15.449] [txpool] stat                            pending=0 baseFee=0 queued=57
INFO[02-28|19:07:29.207] [4/6 Execution][agg] computing trie      progress=210.34k/663.55k alloc=6.1GB sys=10.0GB
INFO[02-28|19:07:39.241] P2P                                      app=caplin peers=1
INFO[02-28|19:07:49.207] [4/6 Execution][agg] computing trie      progress=289.93k/663.55k alloc=4.8GB sys=10.0GB
INFO[02-28|19:07:56.001] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-1167242259
INFO[02-28|19:08:09.207] [4/6 Execution][agg] computing trie      progress=374.85k/663.55k alloc=7.5GB sys=10.0GB
INFO[02-28|19:08:29.208] [4/6 Execution][agg] computing trie      progress=466.93k/663.55k alloc=6.3GB sys=10.0GB
INFO[02-28|19:08:39.241] P2P                                      app=caplin peers=1
INFO[02-28|19:08:49.207] [4/6 Execution][agg] computing trie      progress=592.77k/663.55k alloc=5.8GB sys=10.0GB
INFO[02-28|19:09:17.064] Committed                                time=2m7.857328s block=3367837 txNum=120395446 step=77.1 flush+commitment=2m3.4420196s tx.commit=4.4136999s prune=0s
INFO[02-28|19:09:17.071] [4/6 Execution]                          blk=3367838 blks=2 blk/s=0.0 txs=8 tx/s=0 gas/s=4.15k buf=4.8KB/512.0MB stepsInDB=0.05 step=77.1 inMem=false alloc=8.6GB sys=10.0GB
INFO[02-28|19:09:29.185] [4/6 Execution]                          blk=3368381 blks=544 blk/s=44.9 txs=10839 tx/s=894 gas/s=810.09M buf=121.6MB/512.0MB stepsInDB=0.05 step=77.1 inMem=false alloc=6.5GB sys=10.0GB
INFO[02-28|19:09:39.241] P2P                                      app=caplin peers=1
INFO[02-28|19:09:49.182] [4/6 Execution]                          blk=3369267 blks=887 blk/s=44.4 txs=16034 tx/s=801 gas/s=877.75M buf=340.1MB/512.0MB stepsInDB=0.05 step=77.1 inMem=false alloc=5.2GB sys=10.0GB
INFO[02-28|19:10:09.179] [4/6 Execution]                          blk=3370152 blks=886 blk/s=44.3 txs=16783 tx/s=839 gas/s=861.91M buf=554.8MB/512.0MB stepsInDB=0.05 step=77.1 inMem=false alloc=4.2GB sys=10.0GB
INFO[02-28|19:10:10.880] [p2p] GoodPeers                          eth68=1 eth67=3
INFO[02-28|19:10:10.880] [mem] memory stats                       alloc=4.5GB sys=10.0GB
INFO[02-28|19:10:15.449] [txpool] stat                            pending=0 baseFee=0 queued=76
INFO[02-28|19:10:29.182] [4/6 Execution][agg] computing trie      progress=501.06k/740.97k alloc=4.3GB sys=10.0GB
INFO[02-28|19:10:39.241] P2P                                      app=caplin peers=1
INFO[02-28|19:10:49.184] [4/6 Execution][agg] computing trie      progress=607.15k/740.97k alloc=7.5GB sys=10.0GB
INFO[02-28|19:11:14.037] Committed                                time=1m4.854861s block=3370152 txNum=120439110 step=77.1 flush+commitment=1m4.1415862s tx.commit=711.1276ms prune=0s
INFO[02-28|19:11:14.075] [4/6 Execution]                          blk=3370153 blks=2 blk/s=0.0 txs=22 tx/s=0 gas/s=420.08k buf=398.1KB/512.0MB stepsInDB=0.08 step=77.1 inMem=false alloc=4.6GB sys=10.0GB
INFO[02-28|19:11:29.192] [4/6 Execution]                          blk=3371146 blks=994 blk/s=65.8 txs=21033 tx/s=1.39k gas/s=637.90M buf=95.0MB/512.0MB stepsInDB=0.08 step=77.1 inMem=false alloc=4.1GB sys=10.0GB
INFO[02-28|19:11:37.143] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-3079160905
INFO[02-28|19:11:39.241] P2P                                      app=caplin peers=3
INFO[02-28|19:11:49.184] [4/6 Execution]                          blk=3372318 blks=1173 blk/s=58.7 txs=19696 tx/s=985 gas/s=434.27M buf=154.2MB/512.0MB stepsInDB=0.08 step=77.1 inMem=false alloc=4.6GB sys=10.0GB
INFO[02-28|19:12:09.182] [4/6 Execution]                          blk=3373571 blks=1254 blk/s=62.7 txs=19506 tx/s=975 gas/s=436.05M buf=218.5MB/512.0MB stepsInDB=0.08 step=77.1 inMem=false alloc=6.7GB sys=10.0GB
INFO[02-28|19:12:29.183] [4/6 Execution]                          blk=3374853 blks=1283 blk/s=64.1 txs=20183 tx/s=1.00k gas/s=505.98M buf=305.3MB/512.0MB stepsInDB=0.08 step=77.1 inMem=false alloc=6.0GB sys=10.0GB
INFO[02-28|19:12:39.241] P2P                                      app=caplin peers=3
INFO[02-28|19:12:49.182] [4/6 Execution]                          blk=3376436 blks=1584 blk/s=79.2 txs=25429 tx/s=1.27k gas/s=338.72M buf=338.9MB/512.0MB stepsInDB=0.08 step=77.1 inMem=false alloc=4.9GB sys=10.0GB
INFO[02-28|19:13:09.180] [4/6 Execution]                          blk=3377582 blks=1147 blk/s=57.4 txs=21525 tx/s=1.07k gas/s=628.40M buf=457.7MB/512.0MB stepsInDB=0.08 step=77.2 inMem=false alloc=7.5GB sys=10.0GB
INFO[02-28|19:13:10.881] [mem] memory stats                       alloc=4.4GB sys=10.0GB
INFO[02-28|19:13:10.881] [p2p] GoodPeers                          eth68=2 eth67=3
INFO[02-28|19:13:15.449] [txpool] stat                            pending=0 baseFee=2 queued=83
INFO[02-28|19:13:29.202] [4/6 Execution]                          blk=3378471 blks=890 blk/s=44.5 txs=18608 tx/s=929 gas/s=369.69M buf=496.3MB/512.0MB stepsInDB=0.08 step=77.2 inMem=false alloc=5.1GB sys=10.0GB
INFO[02-28|19:13:39.241] P2P                                      app=caplin peers=4
INFO[02-28|19:13:49.183] [4/6 Execution]                          blk=3380013 blks=1543 blk/s=77.2 txs=29797 tx/s=1.49k gas/s=340.51M buf=525.3MB/512.0MB stepsInDB=0.08 step=77.2 inMem=false alloc=4.9GB sys=10.0GB
INFO[02-28|19:14:09.185] [4/6 Execution][agg] computing trie      progress=1.17M/1.42M alloc=5.0GB sys=10.2GB
INFO[02-28|19:14:30.855] [Caplin-Blocks] Flushed buffer file      name=erigon-sortable-buf-2592919277
INFO[02-28|19:14:37.704] Committed                                time=48.5187504s block=3380013 txNum=120614909 step=77.2 flush+commitment=47.7155353s tx.commit=801.1307ms prune=0s
INFO[02-28|19:14:37.709] [4/6 Execution]                          blk=3380014 blks=2 blk/s=0.0 txs=8 tx/s=0 gas/s=42.76k buf=33.5KB/512.0MB stepsInDB=0.19 step=77.2 inMem=false alloc=6.6GB sys=10.3GB
INFO[02-28|19:14:39.241] P2P                                      app=caplin peers=4
Exception 0xc0000005 0x0 0x10 0x7ff6294c7aaa
PC=0x7ff6294c7aaa
signal arrived during external code execution

runtime.cgocall(0x7ff6294c3ec0, 0xc173d16578)
        runtime/cgocall.go:167 +0x3e fp=0xc173d16550 sp=0xc173d164e8 pc=0x7ff6275a8e5e
github.com/erigontech/mdbx-go/mdbx._Cfunc_mdbx_txn_commit_ex(0x1a454125a40, 0xc0d51135e0)
        _cgo_gotypes.go:1022 +0x50 fp=0xc173d16578 sp=0xc173d16550 pc=0x7ff627af5bb0
github.com/erigontech/mdbx-go/mdbx.(*Txn).commit.func1(...)
        github.com/erigontech/[email protected]/mdbx/txn.go:246
github.com/erigontech/mdbx-go/mdbx.(*Txn).commit(0xc0c938fd10)
        github.com/erigontech/[email protected]/mdbx/txn.go:246 +0xba fp=0xc173d16668 sp=0xc173d16578 pc=0x7ff627afa7da
github.com/erigontech/mdbx-go/mdbx.(*Txn).Commit(...)
        github.com/erigontech/[email protected]/mdbx/txn.go:185
github.com/erigontech/erigon-lib/kv/mdbx.(*MdbxTx).Commit(0xc0c938fd60)
        github.com/erigontech/[email protected]/kv/mdbx/kv_mdbx.go:921 +0x157 fp=0xc173d16888 sp=0xc173d16668 pc=0x7ff627b55a37
github.com/erigontech/erigon-lib/kv/temporal.(*Tx).Commit(0xc0f7409a00)
        github.com/erigontech/[email protected]/kv/temporal/kv_temporal.go:201 +0x45 fp=0xc173d168a0 sp=0xc173d16888 pc=0x7ff628a22985
github.com/erigontech/erigon/eth/stagedsync.ExecV3({_, _}, _, {_, _}, _, {{0x7ff62b7a4630, 0xc001d620f0}, 0x20000000, {0x1, ...}, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/exec3.go:736 +0x2827 fp=0xc173d17478 sp=0xc173d168a0 pc=0x7ff62902d6c7
github.com/erigontech/erigon/eth/stagedsync.ExecBlockV3(_, {_, _}, {{_, _}, _}, _, {_, _}, {{0x7ff62b7a4630, ...}, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/stage_execute.go:162 +0x209 fp=0xc173d17780 sp=0xc173d17478 pc=0x7ff62903efa9
github.com/erigontech/erigon/eth/stagedsync.SpawnExecuteBlocksStage(_, {_, _}, {{_, _}, _}, _, {_, _}, {{0x7ff62b7a4630, ...}, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/stage_execute.go:256 +0xcd fp=0xc173d17a48 sp=0xc173d17780 pc=0x7ff6290404ad
github.com/erigontech/erigon/eth/stagedsync.PipelineStages.func10(0xc0?, 0x7ff62aefc488?, {0x7ff62b786260?, 0xc002fe6dc0?}, {{0x0?, 0x0?}, 0x0?}, {0x7ff62b7a2e08?, 0xc0005ad220?})
        github.com/erigontech/erigon/eth/stagedsync/default_stages.go:238 +0xc6 fp=0xc173d17d00 sp=0xc173d17a48 pc=0x7ff629024626
github.com/erigontech/erigon/eth/stagedsync.(*Sync).runStage(0xc002fe6dc0, 0xc0030fe3c0, {0x7ff62b7a4630, 0xc001d620f0}, {{0x0?, 0x0?}, 0x0?}, 0x1, 0x1, 0x0)
        github.com/erigontech/erigon/eth/stagedsync/sync.go:529 +0x247 fp=0xc173d17df8 sp=0xc173d17d00 pc=0x7ff629079de7
github.com/erigontech/erigon/eth/stagedsync.(*Sync).Run(0xc002fe6dc0, {0x7ff62b7a4630, 0xc001d620f0}, {{0x0?, 0x0?}, 0x0?}, 0x28?, 0x1)
        github.com/erigontech/erigon/eth/stagedsync/sync.go:412 +0x2cd fp=0xc173d17ec8 sp=0xc173d17df8 pc=0x7ff629078ced
github.com/erigontech/erigon/turbo/stages.ProcessFrozenBlocks({0x7ff62b78bc10, 0xc00155d2c0}, {0x7ff62b7a4630, 0xc001d620f0}, {0x7ff62b7ca968, 0xc001fee960}, 0xc002fe6dc0, 0x0)
        github.com/erigontech/erigon/turbo/stages/stageloop.go:153 +0x133 fp=0xc173d17f28 sp=0xc173d17ec8 pc=0x7ff6293f8653
github.com/erigontech/erigon/turbo/execution/eth1.(*EthereumExecutionModule).Start(0xc0005f2640, {0x7ff62b78bc10, 0xc00155d2c0})
        github.com/erigontech/erigon/turbo/execution/eth1/ethereum_execution.go:327 +0x165 fp=0xc173d17fb8 sp=0xc173d17f28 pc=0x7ff629406045
github.com/erigontech/erigon/eth.(*Ethereum).Start.gowrap1()
        github.com/erigontech/erigon/eth/backend.go:1591 +0x28 fp=0xc173d17fe0 sp=0xc173d17fb8 pc=0x7ff6294251c8
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc173d17fe8 sp=0xc173d17fe0 pc=0x7ff6275b4581
created by github.com/erigontech/erigon/eth.(*Ethereum).Start in goroutine 1
        github.com/erigontech/erigon/eth/backend.go:1591 +0xdd5

goroutine 1 gp=0xc0000021c0 m=nil [chan receive, 40 minutes]:
runtime.gopark(0xc000d93740?, 0x7ff62758209f?, 0x20?, 0x37?, 0x7ff6275820e0?)
        runtime/proc.go:435 +0xce fp=0xc00298f708 sp=0xc00298f6e8 pc=0x7ff6275ac8ce
runtime.chanrecv(0xc000056310, 0x0, 0x1)
        runtime/chan.go:664 +0x445 fp=0xc00298f780 sp=0xc00298f708 pc=0x7ff6275431a5
runtime.chanrecv1(0xc001a5c380?, 0xc000a8cdc0?)
        runtime/chan.go:506 +0x12 fp=0xc00298f7a8 sp=0xc00298f780 pc=0x7ff627542d32
github.com/erigontech/erigon/node.(*Node).Wait(...)
        github.com/erigontech/erigon/node/node.go:270
github.com/erigontech/erigon/turbo/node.(*ErigonNode).Serve(0xc002fb3330)
        github.com/erigontech/erigon/turbo/node/node.go:51 +0x65 fp=0xc00298f7f8 sp=0xc00298f7a8 pc=0x7ff629429325
main.runErigon(0xc001580c40)
        github.com/erigontech/erigon/cmd/erigon/main.go:106 +0x425 fp=0xc00298fad0 sp=0xc00298f7f8 pc=0x7ff629454aa5
github.com/erigontech/erigon/turbo/app.MakeApp.func1(0xc001580c40)
        github.com/erigontech/erigon/turbo/app/make_app.go:71 +0x17f fp=0xc00298fbf8 sp=0xc00298fad0 pc=0x7ff62943de9f
github.com/urfave/cli/v2.(*Command).Run(0xc000b47340, 0xc001580c40, {0xc00006a100, 0xc, 0x10})
        github.com/urfave/cli/[email protected]/command.go:276 +0x7be fp=0xc00298fe68 sp=0xc00298fbf8 pc=0x7ff62791113e
github.com/urfave/cli/v2.(*App).RunContext(0xc0012a5c00, {0x7ff62b78b6d0, 0x7ff62cfabd40}, {0xc00006a100, 0xc, 0x10})
        github.com/urfave/cli/[email protected]/app.go:333 +0x5a5 fp=0xc00298fec8 sp=0xc00298fe68 pc=0x7ff62790db85
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/[email protected]/app.go:307
main.main()
        github.com/erigontech/erigon/cmd/erigon/main.go:51 +0x85 fp=0xc00298ff50 sp=0xc00298fec8 pc=0x7ff6294545a5
runtime.main()
        runtime/proc.go:283 +0x27d fp=0xc00298ffe0 sp=0xc00298ff50 pc=0x7ff6275783fd
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00298ffe8 sp=0xc00298ffe0 pc=0x7ff6275b4581

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle), 13 minutes]:
runtime.gopark(0x7edc612d9f80?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000c1fa8 sp=0xc0000c1f88 pc=0x7ff6275ac8ce
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.forcegchelper()
        runtime/proc.go:348 +0xb8 fp=0xc0000c1fe0 sp=0xc0000c1fa8 pc=0x7ff627578718
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c1fe8 sp=0xc0000c1fe0 pc=0x7ff6275b4581
created by runtime.init.7 in goroutine 1
        runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x7ff62cf54e01?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000c3f80 sp=0xc0000c3f60 pc=0x7ff6275ac8ce
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.bgsweep(0xc0000d0000)
        runtime/mgcsweep.go:316 +0xdf fp=0xc0000c3fc8 sp=0xc0000c3f80 pc=0x7ff62755e37f
runtime.gcenable.gowrap1()
        runtime/mgc.go:204 +0x25 fp=0xc0000c3fe0 sp=0xc0000c3fc8 pc=0x7ff627552685
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x7ff6275b4581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x18d3b2?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000d7f78 sp=0xc0000d7f58 pc=0x7ff6275ac8ce
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.(*scavengerState).park(0x7ff62cf58220)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc0000d7fa8 sp=0xc0000d7f78 pc=0x7ff62755bdc9
runtime.bgscavenge(0xc0000d0000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc0000d7fc8 sp=0xc0000d7fa8 pc=0x7ff62755c359
runtime.gcenable.gowrap2()
        runtime/mgc.go:205 +0x25 fp=0xc0000d7fe0 sp=0xc0000d7fc8 pc=0x7ff627552625
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d7fe8 sp=0xc0000d7fe0 pc=0x7ff6275b4581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000003340 m=nil [finalizer wait, 4 minutes]:
runtime.gopark(0x0?, 0xc021be58f0?, 0x20?, 0xc0?, 0x1000000010?)
        runtime/proc.go:435 +0xce fp=0xc0000d9e30 sp=0xc0000d9e10 pc=0x7ff6275ac8ce
runtime.runfinq()
        runtime/mfinal.go:196 +0x107 fp=0xc0000d9fe0 sp=0xc0000d9e30 pc=0x7ff627551687
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x7ff6275b4581
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:166 +0x3d

goroutine 6 gp=0xc000003500 m=nil [GC worker (idle)]:
runtime.gopark(0x7f7aeb4db288?, 0x1?, 0x4?, 0x8c?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000c5f38 sp=0xc0000c5f18 pc=0x7ff6275ac8ce
runtime.gcBgMarkWorker(0xc000056f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000c5fc8 sp=0xc0000c5f38 pc=0x7ff627554ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000c5fe0 sp=0xc0000c5fc8 pc=0x7ff627554a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c5fe8 sp=0xc0000c5fe0 pc=0x7ff6275b4581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 18 gp=0xc00052a000 m=nil [GC worker (idle)]:
runtime.gopark(0x7f7aeb4db288?, 0x3?, 0xb8?, 0xef?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000d3f38 sp=0xc0000d3f18 pc=0x7ff6275ac8ce
runtime.gcBgMarkWorker(0xc000056f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000d3fc8 sp=0xc0000d3f38 pc=0x7ff627554ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000d3fe0 sp=0xc0000d3fc8 pc=0x7ff627554a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d3fe8 sp=0xc0000d3fe0 pc=0x7ff6275b4581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 19 gp=0xc00052a1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x7f7aeb4db288?, 0x3?, 0xe4?, 0xf0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc0000d5f38 sp=0xc0000d5f18 pc=0x7ff6275ac8ce
runtime.gcBgMarkWorker(0xc000056f50)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000d5fc8 sp=0xc0000d5f38 pc=0x7ff627554ba9
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000d5fe0 sp=0xc0000d5fc8 pc=0x7ff627554a85
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d5fe8 sp=0xc0000d5fe0 pc=0x7ff6275b4581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

@BjarkeCK
Copy link
Author

I will try on my main drive in case the drive is faulty.

@BjarkeCK
Copy link
Author

no difference there

@BjarkeCK
Copy link
Author

BjarkeCK commented Feb 28, 2025 •
edited
Loading

#9808
#8623
#8460
#7754

Related issues.

@erthink
Copy link
Contributor

erthink commented Feb 28, 2025

Image
Find the rabbit ;)

@erthink
Copy link
Contributor

erthink commented Feb 28, 2025

#9808 #8623 #8460 #7754

Related issues.

Oh, a lot of.
Seems a long time nobody test/maintain Windows build.
So, we should expect a bugs/glitches even with libmdbx 0.12.13, i.e. probability is high.

@BjarkeCK
Copy link
Author

So with the latest build and the latest version of mingw64 it seems to be working. I've not had a crash in an hour or so. It's taking forever to "DownloadHistoricalBlocks", but I guess that's a separate issue, and maybe a normal one.

To anyone with this issue. Do not install mingw64 through choco, they have an old version! Install manually, and add env paths manually, then recompile erigon.

@JkLondon
Copy link
Member

JkLondon commented Mar 1, 2025 •
edited
Loading

@BjarkeCK @c184103 maybe I'm late here but u can pull mdbx-mega-stable branch to try if it's needed. Just've missed old arch style of our bindings)

Related PR: #14016

@JkLondon JkLondon self-assigned this Mar 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JkLondon JkLondon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@erthink @BjarkeCK @JkLondon