Node stuck at a block with mismatched receipt headers

[snail-inO]

System information

Erigon version: 2.60.9-501a53b7

OS & Version: Ubuntu 22.04.5 LTS

Docker version: 27.2.0

Erigon Command (with flags/config):

15 command: [
1 "--chain=mainnet",
2 "--http", # Enable HTTP-RPC server
3 "--http.addr=0.0.0.0", # Bind RPC to all IPs
4 "--http.port=8545", # JSON-RPC port
5 "--http.api=eth,web3,net,debug,trace,txpool",
6 "--http.corsdomain=*",
7 "--http.vhosts=*",
8 "--ws", # Enable WebSocket server
9 "--ws.port=8546", # WebSocket port
10 "--internalcl", # Enables internal consensus-layer mode
11 "--metrics", # Optional: Enable metrics if monitoring with Prometheus
12 "--nat=none",
13 "--bad.block=21325500"
14 ]

Consensus Layer: Caplin

Chain/Network: Ethereum mainnet

Expected behaviour

Stay synced with the latest head.

Actual behaviour

block stuck at 21326547.

Steps to reproduce the behaviour

Run the node using docker-compose with the following docker-compose file:

  1  version: '3'
   1
   2 services:
   3   erigon:
   4     image: erigontech/erigon:latest  # Use the latest version or specify a version tag
   5     container_name: erigon-node
   6     restart: unless-stopped
   7     ports:
   8       - "16325:8545"       # JSON-RPC port
   9       - "16326:8546"       # WebSocket port
  10       - "30303:30303"     # P2P port (for peer discovery and connections)
  11       - "30303:30303/udp" # UDP P2P port
  12     volumes:
  13       - ./erigon_data_mainnet:/home/erigon/.local/share/erigon  # Volume for persistent data
  14     command: [
  15       "--chain=mainnet",
  16       "--http",                  # Enable HTTP-RPC server
  17       "--http.addr=0.0.0.0",     # Bind RPC to all IPs
  18       "--http.port=8545",        # JSON-RPC port
  19       "--http.api=eth,web3,net,debug,trace,txpool",
  20       "--http.corsdomain=*",
  21       "--http.vhosts=*",
  22       "--ws",                    # Enable WebSocket server
  23       "--ws.port=8546",          # WebSocket port
  24       "--internalcl",  # Enables internal consensus-layer mode
  25       "--metrics",               # Optional: Enable metrics if monitoring with Prometheus
  26       "--nat=none",
  27       "--bad.block=21325500"
  28     ]

Backtrace

[INFO] [12-04|21:09:25.684] [3/12 Senders] Started                   from=21326547 to=21331728
[INFO] [12-04|21:09:31.844] [4/12 Execution] Blocks execution        from=21326547 to=21331728
[WARN] [12-04|21:09:32.040] [4/12 Execution] Execution failed        block=21326550 hash=0xf2de023215f5cc46786c4bf4680921dd207e7bb839e9dfca3ad87ac8aabcea2d err="invalid block: mismatched receipt headers for block 21326550 (0xd660cbbc44b10d6fb36ba6df4bde75ca3fdec45f5c7868de2d313aa5c01b0890 != 0x3eeca00fc16689b8f84b6bdec812a6220091f6bdf91777f3916809b880efbe28)"
[INFO] [12-04|21:09:32.066] [4/12 Execution] Completed on            block=21326549
[WARN] [12-04|21:09:32.066] bad forkchoice                           head=0x6296c032548ea12ea903a21511a7a4bc981dd62885f27ba6df51595575e8d89a hash=0xe6432b578072560e6ff251f2892c081c3a2d7f06e20e46f3c1be0d2b059cce48
[INFO] [12-04|21:09:37.399] [3/12 Senders] Started                   from=21326547 to=21331729
[INFO] [12-04|21:09:43.872] [4/12 Execution] Blocks execution        from=21326547 to=21331729
[WARN] [12-04|21:09:44.066] [4/12 Execution] Execution failed        block=21326550 hash=0xf2de023215f5cc46786c4bf4680921dd207e7bb839e9dfca3ad87ac8aabcea2d err="invalid block: mismatched receipt headers for block 21326550 (0xd660cbbc44b10d6fb36ba6df4bde75ca3fdec45f5c7868de2d313aa5c01b0890 != 0x3eeca00fc16689b8f84b6bdec812a6220091f6bdf91777f3916809b880efbe28)"
[INFO] [12-04|21:09:44.091] [4/12 Execution] Completed on            block=21326549
[WARN] [12-04|21:09:44.091] bad forkchoice                           head=0x6296c032548ea12ea903a21511a7a4bc981dd62885f27ba6df51595575e8d89a hash=0xea7c69e1d840ffc4632e6ca472287d2bcfe6ebfa402a4fe2554b036fb3f92dd5
[INFO] [12-04|21:09:49.907] [3/12 Senders] Started                   from=21326547 to=21331730
[INFO] [12-04|21:09:56.199] [4/12 Execution] Blocks execution        from=21326547 to=21331730
[WARN] [12-04|21:09:56.395] [4/12 Execution] Execution failed        block=21326550 hash=0xf2de023215f5cc46786c4bf4680921dd207e7bb839e9dfca3ad87ac8aabcea2d err="invalid block: mismatched receipt headers for block 21326550 (0xd660cbbc44b10d6fb36ba6df4bde75ca3fdec45f5c7868de2d313aa5c01b0890 != 0x3eeca00fc16689b8f84b6bdec812a6220091f6bdf91777f3916809b880efbe28)"
[INFO] [12-04|21:09:56.420] [4/12 Execution] Completed on            block=21326549
[WARN] [12-04|21:09:56.421] bad forkchoice                           head=0x6296c032548ea12ea903a21511a7a4bc981dd62885f27ba6df51595575e8d89a hash=0x455c1e636cf1781de2969d3ad12e3a6694f5879ccc1b279391142ef8b1c22d53
[INFO] [12-04|21:10:01.462] [3/12 Senders] Started                   from=21326547 to=21331731
[INFO] [12-04|21:10:07.913] [4/12 Execution] Blocks execution        from=21326547 to=21331731
[WARN] [12-04|21:10:08.111] [4/12 Execution] Execution failed        block=21326550 hash=0xf2de023215f5cc46786c4bf4680921dd207e7bb839e9dfca3ad87ac8aabcea2d err="invalid block: mismatched receipt headers for block 21326550 (0xd660cbbc44b10d6fb36ba6df4bde75ca3fdec45f5c7868de2d313aa5c01b0890 != 0x3eeca00fc16689b8f84b6bdec812a6220091f6bdf91777f3916809b880efbe28)"
[INFO] [12-04|21:10:08.136] [4/12 Execution] Completed on            block=21326549
[WARN] [12-04|21:10:08.136] bad forkchoice                           head=0x6296c032548ea12ea903a21511a7a4bc981dd62885f27ba6df51595575e8d89a hash=0x15d131fca3cdf6b25556dc5d5629d379780ab65459b4b907ad597fae9560fe17
[INFO] [12-04|21:10:08.613] [p2p] GoodPeers                          eth68=28 eth67=7
[INFO] [12-04|21:10:09.324] [mem] memory stats                       Rss=17.1GB Size=0B Pss=17.1GB SharedClean=4.0KB SharedDirty=0B PrivateClean=2.2GB PrivateDirty=14.8GB Referenced=17.1GB Anonymous=14.8GB Swap=0B alloc=8.7GB sys=15.6GB
[INFO] [12-04|21:10:09.340] [txpool] stat                            pending=0 baseFee=0 queued=13959 alloc=8.7GB sys=15.6GB
[INFO] [12-04|21:10:13.420] [3/12 Senders] Started                   from=21326547 to=21331732
[INFO] [12-04|21:10:19.647] [4/12 Execution] Blocks execution        from=21326547 to=21331732
[WARN] [12-04|21:10:19.850] [4/12 Execution] Execution failed        block=21326550 hash=0xf2de023215f5cc46786c4bf4680921dd207e7bb839e9dfca3ad87ac8aabcea2d err="invalid block: mismatched receipt headers for block 21326550 (0xd660cbbc44b10d6fb36ba6df4bde75ca3fdec45f5c7868de2d313aa5c01b0890 != 0x3eeca00fc16689b8f84b6bdec812a6220091f6bdf91777f3916809b880efbe28)"
[INFO] [12-04|21:10:19.875] [4/12 Execution] Completed on            block=21326549
[WARN] [12-04|21:10:19.875] bad forkchoice                           head=0x6296c032548ea12ea903a21511a7a4bc981dd62885f27ba6df51595575e8d89a hash=0x39b1b8be138f2bcafee56cac0dc8eee3f37985211def530c7a3bc624cab0c963

[somnathb1]

@snail-inO

Please share all historical logs before this probelm in a zip file. Thank you in advance.

Till then, there are a few things you can do to resolve it

• There is an issue with a bad block and/or bad state. You can try stopping the node.

integration stage_exec --unwind=N  # with N being 100 or 1000 - to clear up till the bad block
integration stage_bodies --reset
integration clear_bad_blocks --datadir=<datadir>

• Retry with an external CL client to isolate possible issues with Caplin (internal CL) as, Caplin in Erigon-2 may an issue