Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

containerd crashing with - panic: freepages: failed to get all reachable pages #883

Open
vikumar-ciena opened this issue Jan 10, 2025 · 2 comments
Open

containerd crashing with - panic: freepages: failed to get all reachable pages #883

vikumar-ciena opened this issue Jan 10, 2025 · 2 comments

Comments

@vikumar-ciena
Copy link

docker daemon unable to start conatinerd . I see the following trace .

Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: panic: freepages: failed to get all reachable pages (key[0]=(hex)35623764373364343838373233636333396537396261656365323161333731386339376437663133303033316261366566643762613837613564636633343438 on leaf page(2) needs to be < than key of the next element in ancestor (hex)30346135303461633136653430646331666166646331666666633065386133353030663139396662653836656561663961383830323938333734333964343366. Pages stack: [11 2])
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 71 gp=0xc000503880 m=12 mp=0xc000601808 [running]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: panic({0x5611da793340?, 0xc000445c40?})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/panic.go:779 +0x158 fp=0xc00049bf88 sp=0xc00049bed8 pc=0x5611d8f551f8
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).freepages.func2()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:1202 +0x8d fp=0xc00049bfe0 sp=0xc00049bf88 pc=0x5611d97a2e4d
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by go.etcd.io/bbolt.(*DB).freepages in goroutine 48
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:1200 +0x1e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 1 gp=0xc0000061c0 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc00059f770?, 0x2?, 0x60?, 0x0?, 0xc00059f59c?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000787438 sp=0xc000787418 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc000787770, 0xc00059f598, 0x5611da1e1efc?, 0x0, 0xc00005fa90?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000787558 sp=0xc000787438 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc00062a000)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/cmd/containerd/command/main.go:215 +0x8e5 fp=0xc000787900 sp=0xc000787558 pc=0x5611d9793085
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/urfave/cli.HandleAction({0x5611da7cd4c0?, 0x5611daaf57a8?}, 0xc0000fa700?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/github.com/urfave/cli/app.go:524 +0x50 fp=0xc000787918 sp=0xc000787900 pc=0x5611d976d7b0
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/urfave/cli.(*App).Run(0xc0000fa700, {0xc0001102d0, 0x3, 0x3})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/github.com/urfave/cli/app.go:286 +0x79b fp=0xc000787ef8 sp=0xc000787918 pc=0x5611d976ae1b
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: main.main()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/cmd/containerd/main.go:39 +0x2d fp=0xc000787f50 sp=0xc000787ef8 pc=0x5611da1d80ad
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.main()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:271 +0x29d fp=0xc000787fe0 sp=0xc000787f50 pc=0x5611d8f5951d
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000787fe8 sp=0xc000787fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000098fa8 sp=0xc000098f88 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goparkunlock(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:408
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.forcegchelper()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000098fe0 sp=0xc000098fa8 pc=0x5611d8f597d8
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000098fe8 sp=0xc000098fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.init.6 in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:314 +0x1a
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000099780 sp=0xc000099760 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goparkunlock(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:408
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.bgsweep(0xc0000c2000)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgcsweep.go:318 +0xdf fp=0xc0000997c8 sp=0xc000099780 pc=0x5611d8f410ff
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcenable.gowrap1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc0000997e0 sp=0xc0000997c8 pc=0x5611d8f357e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000997e8 sp=0xc0000997e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcenable in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:203 +0x66
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x10000?, 0x5611da47b840?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000099f78 sp=0xc000099f58 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goparkunlock(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:408
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.(*scavengerState).park(0x5611db90d0c0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000099fa8 sp=0xc000099f78 pc=0x5611d8f3e8e9
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.bgscavenge(0xc0000c2000)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000099fc8 sp=0xc000099fa8 pc=0x5611d8f3eeb9
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcenable.gowrap2()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000099fe0 sp=0xc000099fc8 pc=0x5611d8f35785
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcenable in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:204 +0xa5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 18 gp=0xc000104380 m=nil [finalizer wait]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc000098648?, 0x5611d8f27a25?, 0xa8?, 0x1?, 0xc0000061c0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000098620 sp=0xc000098600 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.runfinq()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc0000987e0 sp=0xc000098620 pc=0x5611d8f347a7
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.createfing in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mfinal.go:164 +0x3d
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 21 gp=0xc0003be540 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc0000947a8?, 0x5611d8f1f931?, 0x60?, 0x60?, 0x1?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000094750 sp=0xc000094730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000947e0 sp=0xc000094750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000947e8 sp=0xc0000947e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 22 gp=0xc0003befc0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000094f50 sp=0xc000094f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000094fe0 sp=0xc000094f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 23 gp=0xc0003bf180 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000095750 sp=0xc000095730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000957e0 sp=0xc000095750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000957e8 sp=0xc0000957e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 24 gp=0xc0003bf340 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000095f50 sp=0xc000095f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000095fe0 sp=0xc000095f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 6 gp=0xc000007c00 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009a750 sp=0xc00009a730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009a7e0 sp=0xc00009a750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009a7e8 sp=0xc00009a7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 7 gp=0xc000007dc0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009af50 sp=0xc00009af30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009afe0 sp=0xc00009af50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009afe8 sp=0xc00009afe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 8 gp=0xc0000fa000 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009b750 sp=0xc00009b730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009b7e0 sp=0xc00009b750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009b7e8 sp=0xc00009b7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 9 gp=0xc0000fa1c0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009bf50 sp=0xc00009bf30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009bfe0 sp=0xc00009bf50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009bfe8 sp=0xc00009bfe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 35 gp=0xc00041c000 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000422750 sp=0xc000422730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004227e0 sp=0xc000422750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004227e8 sp=0xc0004227e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 36 gp=0xc00041c1c0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000422f50 sp=0xc000422f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000422fe0 sp=0xc000422f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000422fe8 sp=0xc000422fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 25 gp=0xc0003bf500 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb11c66?, 0x3?, 0xb5?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000096750 sp=0xc000096730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000967e0 sp=0xc000096750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 10 gp=0xc0000fa380 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb1567c?, 0x1?, 0xd8?, 0x8b?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00041e750 sp=0xc00041e730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00041e7e0 sp=0xc00041e750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00041e7e8 sp=0xc00041e7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 37 gp=0xc00041c380 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb14f1a?, 0x3?, 0xb2?, 0xa6?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000423750 sp=0xc000423730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004237e0 sp=0xc000423750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004237e8 sp=0xc0004237e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 38 gp=0xc00041c540 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb0d95e?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000423f50 sp=0xc000423f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000423fe0 sp=0xc000423f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000423fe8 sp=0xc000423fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 26 gp=0xc0003bf6c0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x5611db972420?, 0x1?, 0x45?, 0x7a?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000096f50 sp=0xc000096f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000096fe0 sp=0xc000096f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 39 gp=0xc00041c700 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb0db19?, 0x1?, 0xd2?, 0xa4?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000424750 sp=0xc000424730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004247e0 sp=0xc000424750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004247e8 sp=0xc0004247e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 48 gp=0xc000602380 m=nil [chan send]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x6?, 0x5611d8f751eb?, 0x80?, 0x0?, 0x16e?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000062a60 sp=0xc000062a40 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.chansend(0xc000017380, 0xc000062b58, 0x1, 0x5611d8f27601?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/chan.go:259 +0x38d fp=0xc000062ad0 sp=0xc000062a60 pc=0x5611d8f1f0cd
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.chansend1(0x5611da27261b?, 0x6e?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/chan.go:145 +0x17 fp=0xc000062b00 sp=0xc000062ad0 pc=0x5611d8f1ed37
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.verifyKeyOrder(0x2, {0x5611da1dc67b, 0x4}, 0x1, {0x7fabd83fb210, 0x40, 0x40}, {0x7fabd83fb1c0, 0x40, 0x40}, ...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:188 +0x96d fp=0xc000062be8 sp=0xc000062b00 pc=0x5611d97afaed
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).recursivelyCheckPagesInternal(0xc00001a000, 0x2, {0x7fabd8404040, 0x40, 0x40}, {0x7fabd8404080, 0x40, 0x40}, {0xc0004143e8, 0x1, ...}, ...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:155 +0x465 fp=0xc000062d30 sp=0xc000062be8 pc=0x5611d97aecc5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).recursivelyCheckPagesInternal(0xc00001a000, 0xb, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:147 +0x59e fp=0xc000062e78 sp=0xc000062d30 pc=0x5611d97aedfe
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).recursivelyCheckPages(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:121
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).checkBucket(0xc00001a000, 0xc000291780, 0xc000063310, 0xc000063250, {0x5611dab0f268, 0x5611db96ffc0}, 0xc000017380)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:105 +0x1a5 fp=0xc000062f88 sp=0xc000062e78 pc=0x5611d97ae0e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).checkBucket.func2({0x7fabd8407030?, 0xc000018d20?, 0xc0004143e0?})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:110 +0x90 fp=0xc000063000 sp=0xc000062f88 pc=0x5611d97ae290
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Bucket).ForEachBucket(0x0?, 0xc000063108)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/bucket.go:411 +0x96 fp=0xc000063048 sp=0xc000063000 pc=0x5611d979b596
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).checkBucket(0xc00001a000, 0xc00001a018, 0xc000063310, 0xc000063250, {0x5611dab0f268, 0x5611db96ffc0}, 0xc000017380)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:108 +0x255 fp=0xc000063158 sp=0xc000063048 pc=0x5611d97ae195
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).freepages(0xc00048f8c8)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:1205 +0x225 fp=0xc000063370 sp=0xc000063158 pc=0x5611d97a2c45
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).loadFreelist.func1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:417 +0xc5 fp=0xc0000633a0 sp=0xc000063370 pc=0x5611d979f605
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: sync.(*Once).doSlow(0x5611d979fd20?, 0xc00048fa90?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/sync/once.go:74 +0xc2 fp=0xc000063400 sp=0xc0000633a0 pc=0x5611d8fa3162
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: sync.(*Once).Do(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/sync/once.go:65
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).loadFreelist(0xc00048f8c8?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:413 +0x45 fp=0xc000063430 sp=0xc000063400 pc=0x5611d979f525
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.Open({0xc000018c80, 0x4f}, 0x1a4, 0xc00034f5a0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:295 +0x430 fp=0xc0000634f8 sp=0xc000063430 pc=0x5611d979ef50
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/metadata/plugin.init.1.func1(0xc0003ad280)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/metadata/plugin/plugin.go:159 +0x728 fp=0xc000063850 sp=0xc0000634f8 pc=0x5611d9813548
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/plugin.(*Registration).Init(0xc0004005a0, 0xc0003ad280)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/plugin/plugin.go:132 +0x28 fp=0xc000063890 sp=0xc000063850 pc=0x5611d9665228
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/services/server.New({0x5611dab1c4b8, 0xc0004c8d70}, 0xc0000fa8c0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/services/server/server.go:251 +0x1645 fp=0xc000063ef8 sp=0xc000063890 pc=0x5611d9764265
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/cmd/containerd/command.App.func1.1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/cmd/containerd/command/main.go:194 +0x85 fp=0xc000063fe0 sp=0xc000063ef8 pc=0x5611d9793b45
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/containerd/containerd/cmd/containerd/command.App.func1 in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/cmd/containerd/command/main.go:191 +0x85b
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 68 gp=0xc0000fa540 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc0000a8c78?, 0x4?, 0x50?, 0x0?, 0xc0000a8a70?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000a8900 sp=0xc0000a88e0 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc0000a8c78, 0xc0000a8a68, 0xc000111860?, 0x0, 0xc00018c3c0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0000a8a20 sp=0xc0000a8900 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/docker/go-events.(*Broadcaster).run(0xc0004103c0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/github.com/docker/go-events/broadcast.go:117 +0x14a fp=0xc0000a8fc8 sp=0xc0000a8a20 pc=0x5611d965e92a
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/docker/go-events.NewBroadcaster.gowrap1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/github.com/docker/go-events/broadcast.go:39 +0x25 fp=0xc0000a8fe0 sp=0xc0000a8fc8 pc=0x5611d965e1a5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a8fe8 sp=0xc0000a8fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/docker/go-events.NewBroadcaster in goroutine 48
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b9
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 54 gp=0xc0000fafc0 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc00061c7a0?, 0x2?, 0x0?, 0x0?, 0xc00061c72c?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00061c5d8 sp=0xc00061c5b8 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc00061c7a0, 0xc00061c728, 0x0?, 0x0, 0x0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00061c6f8 sp=0xc00061c5d8 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/cmd/containerd/command/main_unix.go:43 +0xe5 fp=0xc00061c7e0 sp=0xc00061c6f8 pc=0x5611d97912a5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00061c7e8 sp=0xc00061c7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/containerd/containerd/cmd/containerd/command.handleSignals in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/cmd/containerd/command/main_unix.go:40 +0xd3
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 55 gp=0xc0000fb180 m=nil [select, locked to thread]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc00061cfa8?, 0x2?, 0x9?, 0x9c?, 0xc00061cf94?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00061ce38 sp=0xc00061ce18 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc00061cfa8, 0xc00061cf90, 0x0?, 0x0, 0x0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00061cf58 sp=0xc00061ce38 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.ensureSigM.func1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/signal_unix.go:1034 +0x1a5 fp=0xc00061cfe0 sp=0xc00061cf58 pc=0x5611d8f894e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00061cfe8 sp=0xc00061cfe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.ensureSigM in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/signal_unix.go:1017 +0xc8
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 56 gp=0xc0000fb340 m=3 mp=0xc00009f008 [syscall]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.notetsleepg(0x5611db971740, 0xffffffffffffffff)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/lock_futex.go:246 +0x29 fp=0xc00061d7a0 sp=0xc00061d778 pc=0x5611d8f26049
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: os/signal.signal_recv()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00061d7c0 sp=0xc00061d7a0 pc=0x5611d8f8f749
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: os/signal.loop()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00061d7e0 sp=0xc00061d7c0 pc=0x5611d9782233
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00061d7e8 sp=0xc00061d7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by os/signal.Notify.func1.1 in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/os/signal/signal.go:151 +0x1f
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 70 gp=0xc0005036c0 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc000420f88?, 0x2?, 0x40?, 0xe?, 0xc000420f74?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000420e18 sp=0xc000420df8 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc000420f88, 0xc000420f70, 0x0?, 0x0, 0x0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000420f38 sp=0xc000420e18 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/metadata/plugin.init.1.func1.1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/metadata/plugin/plugin.go:152 +0xaf fp=0xc000420fe0 sp=0xc000420f38 pc=0x5611d981394f
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000420fe8 sp=0xc000420fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/containerd/containerd/metadata/plugin.init.1.func1 in goroutine 48
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]:         /root/rpmbuild/BUILD/metadata/plugin/plugin.go:149 +0x709
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28519]: time="2025-01-10T10:07:42.850030681Z" level=error msg="containerd did not exit successfully" error="signal: aborted (core dumped)" module=libcontainerd
Jan 10 10:07:48 onxl7532.ott.ciena.com dockerd[28519]: failed to start containerd: timeout waiting for containerd to start
Jan 10 10:07:48 onxl7532.ott.ciena.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

I have also tried to recover using bbolt surgery command but it does not help . recovered db has the same issue .

bbolt@latest surgery revert-meta-page meta.db new2.db
The meta page is reverted.
[root@onxl7532 io.containerd.metadata.v1.bolt]# go run go.etcd.io/bbolt/cmd/bbolt@latest check new2.db
panic: freepages: failed to get all reachable pages (key[0]=(hex)35623764373364343838373233636333396537396261656365323161333731386339376437663133303033316261366566643762613837613564636633343438 on leaf page(2) needs to be < than key of the next element in ancestor (hex)30346135303461633136653430646331666166646331666666633065386133353030663139396662653836656561663961383830323938333734333964343366. Pages stack: [11 2])

goroutine 17 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()
	/root/go/pkg/mod/go.etcd.io/[email protected]/db.go:1204 +0x8d
created by go.etcd.io/bbolt.(*DB).freepages in goroutine 1
	/root/go/pkg/mod/go.etcd.io/[email protected]/db.go:1202 +0x1c5
exit status 2

version info .


docker info
Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 115
  Running: 114
  Paused: 0
  Stopped: 1
 Images: 176
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: docker-runc io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-206.153.7.1.el8uek.x86_64
 Operating System: Oracle Linux Server 8.10
 OSType: linux
 Architecture: x86_64
 CPUs: 40
 Total Memory: 93.84GiB
 Name: onxl7183.ott.ciena.com
 ID: a4c018be-2ba9-4c42-9297-5312ec73af01
 Docker Root Dir: /opt/ciena/data/docker
 Debug Mode: false
 Labels:
  host_id=0
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
@vikumar-ciena
Copy link
Author

attaching db here
meta.db.gz

@vikumar-ciena
Copy link
Author

i was able to recover using clear-page command .

[root ] [email protected] surgery clear-page meta.db new5.db 2
Page (2) was cleared
[root ] [email protected] check new5.db
panic: freepages: failed to get all reachable pages (key[5]=(hex)62396436363564666536393439353864646663396439656462336630643063653336333961656363333363303435343062313833633536373430623339353032 on leaf page(5) needs to be < than key of the next element in ancestor (hex)62366263353236623964353561663937386461653538643232363162386336343339656163363135333131363432343835643565613965303532333634633537. Pages stack: [11 5])

goroutine 6 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()
	/root/go/pkg/mod/go.etcd.io/[email protected]/db.go:1202 +0x8d
created by go.etcd.io/bbolt.(*DB).freepages in goroutine 1
	/root/go/pkg/mod/go.etcd.io/[email protected]/db.go:1200 +0x1e5
exit status 2

[root] [email protected] surgery clear-page new5.db new6.db 5
Page (5) was cleared
[root] [email protected] check new6.db
OK

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vikumar-ciena