Skip to content
This repository has been archived by the owner on Sep 2, 2024. It is now read-only.

[Consolidated] kafka-ch-dispatcher Observed a panic: "invalid memory address or nil pointer dereference" in dispatcher.(*Reconciler).CleanupChannel #973

Open
maschmid opened this issue Nov 5, 2021 · 3 comments
Open

[Consolidated] kafka-ch-dispatcher Observed a panic: "invalid memory address or nil pointer dereference" in dispatcher.(*Reconciler).CleanupChannel #973

maschmid opened this issue Nov 5, 2021 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/accepted Issues which should be fixed (post-triage)

Comments

@maschmid
Copy link
Contributor

maschmid commented Nov 5, 2021

Describe the bug

kafka-ch-dispatcher observed a panic:

E1105 10:40:48.639573       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 105 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1d17d80, 0x310abe0)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)   
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1d17d80, 0x310abe0)
        /usr/lib/golang/src/runtime/panic.go:969 +0x1b9     
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.(*Reconciler).CleanupChannel(...)
        /opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:196
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.NewController.func2(0x1fbcde0, 0xc00012d380)
        /opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:138 +0x66
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/controller.go:245
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete(0xc000014cf0, 0x239c1a0, 0xc00047d3e0, 0x1fbcde0, 0xc00012d380)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/controller.go:288 +0x6a
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/shared_informer.go:779 +0x166
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006e9760)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000c4ff60, 0x2355000, 0xc0003b8b10, 0x1ccc401, 0xc0003e4540)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006e9760, 0x3b9aca00, 0x0, 0x1, 0xc0003e4540)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0008df700)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0000d6450, 0xc0003c0590)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start    
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b2bc26]

goroutine 105 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1d17d80, 0x310abe0)
        /usr/lib/golang/src/runtime/panic.go:969 +0x1b9
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.(*Reconciler).CleanupChannel(...)
        /opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:196
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.NewController.func2(0x1fbcde0, 0xc00012d380)
        /opt/app-root/src/go/src/knative.dev/eventing/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:138 +0x66
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/controller.go:245
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete(0xc000014cf0, 0x239c1a0, 0xc00047d3e0, 0x1fbcde0, 0xc00012d380)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/controller.go:288 +0x6a
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/shared_informer.go:779 +0x166
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006e9760)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000c4ff60, 0x2355000, 0xc0003b8b10, 0x1ccc401, 0xc0003e4540)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006e9760, 0x3b9aca00, 0x0, 0x1, 0xc0003e4540)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0008df700)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0000d6450, 0xc0003c0590)
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /opt/app-root/src/go/src/knative.dev/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

Expected behavior
Should not panic

To Reproduce
Currently unknown, just noticed the kafka-ch-dispatcher has been restarted on the cluster, with the panic in the previous logs

Knative release version
0.24.3

Additional context
Knative installed with OpenShift Serverless Operator (1.18)
@maschmid maschmid added the kind/bug Categorizes issue or PR as related to a bug. label Nov 5, 2021
@github-actions
Copy link

github-actions bot commented Feb 4, 2022

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 4, 2022
@maschmid
Copy link
Contributor Author

maschmid commented Mar 1, 2022

/remove-lifecycle stale

seen on 1.0.0

E0301 06:49:30.934046       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 130 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19fe700, 0x2ef2e20})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x20})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x19fe700, 0x2ef2e20})
	/usr/lib/golang/src/runtime/panic.go:1038 +0x215
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.(*Reconciler).CleanupChannel(...)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:197
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.NewController.func2({0x1c5be40, 0xc0029f2380})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:139 +0x6a
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/controller.go:245
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete({0xc00023b5c0, {0x1fb3740, 0xc00000f6f8}}, {0x1c5be40, 0xc0029f2380})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/controller.go:288 +0x64
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/shared_informer.go:779 +0xdf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f005c1d06b0)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005ba738, {0x1f83200, 0xc000156000}, 0x1, 0xc000154000)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000272c00, 0x3b9aca00, 0x0, 0x51, 0xb25f60)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0002fc800)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x180108a]

goroutine 130 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x20})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x19fe700, 0x2ef2e20})
	/usr/lib/golang/src/runtime/panic.go:1038 +0x215
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.(*Reconciler).CleanupChannel(...)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:197
knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher.NewController.func2({0x1c5be40, 0xc0029f2380})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/pkg/channel/consolidated/reconciler/dispatcher/kafkachannel.go:139 +0x6a
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/controller.go:245
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete({0xc00023b5c0, {0x1fb3740, 0xc00000f6f8}}, {0x1c5be40, 0xc0029f2380})
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/controller.go:288 +0x64
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/shared_informer.go:779 +0xdf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f005c1d06b0)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005ba738, {0x1f83200, 0xc000156000}, 0x1, 0xc000154000)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000272c00, 0x3b9aca00, 0x0, 0x51, 0xb25f60)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0002fc800)
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/opt/app-root/src/go/src/knative.dev/eventing-kafka/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88

@knative-prow-robot knative-prow-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 1, 2022
@pierDipi
Copy link
Member

pierDipi commented Mar 1, 2022

/triage accepted

@knative-prow-robot knative-prow-robot added the triage/accepted Issues which should be fixed (post-triage) label Mar 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/accepted Issues which should be fixed (post-triage)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maschmid @pierDipi @knative-prow-robot