Skip to content

Missing error handling when writing to unknown Clickhouse columns #55

@fschoell

Description

@fschoell

When the sink receives a database entry for a column that doesn't exist in Clickhouse, the sink crashes with a nil pointer exception:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xf99e6e8]

goroutine 137 [running]:
github.com/streamingfast/substreams-sink-sql/db.convertOpToClickhouseValues(0xc000e12330)
	/Users/work/GoLand/substreams-sink-sql/db/dialect_clickhouse.go:172 +0x208
github.com/streamingfast/substreams-sink-sql/db.clickhouseDialect.Flush({}, {0x15?, 0x1?}, {0x107da570, 0xc000e12870}, 0xc00073a700, {0x107da570?, 0x106b9400?}, 0xc0005ab840?)
	/Users/work/GoLand/substreams-sink-sql/db/dialect_clickhouse.go:64 +0x9c5
github.com/streamingfast/substreams-sink-sql/db.(*Loader).Flush(0xc00073a700, {0xfb36cf3?, 0xbe?}, {0xc000d7b530, 0x28}, 0xc001002130, 0xbe)
	/Users/work/GoLand/substreams-sink-sql/db/flush.go:29 +0x279
github.com/streamingfast/substreams-sink-sql/sinker.(*SQLSinker).HandleBlockScopedData(0xc0006be600, {0x107da5a8, 0xc000836dc0}, 0xc00093b100, 0x119a51aa, 0xc001002130)
	/Users/work/GoLand/substreams-sink-sql/sinker/sinker.go:125 +0x5c5
github.com/streamingfast/substreams-sink.(*Sinker).doRequest(0xc0006e44d0, {0x107da5a8, 0xc000836dc0}, 0xc000820dd0, 0xc00073ab00, {0x107b78e0, 0xc000066270}, {0xc0000661a0, 0x1, 0x1}, ...)
	/Users/work/.go/pkg/mod/github.com/streamingfast/substreams-sink@v0.3.3-0.20231222213313-af36d45db340/sinker.go:472 +0x1974
github.com/streamingfast/substreams-sink.(*Sinker).run(0xc0006e44d0, {0x107da5a8, 0xc000836dc0}, 0xc000820dd0, {0x107cf668, 0xc0006be600})
	/Users/work/.go/pkg/mod/github.com/streamingfast/substreams-sink@v0.3.3-0.20231222213313-af36d45db340/sinker.go:274 +0x867
github.com/streamingfast/substreams-sink.(*Sinker).Run(0xc0006e44d0, {0x107da5a8, 0xc000836dc0}, 0xc000820dd0, {0x107cf668, 0xc0006be600})
	/Users/work/.go/pkg/mod/github.com/streamingfast/substreams-sink@v0.3.3-0.20231222213313-af36d45db340/sinker.go:199 +0x488
github.com/streamingfast/substreams-sink-sql/sinker.(*SQLSinker).Run(0xc0006be600, {0x107da5a8, 0xc000836dc0})
	/Users/work/GoLand/substreams-sink-sql/sinker/sinker.go:94 +0x7c5
created by github.com/streamingfast/cli.(*Application).SuperviseAndStart in goroutine 1
	/Users/work/.go/pkg/mod/github.com/streamingfast/cli@v0.0.4-0.20230825151644-8cc84512cd80/application.go:102 +0x37a

Seems like there is no check if the column actually exists before accessing it in here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions