recover from filestorage panic on corrupted DB

[newly12]

Component(s)

extension/storage/filestorage

Is your feature request related to a problem? Please describe.

we are using filelog receiver to collect logs and file_storage extension to persist read states, we have a large number of collectors running and occasionally we saw collectors are crashing while opening the DB, not sure why and how it ran into this state, at the mean time, should we add recover to prevent collector from crashing because of this?

(masked receiver name)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb5f2a0]

goroutine 95 [running]:
go.etcd.io/bbolt.(*Bucket).Sequence(0x7eff026ea057?)
	go.etcd.io/[email protected]/bucket.go:346
go.etcd.io/bbolt.Compact.walk.func3.1({0x7eff026ea057, 0x15, 0x15}, 0x0)
	go.etcd.io/[email protected]/compact.go:94 +0x3c
go.etcd.io/bbolt.(*Tx).ForEach.func1({0x7eff026ea057, 0x15, 0x15}, {0x0?, 0x0?, 0x3fe0000000000000?})
	go.etcd.io/[email protected]/tx.go:131 +0x62
go.etcd.io/bbolt.(*Bucket).ForEach(0xffffffffffffffff?, 0xc001d1cd58)
	go.etcd.io/[email protected]/bucket.go:397 +0x90
go.etcd.io/bbolt.(*Tx).ForEach(0xc000c75f28?, 0xc001d1cda0?)
	go.etcd.io/[email protected]/tx.go:130 +0x45
go.etcd.io/bbolt.Compact.walk.func3(0x0?)
	go.etcd.io/[email protected]/compact.go:93 +0x33
go.etcd.io/bbolt.(*DB).View(0x3b9aca00?, 0xc001d1cea8)
	go.etcd.io/[email protected]/db.go:917 +0x72
go.etcd.io/bbolt.walk(...)
	go.etcd.io/[email protected]/compact.go:92
go.etcd.io/bbolt.Compact(0xc000954248, 0xc000c75d48, 0x10000)
	go.etcd.io/[email protected]/compact.go:21 +0x14c
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage.(*fileStorageClient).Compact(0xc000886d70, {0xc0000479b0?, 0xc001d1d3e8?}, 0x3b9aca00, 0x10000)
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/[email protected]/client.go:198 +0x425
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage.(*localFileStorage).GetClient(0xc0009e8a80, {0x7c8e00?, 0xc00094cc80?}, 0x0?, {{{0xc000a9b920?, 0x82dc76e33?}}, {0xc000a9b928?, 0x0?}}, {0x0, 0x0})
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/[email protected]/extension.go:82 +0x405
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.GetStorageClient({0x1b722b0, 0x29c2000}, {0x1b5c3c0?, 0xc0009f7160?}, 0xc001b3de20, {{{0xc000a9b920?, 0x0?}}, {0xc000a9b928?, 0x0?}})
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/storage.go:29 +0x173
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).setStorageClient(0xc001c87420, {0x1b722b0?, 0x29c2000?}, {0x1b5c3c0?, 0xc0009f7160?})
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/storage.go:34 +0x56
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).Start(0xc001c87420, {0x1b722b0, 0x29c2000}, {0x1b5c3c0, 0xc0009f7160})
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/receiver.go:49 +0xaf

Describe the solution you'd like

recover on DB error

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• extension/storage/filestorage: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@newly12 I agree that we should have a way to recover from this.
I'm keen to reproduce this. Is it possible to share the corrupted DB file?
If it contains confidential data, then can you share some hints to reproduce this? I can give that a shot

[newly12]

The error I posted in description is for the compaction phase, I also find another example for opening DB, please check receiver_filelog_.tgz, it is sort of an empty and no sensitive data.

$ bbolt check receiver_filelog_
panic: freepages: failed to get all reachable pages (page 2: invalid type: freelist (stack: [2]))

goroutine 6 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()
	/Users/xxx/go/pkg/mod/go.etcd.io/[email protected]/db.go:1202 +0x8c
created by go.etcd.io/bbolt.(*DB).freepages in goroutine 1
	/Users/xxx/go/pkg/mod/go.etcd.io/[email protected]/db.go:1200 +0x13c

$ strings receiver_filelog_
default