Unable to access StorageGRID through the S3 API due to RSM is down

Applies to

StorageGRID 11.2.0.5 and earlier

Issue

• Unable to access StorageGRID through the S3 API.
• SVST(Status - Replicated State Machine Service (RSM)) alarm is detected due to Not Running.

Example:

• bycast-err.log on the ADC node shows "rsm: rsm panic" repeatedly.

Example:

<HOST> rsm: rsm panic

• servermanager.log shows that Replicated State Machine (RSM) service run into error state as the result of repeated failures for trying to start.

Example:

rsm                       | start initiated
rsm                       | rsm starting
rsm                       | rsm ended
rsm                       | rsm starting
rsm                       | rsm ended
rsm                       | rsm starting
rsm                       | rsm ended
..
rsm                       | rsm starting
rsm                       | rsm ended
rsm                       | Too many failed attempts, entering error state
rsm                       | rsm ended
 

• rsm.errlog shows RSM gets PANIC with SIGSEGV.

Example:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xa4cf23]

goroutine 150 [running]:
storagegrid/rsm/client.(*ESClient).Execute(0xc42038a6a0, 0xd21d40, 0xc420c2c1e0, 0x0)
    /build/go/src/storagegrid/rsm/client/search.go:198 +0x603
main.requestExecutor(0xc420520360, 0xc420512000, 0xc420514200, 0xc420d43f98, 0xc420514300)
    /build/go/src/storagegrid/rsm/request_runner.go:159 +0x6e8
main.run.func5(0xc4203ba3f0, 0xc420514200, 0xc420512000, 0xc42021c000, 0xc420514300)
    /build/go/src/storagegrid/rsm/run.go:162 +0x92
created by main.run
    /build/go/src/storagegrid/rsm/run.go:160 +0x899