Storage nodes in unknown state with unresponsive LDR service after upgrading to 11.5
Applies to
StorageGRID 11.5, 11.6, 11.7
Issue
- All StorageGRID storage nodes are in unknown state after upgrading to 11.5 due to unresponsive LDR service
- Below error trace observed in bycast.log:
CRITICAL 0138 SCAN: Assertion Failed#012[/build/src/modules/SignalHandler_Module/SignalHandler_Module.cc:138 void Signal_FatalHandler(int32_t)] ASSERT FAILED ("segmentation fault" == nullptr) is false. []
ERROR 0382 SCAN: 0# sgutil::stacktrace::current[abi:cxx11]() in /usr/local/ldr/ldr
ERROR 0382 SCAN: 1# CXD_Debug_PrintStackCrawl() in /usr/local/ldr/ldr
ERROR 0382 SCAN: 2# Signal_FatalHandler(int) in /usr/local/ldr/ldr
ERROR 0382 SCAN: 3# 0x00007FCB90314730 in /lib/x86_64-linux-gnu/libpthread.so.0
ERROR 0382 SCAN: 4# cassandra::ProductionObjectEventReader::event(cassandra::Event const&) in /usr/local/ldr/ldr
ERROR 0382 SCAN: 5# cassandra::ProductionObjectFactory::objectFromEvents(cassandra::lookup::Type const&, byc::ObjectKey, cassandra::EventStream&, cassandra::Object&) in /usr/local/ldr/ldr
ERROR 0382 SCAN: 6# 0x00000000015D24F3 in /usr/local/ldr/ldr
ERROR 0382 SCAN: 7# cassandra::ProductionObjectRepository::lookupObjectByUUID(CXD_UUID const&, cassandra::lookup::Type const&, cassandra::Object&, CassConsistency_) const in /usr/local/ldr/ldr
ERROR 0382 SCAN: 8# ILMScanner::leakedUUIDCleanup(cassandra::Cluster const&, cassandra::ObjectRepository&, cassandra::Object const&) in /usr/local/ldr/ldr
ERROR 0382 SCAN: 9# 0x00000000010EAFBB in /usr/local/ldr/ldr
ERROR 0382 SCAN: 10# ade::detail::asyncHelper<void>::operator()(std::function<void ()>)::{lambda()#1}::operator()() in /usr/local/ldr/ldr
ERROR 0382 SCAN: 11# ade_details::SpawnContext::processEntry() in /usr/local/ldr/ldr
ERROR 0382 SCAN: 12# void boost::context::detail::ecv2_context_etry<boost::context::detail::ecv2_record_void<boost::context::v2::execution_context<void>, ade_details::SpawnContext::CachingStackAllocator<boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >, ade_details::SpawnContext::CachingStackAllocator(unsigned int, std::function<void ()>, unsigned int, bool)::{lambda(boost::context::v2::execution_context<void>)#1}> >(boost::context::detail::transfer_t) in /usr/local/ldr/ldr
ERROR 0382 SCAN: 13# make_fcontext in /lib/x86_64-linux-gnu/libboost_context.so.1.67.0