StorageGRID bare metal node reboots unexpectedly due to hung task timeout
Applies to
- StorageGRID bare metal
- RedHat Linux
Issue
- StorageGRID bare metal node reboots unexpectedly.
- No error logs found in Lumberjack.
- Multiple tasks(ldr/java/ssm) panic found in kernel.log of Linux due to hung task timeout.
Sep 19 13:38:30 dxl-09 kernel: INFO: task ldr:30362 blocked for more than 120 seconds.Sep 19 13:38:30 dxl-09 kernel: Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1Sep 19 13:38:30 dxl-09 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.Sep 19 13:38:30 dxl-09 kernel: task:ldr state:D stack: 0 pid:30362 ppid: 29868 flags:0x00000080Sep 19 13:38:30 dxl-09 kernel: Call Trace:Sep 19 13:38:30 dxl-09 kernel: __schedule+0x2d1/0x870Sep 19 13:38:30 dxl-09 kernel: schedule+0x55/0xf0Sep 19 13:38:30 dxl-09 kernel: schedule_preempt_disabled+0xa/0x10Sep 19 13:38:30 dxl-09 kernel: rwsem_down_read_slowpath+0x26e/0x3f0Sep 19 13:38:30 dxl-09 kernel: down_read+0x95/0xa0Sep 19 13:38:30 dxl-09 kernel: __do_page_fault+0x373/0x450Sep 19 13:38:30 dxl-09 kernel: do_page_fault+0x37/0x130Sep 19 13:38:30 dxl-09 kernel: ? page_fault+0x8/0x30Sep 19 13:38:30 dxl-09 kernel: page_fault+0x1e/0x30Sep 19 13:38:30 dxl-09 kernel: RIP: 0033:0x1b0ca22Sep 19 13:38:30 dxl-09 kernel: Code: Unable to access opcode bytes at RIP 0x1b0c9f8.Sep 19 13:38:30 dxl-09 kernel: RSP: 002b:00007f1cd2b16cf0 EFLAGS: 00010246Sep 19 13:38:30 dxl-09 kernel: RAX: 00007f1cb6920f40 RBX: 00007f1cd2b16d38 RCX: 00007f1cb6920f70Sep 19 13:38:30 dxl-09 kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f1d6fdecef8Sep 19 13:38:30 dxl-09 kernel: RBP: 0000000000010000 R08: 0000000000013fff R09: 00007f1d78d31000Sep 19 13:38:30 dxl-09 kernel: R10: 00007f1cbae000c0 R11: 0000000000000080 R12: 0000000001c22c37Sep 19 13:38:30 dxl-09 kernel: R13: 00007f1d34e04be0 R14: 0000000044415441 R15: 00000000000001eb
