The SAS port went down followed by a software crash of the IOM modules
Applies to
- IOM6
- IOM12
Issue
- The SAS port went down, which was followed by a software crash of the IOM modules:
node01
Sun Apr 20 04:49:31 [node01: asd_asyncd_3a: sas.port.down:debug]: SAS port "3b" went down.Sun Apr 20 04:50:25 [node01: dsa_worker1: ses.exceptionShelfLog:debug]: Retrieving Exception SES Shelf Log information on channel 3b IOM module B disk shelf ID 1.Sun Apr 20 04:50:25 [node01: dsa_worker3: ses.IOMLogCrash:debug]: Encountered a crash in Exception Log on IOM module B SN [xxxxxxxxxxxx] FW 0141, channel 3b.01.99 shelf ID 1 shelf SN [SHxxxxxxxxxxxx].Sun Apr 20 04:50:25 [node01: dsa_worker3: ses.IOMLogCrash:debug]: Encountered a crash in Exception Log on IOM module B SN [xxxxxxxxxxxx] FW 0141, channel 3b.01.99 shelf ID 1 shelf SN [SHxxxxxxxxxxxx].Sun Apr 20 04:50:34 [node01: dsa_worker1: ses.exceptionShelfLog:debug]: Retrieving Exception SES Shelf Log information on channel 3b IOM module B disk shelf ID 0.Sun Apr 20 04:50:34 [node01: dsa_worker2: ses.IOMLogCrash:debug]: Encountered a crash in Exception Log on IOM module B SN [xxxxxxxxxxxx] FW 0141, channel 3b.00.99 shelf ID 0 shelf SN [SHxxxxxxxxxxxx].Sun Apr 20 04:50:34 [node01: dsa_worker2: ses.IOMLogCrash:debug]: Encountered a crash in Exception Log on IOM module B SN [xxxxxxxxxxxx] FW 0141, channel 3b.00.99 shelf ID 0 shelf SN [SHxxxxxxxxxxxx].Sun Apr 20 04:50:35 [node01: storlog_admin: sla.shelf.mod.reboot.unexp:error]: Unexpected reboot event reported by module B in shelf: 3b.01.99.1, log: Thu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 02000093; U?; HAL; hal; 04; Module Reboot: Startup type 5-Crash resetSun Apr 20 04:51:02 [node01: storlog_admin: sla.shelf.mod.reboot.unexp:error]: Unexpected reboot event reported by module B in shelf: 3b.00.99.0, log: Thu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 02000093; U?; HAL; hal; 04; Module Reboot: Startup type 5-Crash resetnode02
Sun Apr 20 04:49:54 [node02: asd_asyncd_0: sas.port.down:debug]: SAS port "0a" went down.SHELF LOG:
--------------------------------------------------------------Shelflog start time: Thu Jul 25 06:52:36 GMT 2024Controller Id: xxxxxxxxxChannel: 3b Shelf: 1 Module type: IOM12B Firmware rev: 0141Shelf product id: DS46012IOM12Shelf Serial Number: SHxxxxxxxxxxxxModule B Serial Number: xxxxxxxxxxxxLog ID: SHxxxxxxxxxxxx_B_1eTimestamp: Sun Apr 20 04:50:35 2025--------------------------------------------------------------EVENT LOGSTimestamp Sat Apr 19 19:53:17 2025 (271+20:33:55.254)Thu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 02000027; U?; HAL; hal; 04; NDU Data Invalid.Thu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 02000093; U?; HAL; hal; 04; Module Reboot: Startup type 5-Crash resetThu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 02000094; U?; HAL; hal; 04; NMI_Type:badc0de4 CAUSE:0x00000004 PC:0x5a08b3b4Thu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 020001F6; U?; HAL; hal; 04; Module Reboot: Latched power registers - EBOD = FF, EXT EBOD = 01Thu Jan 1 00:00:00 1970 ( 0+00:00:00.191); 0200023F; U?; HAL; hal; 02; BoardType P3 designThu Jan 1 00:00:00 1970 ( 0+00:00:00.193); 0200011E; U?; HAL; hal; 02; Canister CPLD POST passedThu Jan 1 00:00:00 1970 ( 0+00:00:00.193); 02000120; U?; HAL; hal; 02; Canister CPLD: V40Thu Jan 1 00:00:00 1970 ( 0+00:00:00.193); 020001E3; U?; HAL; hal; 02; Sideband CPLD: V11Thu Jan 1 00:00:00 1970 ( 0+00:00:00.243); 01000001; U?; CLI; cli; 02; Failed to bind CLI command: gpio_get_status, status 4Thu Jan 1 00:00:00 1970 ( 0+00:00:00.252); 0200013C; U?; HAL; hal; 02; Reboot after software crash.Thu Jan 1 00:00:00 1970 ( 0+00:00:00.252); 0200013D; U?; HAL; hal; 02; Failure info:Cause:00000001[Undefined] PC:00236ebc Thread(current):HID Pulse Generator ThreadThu Jan 1 00:00:00 1970 ( 0+00:00:00.252); 0200013E; U?; HAL; hal; 02; Failed firmware version: 0141, Code Image BThu Jan 1 00:00:00 1970 ( 0+00:00:00.252); 020001E3; U?; HAL; hal; 02; Sideband CPLD: V11Thu Jan 1 00:00:00 1970 ( 0+00:00:01.992); 02000244; U?; HAL; hal; 02; Midplane VPD Customer Region CRC: 0x73DB9F02Thu Jan 1 00:00:00 1970 ( 0+00:00:01.993); 020001AE; U?; HAL; hal; 02; Midplane VPD CRC: 0xC1258521Thu Jan 1 00:00:00 1970 ( 0+00:00:01.993); 02000102; U?; HAL; hal; 02; Midplane VPD Structure: 0x0DThu Jan 1 00:00:00 1970 ( 0+00:00:03.190); 020001AF; U?; HAL; hal; 02; Canister VPD CRC: 0x6ECA151DThu Jan 1 00:00:00 1970 ( 0+00:00:03.191); 020000BC; U?; HAL; hal; 02; Canister VPD Structure: 0x0BThu Jan 1 00:00:00 1970 ( 0+00:00:03.195); 020001FD; U?; HAL; hal; 04; Invalid customer logging mode area. Default to FILL-STOP modeThu Jan 1 00:00:00 1970 ( 0+00:00:03.198); 02000188; U?; HAL; hal; 02; Bootloader: Version 01.00Thu Jan 1 00:00:00 1970 ( 0+00:00:03.200); 020000C4; U?; HAL; hal; 02; EBOD FW: V0141