Detected firmware fault 0xffffffff on SAS adapter 0c
Applies to
- FAS 80x0
- ONTAP 9
Issue
- SAS adapter has detected a firmware fault
- SAS adapter resets itself and recovers automatically
Mon Jan 30 03:31:45 [node01: pmcsas_timeout_1: sas.adapter.firmware.fault:notice]: Detected firmware fault 0xffffffff on SAS adapter 0c.
Mon Jan 30 03:31:48 [node01: pmcsas_timeout_1: sas.adapter.exception:debug]: SAS adapter driver encountered error "Failed to get a response in state 43" on adapter 0c.
Mon Jan 30 03:31:48 [node01: pmcsas_asyncd_1: sas.adapter.exception:debug]: SAS adapter driver encountered error "ASYNC SMP timed out -- ABORTING" on adapter 0c.
Mon Jan 30 03:31:49 [node01: pmcsas_timeout_1: sas.device.quiesce:debug]: Adapter 0c encountered a command timeout on disk device 0c.40.0. Quiescing the device.
Mon Jan 30 03:31:54 [node01: pmcsas_timeout_1: sas.adapter.exception:debug]: SAS adapter driver encountered error "Failed to get a response in state 46" on adapter 0c.
Mon Jan 30 03:31:54 [node01: pmcsas_asyncd_1: sas.adapter.exception:debug]: SAS adapter driver encountered error "ASYNC SMP ABORT timed out" on adapter 0c.
Mon Jan 30 03:31:54 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Adapter debug dump is being collected', 'adapterName': '0c'}
Mon Jan 30 03:31:55 [node01: pmcsas_asyncd_1: sas.adapter.reset:debug]: Resetting SAS adapter 0c.
Mon Jan 30 03:31:55 [node01: pmcsas_asyncd_1: sas.adapter.not.ready:error]: SAS adapter 0c did not become ready.
Mon Jan 30 03:31:59 [node01: pmcsas_timeout_1: sas.adapter.debug:info]: params: {'debug_string': "PHYs didn't come up.", 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Port 0: disabled 0, up 0, down 4: old state 3 --> new state 1', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Port 1: disabled 0, up 0, down 4: old state 3 --> new state 1', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [0] RATE 0x4 PS 0x1, NPIP 1 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098007ae05ff', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PORT UP -- 0c', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [1] RATE 0x4 PS 0x1, NPIP 2 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098007ae05ff', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [2] RATE 0x4 PS 0x1, NPIP 3 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098007ae05ff', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [3] RATE 0x4 PS 0x1, NPIP 4 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098007ae05ff', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [4] RATE 0x4 PS 0x1, NPIP 1 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098008004a7f', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PORT UP -- 0d', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [5] RATE 0x4 PS 0x1, NPIP 2 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098008004a7f', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [6] RATE 0x4 PS 0x1, NPIP 3 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098008004a7f', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_admin_1: sas.adapter.debug:info]: params: {'debug_string': 'PHY UP [7] RATE 0x4 PS 0x1, NPIP 4 :: DEVTYPE 0x20, TGT 0x2, INIT 0x2, ADDR 500a098008004a7f', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Port 0: disabled 0, up 4, down 0: old state 1 --> new state 3', 'adapterName': '0c'}
Mon Jan 30 03:32:00 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Port 1: disabled 0, up 4, down 0: old state 1 --> new state 3', 'adapterName': '0c'}
Mon Jan 30 03:32:03 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Transition Comparing this: [IOM6] and next: [IOM6] ...', 'adapterName': '0c'}
Mon Jan 30 03:32:03 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'Asyncd device scan done.', 'adapterName': '0c'}
Mon Jan 30 03:32:03 [node01: pmcsas_asyncd_1: sas.adapter.debug:info]: params: {'debug_string': 'device_count: expected 144 (144), registered 144', 'adapterName': '0c'}
Mon Jan 30 03:32:04 [node01: pmcsas_timeout_1: sas.adapter.debug:info]: params: {'debug_string': 'Stop kicking open SDM batch window.', 'adapterName': '0c'}
- SAS adapter firmware version is 01.12.09.00
slot 0: SAS Host Adapter 0c (PMC-Sierra PM8001 rev. C, SAS, <UP>)
Firmware rev: 01.12.09.00
Base WWN: 5:xxxxxx:xxxxxxx:xx
Phy State: [0] Enabled, 6.0 Gb/s
[1] Enabled, 6.0 Gb/s
[2] Enabled, 6.0 Gb/s
[3] Enabled, 6.0 Gb/s
QSFP Vendor: Amphenol
QSFP Part Number: X6594-R6+A0
QSFP Type: Passive Copper 0.5m ID:01
QSFP Serial Number: xxxxxxxxxxxxx