CHW-316: AFF A800 FCPr UMCE panic PCIe device due to motherboard onboard switch
Issue
- AFF A800, AFF C800, ASA A800 or ASA C800 node reboots unexpectedly with an UMCE panic similar to:
PANIC: Uncorrectable Machine Check Error at CPU45. SKL_IIO Error: STATUS<0xbb80000000000e0b>(VALID,UC,EN,MISCV,PCC,S,AR,CORR_ERR_STATUS(0),CORR_ERR_CNT(0),MSCOD(0),MCACOD(0xe0b))MISC<0x00000000ae000000>(UCR_BUS_LOG(174),UCR_DEVICE_LOG(0),UCR_FUNCTION_LOG(0),UCR_SEGMENT_LOG(0))IIO Machine Check from device(s):IIO Machine Check from device(s):RPT(174,0,0):ErrSrcID(CorrSrc(0xaf00),UCorrSrc(0xb100)), PLX PCIE 8749 switch on Controller, PCI Device 1425:600d in slot 1 on Controller, PCI Device 1425:600d in slot 1 on Controller, PCI Device 1425:600d in slot 1 on Controller, PCI Device 1425:600d in slot 1 on Controller, T62100-CR Dual 40/100G NIC in slot 1 on Controller, PCI Device 1425:650d in slot 1 on Controller, PCI Device 1425:660d in slot 1 on Controller. ,. in process idle: cpu45 on release 9.15.1P11 (C)
- SSRAM-LOG indicates the uncorrectable error is due to Flow Control Protocol (FCPr) errors on each device in the slot:
SRAM record type(LOG) from Data ONTAP: APL,1_0. GFF<0x00000040>; IIO6: LFERR<0x00001000>(PCIE), LFFERR<0x00001000>(PCIE);
RPT(174,0,0): GLB<0x00000001>(FERR), PTR<0x0>,
Br[8749](175,0,0): RcvErr(P8(40));
Br[8749](176,8,0): DevStatus(Corr), CorrErr(Rcvr);
Dv[600d](177,0,0) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr);
Dv[600d](177,0,1) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr);
Dv[600d](177,0,2) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr);
Dv[600d](177,0,3) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr);
Dv[640d](177,0,4) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr);
Dv[650d](177,0,5) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr);
Dv[660d](177,0,6) in slot 1: Status(SigSysErr), DevStatus(Fatal), UCorrErr(FCPr), FirstUCorrErr(FCPr).