How to troubleshoot frontend SAN issues using the fcHosts 3 command (part of supportdata collection)
This article describes how the
fcHosts 3 shell command can be used to find the bad component in a frontend SAN if too many bad FC frames are received. In such a case, the controller logs one of the following events in the Major Event Log (MEL):
0x1207 Fibre channel link errors - threshold exceeded
0x1206 Fibre channel link errors continue
The thresholds are defined in NVSRAM in Offset 0x38. The rule of thumb is, if the event mentioned above is seen in the Event Log, the user should also see an impact on the affected server(s), and therefore, the issue should be investigated.
fcHosts 3shell command is very old but a useful command. It is part of the supportdata collection (captured in
statecapturedata.txt) if an FC host connection is found.
- The output displays the history of the communication between the FC HBA and the controller port to which the HBA is logged.
- The maximum number of events that are listed is 50.
- The downside of the command is that the output lists only the time and not the date; therefore, the events could have occurred days ago. However, in case of an issue, a lot of events are usually logged within a short period of time and, most of the time, it is from the present day the support data was captured.
The following is an example of the information provided:
Executing fcHosts(3,0,0,0,0,0,0,0,0,0) on controller A:
=============== HOST 10 =====================
Hst-Role(Ch) PortId PortWwn NodeWwn DstNPort CmdRecv Label
10-Host( 2) 0e0000 10008c7c-ff2057ba 20008c7c-ff2057ba 0eaabf80 195719 SRV-MDC2-HBA-P0
PERMITS: 0x00000008 HsdPort
FLAGS: 0x00001406 Plogi Prli LoginRcvd Analyze
LastActivity: 11/19/13-17:15:47 (GMT)
HOST LOG==> logCtl:0eab1540 logIndex: 5 goodIoCount:88836
dstNPort:0eaabf80 maxIndex: 50 logIoCount:1
RepeatCounts -- IO Types
Num Time LogCode Qualifier LogCode GoodIo Outstand
Cnt Type Cnt Type Cnt Type
1 11:58:40 First 00000000 1 ---- >100K RWO- 1 ----
2 13:45:47 RscnRecv 000e0000 1 ---- 1 ---- 1 ----
3 13:45:47 Logout RscnMis 1 ---- 1 ---- 1 ----
4 13:49:51 Login ff2057ba 1 ---- <100 R--- 1 ----
5 13:49:51 ChkCond 06290400 <10 R--- <100 R--- 1 R---
6 13:49:51 RscnRecv 000e0000 1 ---- <100K RWO- 1 ----
10-Host: 10 is the same as the ITN number in tditnall output of the same controller
Ch(X): 2 is the channel (Host Port) the HBA is logged into. Use fcAll/chall to find out the host port
PortId: 0e0000 is the 24 bit address of the switch port the HBA is connected to
PortWwn: 10008c7c-ff2057ba is the FC Port WWN of the HBA
NodeWwn: 20008c7c-ff2057ba is the FC Node WWN of the HBA
CmdRecv: 195719 shows how many SCSI command where received from this HBA
Label: SRV-MDC2-HBA-P0 is the Alias of the HBA defined in Santricity
Time: The Time when the event happened (there is no Date).
LogCode: The event that happened
Qualifier: A qualifies of the event in case of a check condition (ChkCond) it is the sense data
LogCodeCnt: Number of consecutive occurrences of this logCode event
GoodIo Cnt: Number of IOs returned with good status after 1st occurrence
Outst. Cnt: Number of outstanding IOs when 1st occurrence logged
The following is seen from the example above:
The HBA with WWPN 10008c7c-ff2057ba is connected to the FC switch port 0x0e0000. It is connected to the controller through channel 2 (use
fcAll/chall to find the real host port). The user has given the HBA in the Host mapping section of SANtricity the Alias SRV-MDC2-HBA-P0. From the History, it can be observed that the HBA First (beginning of the capture) sent multiple IOs without issues, and then sent a Rescan following an FC Port Logout and Login. The controller confirmed the Logout/Login by returning a check condition with a sense key of 06 asc 29 ascq 04 which decodes to "Device Internal Reset". The HBA then sent another Rescan. Overall, there is no indication of a communication issue between the HBA and the controller. A few Login/Logouts are usually not an issue.
List of LogCodes and Qualifiers
AbtsRecv= Session Abort received (is an indication of a path issue)
ChkCond= Controller send SCSI check condition (sense data) to HBA (see Qualifier for details)
First= Start of capture
GoodIo= HBA send good IO
Login= HBA did a Port Login into the controller
Logout= HBA logged out of the controller
LinkDown= Connection to the HBA is down
Qfull= Queue full condition met
ResvCon= Controller returned a reservation conflict to the HBA (could be normal in a cluster configuration!)
RscnRecv= HBA send a Rescan
ScsiStat= Other SCSI status occurred
Qualifiers (most common only)
Count= Count (Lowlevel FC error. Indication of a path issue)
FreezeTO= Freeze Timeout
Observed= Event observed
ReplyTO= Replay Timeout (Indication of a path issue)
RscnMis= Rescan device missing
The following are two examples of an HBA having issues talking to the controller:
RepeatCounts -- IO Types (R=read,W=write,O=other,N=nonScsi) Num Time LogCode Qualifier LogCode GoodIo Outstand Cnt Type Cnt Type Cnt Type42 15:20:40 GoodIo 00000000 1 ---- <100 -W-- 1 -W-- 43 15:21:16 SetError ReplyTO 1 -W-- <100 -W-- 1 -W-- 44 15:21:21 ChkCond 0b470000 <5 -W-- <100 -W-- <5 -W-- 45 15:21:26 SetError ReplyTO 1 -W-- <10 RW-- 1 -W-- 46 15:21:28 ChkCond 0b470000 <5 -W-- <100 -W-- 1 -W-- 47 15:22:36 AbtsRecv 00000000 1 -W-- <1K RW-- 1 -W-- 48 15:22:42 ChkCond 0b470000 <10 -W-- <1K RW-- 1 -W-- 49 15:23:11 GoodIo 00000000 1 ---- <10 -W-- 1 -W-- 50 15:23:12 ChkCond 0b470000 <10 -W-- <1K RW-- 1 -W--
The sense 0b/47/00 decodes to "SCSI Parity Error". In a FC work this means that the the controller received a FC frame with incorrect CRC.
42 17:13:11 SetError ReplyTO <5 -W-- <1K -W-- <5 -W-- 43 17:13:14 GoodIo 00000000 1 ---- <1K -W-- <10 -W-- 44 17:13:14 SetError ReplyTO <5 -W-- <1K -W-- <5 -W-- 45 17:13:16 GoodIo 00000000 1 ---- <1K -W-- <5 -W-- 46 17:13:19 SetError ReplyTO 1 -W-- <100K RW-- 1 -W-- 47 17:13:55 SetError Count <5 -W-- <1K -W-- 1 -W-- 48 17:13:56 GoodIo 00000000 1 ---- <1K -W-- <5 -W-- 49 17:14:05 SetError ReplyTO <5 -W-- <100 R--- <5 -W-- 50 17:14:12 GoodIo 00000000 1 ---- <100K RW-- 1 R---
Note: If there is an HBA with a lot of the above errors, it does NOT automatically mean the HBA is faulty. It means the fc frames are corrupted or dropped somewhere between this HBA and the Controller. A bad HBA is just one possible candidate causing the issue.