Skip to main content
NetApp Knowledge Base

FabricPool ESXi SAN host remediation

  1. Last updated
    Sep 14, 2023
  2. Save as PDF
Views:
444
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
san
Last Updated:
9/14/2023, 7:54:24 AM

Applies to

  • ONTAP FabricPool
  • VMware ESXi and Guest OS hosted on ESXi
  • VMFS
  • RDMs
  • VVOLs

Issue

  • During ONTAP FabricPool S3 tiering outage, Guest OS hosted on ESXi might report:
    • I/O failures,
    • Read-only filesystems,
    • Guest OS crash, etc.
  • In some cases, VMFS datastore could go read-only/inaccessible.  ESXi vmkernel logs records this as medium errors and I/O failure for Guest OS.

Sample error message from ESXi is shown below:

ESXi Medium error

2023-07-03T17:26:35.282Z cpu20:1049308)ScsiDeviceIO: 4161: Cmd(0x4598f5de3cc8) 0x28, CmdSN 0x651 from world 1050920 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 1706672

2023-07-03T17:26:35.777Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5db6cc8) 0x28, CmdSN 0x653 from world 0 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 160384

2023-07-03T17:26:37.922Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5c203c8) 0x28, CmdSN 0x655 from world 0 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 160712

2023-07-03T17:26:37.922Z cpu36:1049358)Vol3: 4458: Failed to get object 28 type 3 uuid 643f357d-8ff50e8a-53be-4c52623393dc FD 5000644 gen 28 :I/O error

2023-07-03T17:26:40.391Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5db62c8) 0x28, CmdSN 0x657 from world 0 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 160760

2023-07-03T17:26:40.391Z cpu36:1049358)Vol3: 4458: Failed to get object 28 type 3 uuid 643f357d-8ff50e8a-53be-4c52623393dc FD 5c00644 gen 2b :I/O error

2023-07-03T17:26:41.783Z cpu8:1049303)ScsiDeviceIO: 4161: Cmd(0x4579011028c8) 0x28, CmdSN 0x658 from world 1048865 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 1278224

2023-07-03T17:26:41.783Z cpu7:1048865)WARNING: Vol3: 2963: 'FP_45GB_W2K19': Failed to clear journal address since JBC could not be Locked. This could result in leak of journal block at <type 6 addr 9>.

2023-07-03T17:26:41.783Z cpu7:1048865)WARNING: Vol3: 3032: 'FP_45GB_W2K19': Failed to clear journal address in on-disk HB. This could result in leak of journal block at <type 6 addr 9>.

2023-07-03T17:26:41.785Z cpu7:1048865)Vol3: 4367: Error closing the volume: . Eviction fails: Failure

2023-07-03T17:27:18.436Z cpu32:1049198)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.600a098038314b39335d554871744465 repeated 22 times

2023-07-03T17:27:23.777Z cpu32:1049302)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x2a (0x4598f5c49cc8, 1048870) to dev "naa.600a098038314b39335d554871744543" on path "vmhba3:C0:T1:L3" Failed:

2023-07-03T17:27:23.777Z cpu32:1049302)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc. Act:NONE. cmdId.initiator=0x4304d3426dc0 CmdSN 0x587

2023-07-03T17:27:23.777Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5c49cc8) 0x2a, CmdSN 0x587 from world 1048870 to dev "naa.600a098038314b39335d554871744543" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 17832200

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4072: 'linbtrfsboot': Fil6 file IO (<FD c19 r13>) : I/O error

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4038: ioCtx: 0x4598f597f8c0, world: 1048870, overallStatus: I/O error, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 2078, maxIOLength: 137246

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4044: numAddrs: 0, numTbz: 0, numPBsToAlloc: 0, numBlocksToAlloc: 0 numTBZBlocks: 0, numPBsToUnmap: 0, numBlocksToUnmap: 0

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4051: opType: 2, ioFlags: 393552, allocFlags: 0, ioStateTxn: 0, ioStateNoTxn: 7, txnState: 2, numResolvedAddrMaps: 1, numTxnPendingAddrMaps: 0

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4056: wakeupOnIOComplete: 0, isIOIssued: 1, isIODone: 1, isRead: 0 isSync: 1, canBlock: 1, lastPartialBlock: 0

2023-07-03T17:27:23.777Z cpu26:1048870)BC: 414: write to vmware.log (f532 28 3 64a149c0 dca15134 524cad38 dc933362 34004c4 17 0 0 0 0 0) 2078 bytes failed: I/O error

2023-07-03T17:27:23.777Z cpu26:1048870)Log: 1577: Generating backtrace for 1048870: bcflushd

2023-07-03T17:27:23.777Z cpu26:1048870)Backtrace for current CPU #26, worldID=1048870, fp=0x100126

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bad0:[0x420039ce8f55]LogBacktraceInt@vmkernel#nover+0xdd stack: 0x149a00, 0x420039c3e164, 0x45389ba1f140, 0x45388d41f140, 0x45388931f140

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931baf0:[0x420039c3e163]BCFileIO@vmkernel#nover+0x1b4 stack: 0x45388931f140, 0x842e31a8, 0x6c2e657261776d76, 0x45388900676f, 0x45389ba1f140

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bd60:[0x420039c3eb5b]BCWriteBuffer@vmkernel#nover+0x1fc stack: 0x149a00, 0xffffffffffffffff, 0x453800000000, 0xe1000, 0x12b734000

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bf70:[0x420039c3ecc7]BCFlushFile@vmkernel#nover+0x44 stack: 0x430707815f10, 0x41fff9d527e0, 0x430707815fd0, 0x420039c3eda7, 0x45388931f000

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bfa0:[0x420039c3eda6]BCFlushAll@vmkernel#nover+0xc3 stack: 0x45388931f140, 0x0, 0x0, 0x420039c3edf9, 0x0

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bfd0:[0x420039c3edf8]BCFlusherWorld@vmkernel#nover+0x9 stack: 0x0, 0x0, 0x0, 0x420039cc4a60, 0x0

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bfe0:[0x420039fb1871]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x420039cc4a60, 0x0, 0x0, 0x0

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931c000:[0x420039cc4a5f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0

2023-07-03T17:27:59.160Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5c7e4c8) 0x2a, CmdSN 0x598 from world 1048870 to dev "naa.600a098038314b39335d554871744543" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 17832200

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4072: 'linbtrfsboot': Fil6 file IO (<FD c19 r13>) : I/O error

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4038: ioCtx: 0x4598f597f8c0, world: 1048870, overallStatus: I/O error, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 2078, maxIOLength: 137246

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4044: numAddrs: 0, numTbz: 0, numPBsToAlloc: 0, numBlocksToAlloc: 0 numTBZBlocks: 0, numPBsToUnmap: 0, numBlocksToUnmap: 0

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4051: opType: 2, ioFlags: 393552, allocFlags: 0, ioStateTxn: 0, ioStateNoTxn: 7, txnState: 2, numResolvedAddrMaps: 1, numTxnPendingAddrMaps: 0

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4056: wakeupOnIOComplete: 0, isIOIssued: 1, isIODone: 1, isRead: 0 isSync: 1, canBlock: 1, lastPartialBlock: 0

2023-07-03T17:27:59.160Z cpu26:1048870)BC: 414: write to vmware.log (f532 28 3 64a149c0 dca15134 524cad38 dc933362 34004c4 17 0 0 0 0 0) 2078 bytes failed: I/O error

2023-07-03T17:27:59.160Z cpu26:1048870)Log: 1577: Generating backtrace for 1048870: bcflushd

2023-07-03T17:27:59.160Z cpu26:1048870)Backtrace for current CPU #26, worldID=1048870, fp=0x100126

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bad0:[0x420039ce8f55]LogBacktraceInt@vmkernel#nover+0xdd stack: 0x149a00, 0x420039c3e164, 0x45388d41f140, 0x453898e9f140, 0x45388931f140

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931baf0:[0x420039c3e163]BCFileIO@vmkernel#nover+0x1b4 stack: 0x45388931f140, 0xf8cda194, 0x6c2e657261776d76, 0x45388900676f, 0x45388d41f140

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bd60:[0x420039c3eb5b]BCWriteBuffer@vmkernel#nover+0x1fc stack: 0x149a00, 0xffffffffffffffff, 0x453800000000, 0x25000, 0x8a563f000

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bf70:[0x420039c3ecc7]BCFlushFile@vmkernel#nover+0x44 stack: 0x430707815f10, 0x41fff9d527e0, 0x430707815fd0, 0x420039c3eda7, 0x45388931f000

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bfa0:[0x420039c3eda6]BCFlushAll@vmkernel#nover+0xc3 stack: 0x45388931f140, 0x0, 0x0, 0x420039c3edf9, 0x0

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bfd0:[0x420039c3edf8]BCFlusherWorld@vmkernel#nover+0x9 stack: 0x0, 0x0, 0x0, 0x420039cc4a60, 0x0

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bfe0:[0x420039fb1871]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x420039cc4a60, 0x0, 0x0, 0x0

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931c000:[0x420039cc4a5f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0

 

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.