FlexCache Volumes Disconnected After Firewall Upgrade
Applies to
- ONTAP 9
- FlexCache
- CIFS/SMB2
Issue
- After a firewall upgrade by a third party, FlexCache volumes between two sites became disconnected
- Users experienced CIFS/SMB timeouts and were unable to create or modify files/folders on FlexCache shares
- Windows clients displayed the error:
An unexpected error is keeping you from creating the folder. Error 0x8007054F: An internal error occurred
- CIFS Operation Timeout errors seen in ONTAP:
[cache-01: kernel: Nblade_CifsOperationTimedOut_1:error]: params: {'commandName': 'SMB2_COM_CREATE', 'suspensionCnt': '10640', 'cmdRestartCnt': '0', 'lastCsmError': 'CSM_OK', 'remoteBladeID': 'da0e58ff-62a8-11f0-9515-d039ead6e8ad (cache-02)', 'isQosEnabled': 'QoS_disabled', 'lastSpinNpError': '3010', 'clientIpAddress': '10.1.2.9', 'localIpAddress': '10.1.1.14', 'vserverId': '5', 'dsId': '1027', 'vserverName': 'cache_svm', 'volumeName': 'Volume Name unavailable'}[cache-01: kernel: ems.engine.suppressed:debug]: Event 'Nblade.CifsOperationTimedOut' suppressed 15 times in last 2877 seconds.[cache-01: kernel: Nblade_CifsOperationTimedOut_1:error]: params: {'commandName': 'SMB2_COM_QUERY_DIRECTORY', 'suspensionCnt': '10640', 'cmdRestartCnt': '0', 'lastCsmError': 'CSM_OK', 'remoteBladeID': 'da0e58ff-62a8-11f0-9515-d039ead6e8ad (cache-02)', 'isQosEnabled': 'QoS_disabled', 'lastSpinNpError': '3010', 'clientIpAddress': '10.1.2.60', 'localIpAddress': '10.1.1.14', 'vserverId': '5', 'dsId': '1034', 'vserverName': 'cache_svm', 'volumeName': 'data01_cache__0008'}[cache-01: kernel: ems.engine.suppressed:debug]: Event 'Nblade.CifsOperationTimedOut' suppressed 3 times in last 475 seconds.- CSM/Intercluster session errors:
- Cache :
[cache-01: BsdThreadPoolThread: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-01:dblade, uniquifier=0e0651a89f74334e) with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.[cache-01: BsdThreadPoolThread: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-02:dblade, rsp=cache-01:dblade, uniquifier=0e0651a8a0a61da7) with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.[cache-01: kernel: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-02:dblade, rsp=cache-01:dblade, uniquifier=0e0651a8a60b8e7d) with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.[cache-01: kernel: csm.sessionFailed:debug]: Cluster interconnect session (req=cache-01:dblade, rsp=origin-01:dblade, uniquifier=0e0651a8a83c2b4b) failed with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.[cache-01: BsdThreadPoolThread: csm.sessionFailed:debug]: Cluster interconnect session (req=cache-01:dblade, rsp=origin-02:dblade, uniquifier=0e0651a8a83df059) failed with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.[cache-01: BsdThreadPoolThread: csm.sessionFailed:debug]: Cluster interconnect session (req=cache-01:dblade, rsp=origin-01:dblade, uniquifier=0e0651a8a840eddb) failed with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.[cache-01: kernel: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-01:dblade, uniquifier=0e0651a8b19f3a3b) with transport type CT, session tag CPEER, record state ACTIVE, CSM error CSM_CONNABORTED, low-level error , socket error 0, and TLS error 2113929222.-
- Origin:
[origin-01: CSM_BTLS: csm.connectionFailed:debug]: CSM failed to create a connection: localBladeUUID = origin-01:dblade, remoteBladeUUID = cache-01:dblade, uniquifier = 110651a8bfef61cd, transportType = UNASSIGNED, sessionTag = WAFL_REMOTE, localVifId = 1064, remoteVifIP = 10.1.1.207, CsmError = CSM_EXISTS, ctLoError = CTLO_ERR_UNKNOWN, socketError = 0, and TLSerror = 0.[origin-01: CSM_BTLS: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-01:dblade, uniquifier=110651a8bfef61cd) with transport type UNASSIGNED, session tag WAFL_REMOTE, record state STARTING, CSM error CSM_EXISTS, low-level error CTLO_ERR_OK, socket error 0, and TLS error 0.[origin-01: CSM_BTLS: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-02:dblade, uniquifier=110651a8c05a7f68) with transport type UNASSIGNED, session tag WAFL_REMOTE, record state STARTING, CSM error CSM_EXISTS, low-level error CTLO_ERR_OK, socket error 0, and TLS error 0.[origin-01: CSM_BTLS: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-01:dblade, uniquifier=110651a8c1ac780e) with transport type UNASSIGNED, session tag WAFL_REMOTE, record state STARTING, CSM error CSM_EXISTS, low-level error CTLO_ERR_OK, socket error 0, and TLS error 0.[origin-01: CSM_BTLS: csm.connectionFailed:debug]: CSM failed to create a connection: localBladeUUID = origin-01:dblade, remoteBladeUUID = cache-02:dblade, uniquifier = 110651a8c2177e3b, transportType = UNASSIGNED, sessionTag = WAFL_REMOTE, localVifId = 1064, remoteVifIP = 10.1.1.209, CsmError = CSM_EXISTS, ctLoError = CTLO_ERR_UNKNOWN, socketError = 0, and TLSerror = 0.[origin-01: CSM_BTLS: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-02:dblade, uniquifier=110651a8c2177e3b) with transport type UNASSIGNED, session tag WAFL_REMOTE, record state STARTING, CSM error CSM_EXISTS, low-level error CTLO_ERR_OK, socket error 0, and TLS error 0.[origin-01: CSM_BTLS: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-01:dblade, uniquifier=110651a8c36853dd) with transport type UNASSIGNED, session tag WAFL_REMOTE, record state STARTING, CSM error CSM_EXISTS, low-level error CTLO_ERR_OK, socket error 0, and TLS error 0.[origin-01: CSM_BTLS: csm.createSessionFailed:debug]: Cluster Session Manager (CSM) failed to create session (req=origin-01:dblade, rsp=cache-02:dblade, uniquifier=110651a8c3d37d32) with transport type UNASSIGNED, session tag WAFL_REMOTE, record state STARTING, CSM error CSM_EXISTS, low-level error CTLO_ERR_OK, socket error 0, and TLS error 0.