Skip to main content
NetApp Knowledge Base

CONTAP-629364: Quorum loss during CFO takeover due to slow Flashcache destaging

  1. Last updated
  2. Save as PDF
Views:
1
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
CORE
Last Updated:

Issue

  • One node with Flashcache device, member of a FAS 2-node-cluster, is taken over negotiated ("storage failover takeover")
  • SFO takeover phase completes successful
  • Subsequently CFO takeover phase (node shutdown) takes longer than 65 seconds, noticeable by the takeover "ofnode" console not completing shutdown but hanging more than 65 seconds after printing "Uptime: ..." during its shutdown phase on its console before it reboots
  • Initially the takeover "fromnode" remains in-quorum when CFO takeover has started:

    MyCluster::*> storage failover takeover -ofnode MyCluster-01
    MyCluster::*> cluster ring show
    Node UnitName Epoch DB Epoch DB Trnxs Master Online
    --------- -------- -------- -------- -------- --------- ---------
    Warning: Unable to list entries on node MyCluster-01. RPC: Couldn't make
             connection [from mgwd on node "MyCluster-02" (VSID: -1) to mgwd at
             169.254.116.233]
    MyCluster-02 mgmt 28 28 454 MyCluster-02 master
    MyCluster-02 vldb 28 28 58 MyCluster-02 master
    MyCluster-02 vifmgr 28 28 173 MyCluster-02 master
    MyCluster-02 bcomd 28 28 9 MyCluster-02 master
    MyCluster-02 crs 28 28 1 MyCluster-02 master
    5 entries were displayed.

  • 65 seconds after CFO takeover had started but node shutdown did not complete yet, the takeover "fromnode" cluster RDB rings transition "offline":

    MyCluster::*> cluster ring show
    Node UnitName Epoch DB Epoch DB Trnxs Master Online
    --------- -------- -------- -------- -------- --------- ---------
    Warning: Unable to list entries on node MyCluster-01. RPC: Couldn't make
             connection [from mgwd on node "MyCluster-02" (VSID: -1) to mgwd at
             169.254.116.233]
    MyCluster-02 mgmt 0 28 454 - offline
    MyCluster-02 vldb 0 28 58 - offline
    MyCluster-02 vifmgr 0 28 173 - offline
    MyCluster-02 bcomd 0 28 9 - offline
    MyCluster-02 crs 0 28 1 - offline
    5 entries were displayed.

  • At this moment the takeover "fromnode" drops all LIF and clients cannot access data anymore:

    MyCluster::> event log show
    Fri Mar 13 13:13:13 GMT [MyCluster-02: vifmgr: vifmgr.lifBeingRemoved:notice]: LIF MyNFSLIF (on virtual server 3), IP address 10.13.13.13, is being removed from node MyCLuster-02, port a0a-13.

  • Once the takeover "ofnode" completed shutdown, the cluster rings on takeover "fromnode" come online again and data is served again

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.