Skip to main content
NetApp Knowledge Base

"Waiting for cluster applications to come online on the local node" displays after takeover and cannot give back

Views:
6,944
Visibility:
Public
Votes:
2
Category:
ontap-9
Specialty:
core
Last Updated:

Applies to

  • FAS2650
  • ONTAP 9
  • Service Processor (SP)

Issue

  • After takeover occurs on the system, the message "Waiting for cluster applications to come online on the local node." is displayed and cannot give back.

Example:

::> storage failover show
              Takeover          
Node           Partner        Possible State Description  
-------------- -------------- -------- -------------------------------------
fas2650-a               fas2650-b   true     Connected to fas2650-b      

fas2650-b               fas2650-a   true     Connected to fas2650-a.  Waiting for cluster applications to come online on the local node.

2 entries were displayed.

cluster1::> set diag

cluster1::*> cluster kernel-service show

cluster1::*> cluster kernel-service show
Master            Cluster           Quorum Availability  Operational
Node              Node              Status        Status        Status
----------------- ----------------- ------------- ------------- -------------
fas2650-a   fas2650-a   in-quorum     true operational
                      fas2650-b   out-of-quorum false         unknown
fas2650-b   fas2650-a out-of-quorum false         operational
                      fas2650-b   out-of-quorum true          operational
4 entries were displayed.


cluster1::*> cluster kernel-service node show
                 Is     Cluster
     Quorum        Availability Operational
Node             Master Node             Status        Status Status
---------------- ------ ---------------- ------------- ------------ -----------
fas2650-a  true   fas2650-a in-quorum     true         operational
                                fas2650-b  out-of-quorum false unknown
fas2650-b  true   fas2650-a  out-of-quorum false        operational
 fas2650-b  out-of-quorum true         operational
4 entries were displayed.

cluster1::> storage failover giveback -ofnode fas2650-b -override-vetoes true

Warning: Initiating a giveback with vetoes overridden will result in giveback
        proceeding even if the node detects outstanding issues that would make
         a giveback dangerous or disruptive. Do you want to continue?
          {y|n}: y

Error: command failed: Failed to initiate giveback. Reason: A giveback is
       already in progress. 

  • EMS log appears following "bsdsocket" errors and "SPLinkDownAlert" message.

Example:

  [?] Fri Jul 10 18:34:40 JST [fas2650-a: kernel: bsdsocket.queueLimit.reached:error]: Number of connection requests for the socket bound to port 935 having the IP address * has reached the limit of 1536 connections. The socket uses TCP protocol. This event has occurred 62 times since the last warning.
  [?] Fri Jul 10 18:35:16 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:40:00 JST [fas2650-a: kernel: bsdsocket.queueLimit.reached:error]: Number of connection requests for the socket bound to port 935 having the IP address * has reached the limit of 1536 co
nnections. The socket uses TCP protocol. This event has occurred 49 times since the last warning.
  [?] Fri Jul 10 18:40:41 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:45:13 JST [fas2650-a: spmd: spm.servprocd.process.exit:alert]: Service processor daemon (servprocd) with ID 4174 exited as a result of normal exit (1). This server will attempt to restart.
  [?] Fri Jul 10 18:45:15 JST [fas2650-a: servprocd: mdb.recovery.successful:debug]: params: {'name': 'servprocd'}
  [?] Fri Jul 10 18:46:06 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:50:19 JST [fas2650-a: nphmd: hm.alert.cleared:info]: params: {'alert_id': 'SPLinkDownAlert', 'alerting_resource': 'SP Config', 'detailed_info': 'Alert Id = SPLinkDownAlert , Alerting Resource = SP Config', 'monitor': 'controller'}

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.