Skip to main content
NetApp Knowledge Base

"Waiting for cluster applications to come online on the local node" displays after takeover and cannot give back

Views:
3,050
Visibility:
Public
Votes:
2
Category:
fas-systems
Specialty:
hw
Last Updated:

Applies to

  • FAS2650
  • ONTAP 9
  • Service Processor (SP)

Issue

  • After takeover occurs on the system, the message "Waiting for cluster applications to come online on the local node." is displayed and cannot give back.

Example:

::> storage failover show
              Takeover          
Node           Partner        Possible State Description  
-------------- -------------- -------- -------------------------------------
fas2650-a               fas2650-b   true     Connected to fas2650-b      

fas2650-b               fas2650-a   true     Connected to fas2650-a.  Waiting for cluster applications to come online on the local node.

2 entries were displayed.

cluster1::> set diag

cluster1::*> cluster kernel-service show

cluster1::*> cluster kernel-service show
Master            Cluster           Quorum Availability  Operational
Node              Node              Status        Status        Status
----------------- ----------------- ------------- ------------- -------------
fas2650-a   fas2650-a   in-quorum     true operational
                      fas2650-b   out-of-quorum false         unknown
fas2650-b   fas2650-a out-of-quorum false         operational
                      fas2650-b   out-of-quorum true          operational
4 entries were displayed.


cluster1::*> cluster kernel-service node show
                 Is     Cluster
     Quorum        Availability Operational
Node             Master Node             Status        Status Status
---------------- ------ ---------------- ------------- ------------ -----------
fas2650-a  true   fas2650-a in-quorum     true         operational
                                fas2650-b  out-of-quorum false unknown
fas2650-b  true   fas2650-a  out-of-quorum false        operational
 fas2650-b  out-of-quorum true         operational
4 entries were displayed.

cluster1::> storage failover giveback -ofnode fas2650-b -override-vetoes true

Warning: Initiating a giveback with vetoes overridden will result in giveback
        proceeding even if the node detects outstanding issues that would make
         a giveback dangerous or disruptive. Do you want to continue?
          {y|n}: y

Error: command failed: Failed to initiate giveback. Reason: A giveback is
       already in progress. 

  • EMS log appears following "bsdsocket" errors and "SPLinkDownAlert" message.

Example:

  [?] Fri Jul 10 18:34:40 JST [fas2650-a: kernel: bsdsocket.queueLimit.reached:error]: Number of connection requests for the socket bound to port 935 having the IP address * has reached the limit of 1536 connections. The socket uses TCP protocol. This event has occurred 62 times since the last warning.
  [?] Fri Jul 10 18:35:16 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:40:00 JST [fas2650-a: kernel: bsdsocket.queueLimit.reached:error]: Number of connection requests for the socket bound to port 935 having the IP address * has reached the limit of 1536 co
nnections. The socket uses TCP protocol. This event has occurred 49 times since the last warning.
  [?] Fri Jul 10 18:40:41 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:45:13 JST [fas2650-a: spmd: spm.servprocd.process.exit:alert]: Service processor daemon (servprocd) with ID 4174 exited as a result of normal exit (1). This server will attempt to restart.
  [?] Fri Jul 10 18:45:15 JST [fas2650-a: servprocd: mdb.recovery.successful:debug]: params: {'name': 'servprocd'}
  [?] Fri Jul 10 18:46:06 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:50:19 JST [fas2650-a: nphmd: hm.alert.cleared:info]: params: {'alert_id': 'SPLinkDownAlert', 'alerting_resource': 'SP Config', 'detailed_info': 'Alert Id = SPLinkDownAlert , Alerting Resource = SP Config', 'monitor': 'controller'}

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

Scan to view the article on your device