Skip to main content
NetApp Knowledgebase

"Waiting for cluster applications to come online on the local node" displays after takeover and cannot give back

Views:
372
Visibility:
Public
Votes:
1
Category:
fas-systems
Specialty:
hw
Last Updated:

Applies to

  • FAS2650
  • ONTAP 9
  • Service Processor (SP)

Issue

  • After takeover occurs on the system, the message "Waiting for cluster applications to come online on the local node." is displayed and cannot give back.

Example:

::> storage failover show
              Takeover          
Node           Partner        Possible State Description  
-------------- -------------- -------- -------------------------------------
fas2650-a               fas2650-b   true     Connected to fas2650-b      

fas2650-b               fas2650-a   true     Connected to fas2650-a.  Waiting for cluster applications to come online on the local node.

2 entries were displayed.

manage::> set diag -co off

manage::*> cluster kernel-service show

manage::*> cluster kernel-service show
Master            Cluster           Quorum Availability  Operational
Node              Node              Status        Status        Status
----------------- ----------------- ------------- ------------- -------------
fas2650-a   fas2650-a   in-quorum     true operational
                      fas2650-b   out-of-quorum false         unknown
fas2650-b   fas2650-a out-of-quorum false         operational
                      fas2650-b   out-of-quorum true          operational
4 entries were displayed.


ura-fas2650-manage::*> cluster kernel-service node show
                 Is     Cluster
     Quorum        Availability Operational
Node             Master Node             Status        Status Status
---------------- ------ ---------------- ------------- ------------ -----------
fas2650-a  true   fas2650-a in-quorum     true         operational
                                fas2650-b  out-of-quorum false unknown
fas2650-b  true   fas2650-a  out-of-quorum false        operational
 fas2650-b  out-of-quorum true         operational
4 entries were displayed.

manage::> storage failover giveback -ofnode fas2650-b -override-vetoes true

Warning: Initiating a giveback with vetoes overridden will result in giveback
        proceeding even if the node detects outstanding issues that would make
         a giveback dangerous or disruptive. Do you want to continue?
          {y|n}: y

Error: command failed: Failed to initiate giveback. Reason: A giveback is
       already in progress. 

  • EMS log appears following "bsdsocket" errors and "SPLinkDownAlert" message.

Example:

  [?] Fri Jul 10 18:34:40 JST [fas2650-a: kernel: bsdsocket.queueLimit.reached:error]: Number of connection requests for the socket bound to port 935 having the IP address * has reached the limit of 1536 connections. The socket uses TCP protocol. This event has occurred 62 times since the last warning.
  [?] Fri Jul 10 18:35:16 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:40:00 JST [fas2650-a: kernel: bsdsocket.queueLimit.reached:error]: Number of connection requests for the socket bound to port 935 having the IP address * has reached the limit of 1536 co
nnections. The socket uses TCP protocol. This event has occurred 49 times since the last warning.
  [?] Fri Jul 10 18:40:41 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:45:13 JST [fas2650-a: spmd: spm.servprocd.process.exit:alert]: Service processor daemon (servprocd) with ID 4174 exited as a result of normal exit (1). This server will attempt to restart.
  [?] Fri Jul 10 18:45:15 JST [fas2650-a: servprocd: mdb.recovery.successful:debug]: params: {'name': 'servprocd'}
  [?] Fri Jul 10 18:46:06 JST [fas2650-a: dsa_vmm: hamsg.noSend:debug]: params: {'req_op': '6', 'remoteID': 'bb7f5031722311e7ab45a7880059a5e9', 'status': '1', 'scope_err': '0', 'scope': '16'}
  [?] Fri Jul 10 18:50:19 JST [fas2650-a: nphmd: hm.alert.cleared:info]: params: {'alert_id': 'SPLinkDownAlert', 'alerting_resource': 'SP Config', 'detailed_info': 'Alert Id = SPLinkDownAlert , Alerting Resource = SP Config', 'monitor': 'controller'}

 

CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support