Failed to get the information for aggregate AGGR_NAME. Reason: ZSM - failed

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 689

Visibility:: Public

Votes:: 0

Category:: fas-systems

Specialty:: perf

Last Updated:

Applies to

ONTAP 9.2+
Kernel Services and Management Functions (KSMF)

Issue

Clustershell command aggregate show shows unknown state with error Failed to get the information for aggregate aggr_node01_data01. Reason: ZSM - failed

::> aggr show

Info: Failed to get the information for aggregate aggr_node01_data01. Reason: ZSM - failed, status code = 571, extra = Timeout: Operation "ksmfRawZapi_iterator::get_imp()" took longer than 110 seconds to complete [from mgwd on node

"node01" (VSID: -1) to kernel at 127.0.0.1], took 109.999s, max 110s [127.0.0.1:951].

Failed to get the information for aggregate aggr_node01_data02. Reason: ZSM - failed, status code = 571, extra = Timeout: Operation "ksmfRawZapi_iterator::get_imp()" took longer than 110 seconds to complete [from mgwd on node

"node01" (VSID: -1) to kernel at 127.0.0.1], took 109.996s, max 110s [127.0.0.1:951].

Aggregate Size Available Used% State #Vols Nodes RAID Status

--------- -------- --------- ----- ------- ------ ---------------- ------------

aggr_node01_data01

- - - unknown - node01 -

aggr_node01_data02

- - - unknown - node01 -

aggr_node01_root

1.40TB 69.28GB 95% online 1 node01 raid_dp,

normal

aggr_node02_data01

254.4TB 63.30TB 75% online 265 node02 raid_dp,

normal

aggr_node02_data02

254.4TB 56.20TB 78% online 116 node02 raid_dp,

normal

aggr_node02_root

1.40TB 69.31GB 95% online 1 node02 raid_dp,

normal

6 entries were displayed.

Both nodes are reporting ksmf.svc.watchdog messages in EMS:

[?] Mon Nov 28 01:18:26 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 375 times in last 3617 seconds.

[?] Mon Nov 28 01:18:26 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 02:19:05 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 389 times in last 3639 seconds.

[?] Mon Nov 28 02:19:05 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 03:19:31 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 336 times in last 3626 seconds.

[?] Mon Nov 28 03:19:31 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 04:19:58 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 356 times in last 3627 seconds.

[?] Mon Nov 28 04:19:58 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 05:20:16 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 308 times in last 3618 seconds.

[?] Mon Nov 28 05:20:16 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 06:22:38 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 234 times in last 3742 seconds.

[?] Mon Nov 28 06:22:38 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 07:23:01 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 303 times in last 3623 seconds.

[?] Mon Nov 28 07:23:01 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 08:23:44 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 269 times in last 3635 seconds.

[?] Mon Nov 28 08:23:44 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 09:26:40 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 290 times in last 3770 seconds.

[?] Mon Nov 28 09:26:40 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 10:26:55 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 293 times in last 3615 seconds.

[?] Mon Nov 28 10:26:55 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 11:26:58 +0900 [node01: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 291 times in last 3602 seconds.

[?] Mon Nov 28 11:26:58 +0900 [node01: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 01:02:06 +0900 [node02: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 12 times in last 3893 seconds.

[?] Mon Nov 28 01:02:06 +0900 [node02: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 06:22:50 +0900 [node02: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 14 times in last 19244 seconds.

[?] Mon Nov 28 06:22:50 +0900 [node02: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 07:23:57 +0900 [node02: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 247 times in last 3667 seconds.

[?] Mon Nov 28 07:23:57 +0900 [node02: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

[?] Mon Nov 28 10:03:05 +0900 [node02: ksmf_timeout_thread: ems.engine.suppressed:debug]: Event 'ksmf.svc.watchdog' suppressed 57 times in last 9545 seconds.

[?] Mon Nov 28 10:03:05 +0900 [node02: ksmf_timeout_thread: ksmf.svc.watchdog:debug]: "kSMF service thread held > 25 (sec) by application for table ksmfRawZapi"

Depending on the system, rdb.ha.mboxError and configbr.backupFailed may also be encountered:

rdb.ha.mboxError: Bidirectional failover under the 'cluster HA' configuration is not currently functional due to problem with the on-disk mailboxes. configbr.backupFailed: The node configuration backup node-01.8hour.2025-02-05.10_15_05.7z cannot be created. Error: 'Failed to create the Snapshot copy of the root volume for backup "node-01.8hour.2025-02-05.10_15_05.7z". Error: NetApp Manageability SDK API for creating the Snapshot copy on the root volume failed with error code=13057,module=19,msg=ZSM - failed, status code = 571, extra = Timeout: Operation "ksmfRawZapi_iterator::get_imp()" took longer than 250 seconds to complete [from mgwd on node "node-01" (VSID: -3) to kernel at 127.0.0.1], took 249.996s, max 250s [169.254.247.112:95'.