Skip to main content
NetApp Knowledge Base

Very low WRITE throughput on CVO instance

  1. Last updated
  2. Save as PDF
Views:
825
Visibility:
Public
Votes:
0
Category:
cloud-volumes-ontap-cvo
Specialty:
cloud_performance
Last Updated:

Applies to

  • Azure: Cloud Volumes ONTAP
  • ONTAP: 9.7P5

  • CVO instance Standard_DS5_v2 with premium SSDs and fast write mode enabled

Issue

  • Extremely low throughput and high latency on a volume running on single CVO instance with Fast Write Mode and premium SSD aggregate.
  • Write throughput of 30MB/s was much less than expected throughput.

Cause

  • The issue was reproduced using Disk Performance benchmark tools over CIFS.
  • Analysis of data highlighted very high CPU run time for WRITE messages.

Solution

  • Issue resolved after upgrade to ONTAP 9.8RC1.
  • The causes are still investigated under Bug: 1362945

Additional Information

 

Internal Notes

  • There are no suspends, but the permsg_cpu_avg is very high and the cause for which could not be identified.
  • With lack of support to collect HWPMC on CVO we could not identify the cause for the long running WAFL_SPINNP_WRITE messages in STRIPE affinities. 
  • This is CIFS workload.

time interval                         waffi_per_msg                                            waffi_per_msg    waffi_per_msg  waffi_per_msg
                                      instance                                                permsg_cpu_avg  permsg_wait_avg       num_msgs
                                                                                                   (us)            (us)                     
------------------------------------  ------------------------------------------------------  --------------  ---------------  -------------
2020-11-17 06:33:53 - 06:38:53+00:00  STRIPEGROUP-lopri-WAFL_SPINNP_WRITE-85                         5894.00             6.00               
2020-11-17 06:33:53 - 06:38:53+00:00  STRIPE-lopri-WAFL_CA_COOL_VBNS-94                              1033.00            10.00               
2020-11-17 06:33:53 - 06:38:53+00:00  VOL_LOG-lopri-WAFL_SPINNP_WRITE-66                             1929.16           178.58          19.00
2020-11-17 06:33:53 - 06:38:53+00:00  STRIPEGROUP-hipri-WAFL_SPINNP_WRITE-86                         7929.91            55.64          11.00
2020-11-17 06:33:53 - 06:38:53+00:00  STRIPEGROUP-hipri-WAFL_SPINNP_WRITE-84                         7852.89            10.56           9.00
2020-11-17 06:33:53 - 06:38:53+00:00  STRIPEGROUP-hipri-WAFL_SPINNP_WRITE-85                         6591.14           467.43           7.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-139                             7730.80             7.00           5.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-138                             5693.67             8.33           6.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_CA_COOL_VBNS-135                             1015.33             6.33           3.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-142                             3755.14            85.73         112.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-146                             4905.25             6.75           4.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-144                             6254.00            10.00           3.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-145                             7926.50             5.50           2.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-142                             6740.00           149.80           5.00
2020-11-17 06:33:54 - 06:38:54+00:00  STRIPE-lopri-WAFL_SPINNP_WRITE-143                             3978.00            14.00               


2020-11-17 06:58:54 - 07:03:53+00:00  STRIPEGROUP-hipri-WAFL_SPINNP_WRITE-86                        29016.78            19.90          49.00

2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-138                             1008.58          7582.60       13079.00
2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-144                            17190.87         25397.33         468.00 <<<
2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-145                            17665.78         24144.29         463.00
2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-146                            20389.71         27106.90         394.00 <<<<
2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-140                             1028.79          7105.39       12120.00
2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-141                             3789.29         11264.77        3879.00
2020-11-17 07:53:54 - 07:58:53+00:00  STRIPE-hipri-WAFL_SPINNP_WRITE-142                             3180.00         10399.63        4651.00


time interval                         wafl                wafl               wafl                  wafl
                                      instance  total_cp_msecs  wafl_blks_written  avg_wafl_msg_latency
                                                     (ms)                                  (ms)        
------------------------------------  --------  --------------  -----------------  --------------------
2020-11-17 06:33:53 - 06:38:53+00:00  wafl             5424.00           58016.00                      
2020-11-17 06:38:54 - 06:43:53+00:00  wafl             5708.00           61411.00                      
2020-11-17 06:43:53 - 06:48:53+00:00  wafl             5547.00           52833.00                      
2020-11-17 06:48:54 - 06:53:53+00:00  wafl             5970.00           67917.00                      
2020-11-17 06:53:54 - 06:58:53+00:00  wafl             6130.00           66415.00                      
2020-11-17 06:58:54 - 07:03:53+00:00  wafl             7303.00          103000.00                      
2020-11-17 07:03:53 - 07:08:53+00:00  wafl             7126.00           74876.00                      
2020-11-17 07:08:54 - 07:13:53+00:00  wafl             5446.00           59673.00                      
2020-11-17 07:13:53 - 07:18:53+00:00  wafl             5928.00           65018.00                      
2020-11-17 07:18:54 - 07:23:53+00:00  wafl             5648.00           56860.00                      
2020-11-17 07:23:53 - 07:28:53+00:00  wafl            11086.00          122309.00                      
2020-11-17 07:28:54 - 07:33:53+00:00  wafl            11096.00           53612.00                      
2020-11-17 07:33:53 - 07:38:53+00:00  wafl            50477.00           89342.00                  1.93
2020-11-17 07:38:54 - 07:43:53+00:00  wafl            13860.00           65011.00                      
2020-11-17 07:43:54 - 07:48:53+00:00  wafl            44602.00           66752.00                  1.13
2020-11-17 07:48:54 - 07:53:53+00:00  wafl           102505.00           67922.00                  1.97
2020-11-17 07:53:53 - 07:58:53+00:00  wafl           146334.00           70342.00                  2.67

 

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.