Skip to main content
NetApp Knowledge Base

StorageGRID high CPU utilization by Cassandra database result in increased S3 client latency

  1. Last updated
  2. Save as PDF
Views:
451
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • NetApp StorageGRID
  • Software release 11.6 and later versions

Issue

  1. All the metrics presented below can be found in StorageGRID Grid Manager under Support > Metrics.
  2. Storage node shows Cassandra CPU utilization greater than 85% with minimal I/O wait. This can be found under Node (Internal Use) Grafana dashboard.

High amount of Cassandra Tasks Pending Task Queue Too Large KB CPU Utilization (by service).png

High amount of Cassandra Tasks Pending Task Queue Too Large KB CPU Utilization.png

  1. Cassandra ReadStage is consistently at its maximum level. This can be found under Cassandra Node Overview Grafana dashboard.
High amount of Cassandra Tasks Pending Task Queue Too Large KB Threadpools Active Tasks.png
  1. A single Cassandra table shows extremely high latency in the hundreds of seconds. This can be found under Cassandra Node Overview Grafana dashboard.
    1. Below example shows object_by_version with such high latency. Note that it could be any table.

High amount of Cassandra Tasks Pending Task Queue Too Large KB Read Latency By Table.png

  1. In the node's Cassandra directory, the following command returns more than 100 entries. Consider checking the rotated log file (i.e jstack.log.1). 

# cd /var/local/log/cassandra/jstack/
# grep -c Murmur3 jstack.log
8393

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.