Skip to main content
NetApp Knowledge Base

Constantly high Java Heap Usage regardless of reboot on StorageGRID pre 11.1

Views:
404
Visibility:
Public
Votes:
0
Category:
storagegrid-webscale
Specialty:
sgrid
Last Updated:

Applies to

  • StorageGRID Webscale 10.3
  • StorageGRID Webscale 10.4
  • StorageGRID Webscale 11.0

Issue

Alarms of CAHP (Java Heap Usage) are constantly reported.

'top -b -n 1' in system_commands from lumberjack shows cassand+ is heavily consuming memory and CPU:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  652 cassand+  20   0 63.437g 0.011t 2.889g S 125.3 47.1   1355:59 java


'nodetool cfstats' in cassandra/system_commands from lumberjack shows there exists large row in key_prefix_by_bucket:

        Table: key_prefix_by_bucket
        SSTable count: 8
        Space used (live), bytes: 1880268774
        Space used (total), bytes: 1880268774
        Off heap memory used (total), bytes: 724800
        SSTable Compression Ratio: 0.3176106335841224
        Number of keys (estimate): 1024
        Memtable cell count: 2970
        Memtable data size, bytes: 808762
        Memtable switch count: 428
        Local read count: 4225600
        Local read latency: 6.692 ms
        Local write count: 86867
        Local write latency: 0.026 ms
        Pending tasks: 0
        Bloom filter false positives: 0
        Bloom filter false ratio: 0.00000
        Bloom filter space used, bytes: 10688
        Bloom filter off heap memory used, bytes: 10624
        Index summary off heap memory used, bytes: 384
        Compression metadata off heap memory used, bytes: 713792
        Compacted partition minimum bytes: 150
        Compacted partition maximum bytes: 1386179893    <<<< 1.4G
        Compacted partition mean bytes: 132014784
        Average live cells per slice (last five minutes): 1.0
        Average tombstones per slice (last five minutes): 0.0

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.