Constantly high Java Heap Usage regardless of reboot on StorageGRID pre 11.1
Applies to
- StorageGRID Webscale 10.3
- StorageGRID Webscale 10.4
- StorageGRID Webscale 11.0
Issue
Alarms of CAHP (Java Heap Usage) are constantly reported.
'top -b -n 1'
in system_commands from lumberjack shows cassand+ is heavily consuming memory and CPU:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
652 cassand+ 20 0 63.437g 0.011t 2.889g S 125.3 47.1 1355:59 java
'nodetool cfstats'
in cassandra/system_commands from lumberjack shows there exists large row in key_prefix_by_bucket
:
Table: key_prefix_by_bucket
SSTable count: 8
Space used (live), bytes: 1880268774
Space used (total), bytes: 1880268774
Off heap memory used (total), bytes: 724800
SSTable Compression Ratio: 0.3176106335841224
Number of keys (estimate): 1024
Memtable cell count: 2970
Memtable data size, bytes: 808762
Memtable switch count: 428
Local read count: 4225600
Local read latency: 6.692 ms
Local write count: 86867
Local write latency: 0.026 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used, bytes: 10688
Bloom filter off heap memory used, bytes: 10624
Index summary off heap memory used, bytes: 384
Compression metadata off heap memory used, bytes: 713792
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 1386179893 <<<< 1.4G
Compacted partition mean bytes: 132014784
Average live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0