Increased StorageGRID S3 latency and Cassandra metrics

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 137

Visibility:: Public

Votes:: 0

Category:: storagegrid

Specialty:: sgrid

Last Updated:

Applies to

StorageGRID Appliances
Bare-metal nodes

Issue

Increase in S3 latency

2009584645 _Average_Duration1.png

2009584645 _Average_Duration2.png

Increase in Cassandra remote message latency

2009584645 _Remote_Message_Latency.png

Increase in Cassandra dropped messages

2009584645 _Cassandra_Dropped_Messages.png

Increase in Cassandra timeouts

2009584645 _Cassandra_Timeouts.png

Increase in Cassandra pending compaction tasks

2009584645 _Pending_Compaction_Tasks.png

An exception in the Compactor Executor:

// Exception

ERROR [CompactionExecutor:12245] 2023-05-23 20:01:51,515 CassandraDaemon.java (line 231) Exception in thread Thread[CompactionExecutor:12245,5,main] java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code at org.apache.cassandra.db.rows.BTreeRow$Builder$CellResolver.resolve(BTreeRow.java:649) at org.apache.cassandra.db.rows.BTreeRow$Builder$CellResolver.resolve(BTreeRow.java:610) at org.apache.cassandra.utils.btree.BTree$Builder.resolve(BTree.java:1082) at org.apache.cassandra.db.rows.BTreeRow$Builder.build(BTreeRow.java:790) at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:627) at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:480) at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:436) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:74) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:159) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:113) at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:142) at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:145) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:211) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:88) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)